feat(cookbook): MTP --max-running-requests callout + skill sync (#28340)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
zijiexia
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
4c0457f440
commit
7221be2cec
@@ -131,6 +131,7 @@ schemas (full reference in the `_playground.jsx` header):
|
||||
| `pdDisagg` | Mode + transfer backend (+ optional per-backend env via `envWhen` hw-gate) + IB device + optional `router{port, command}` | Model supports prefill/decode disaggregation. When a PD role is active and `router` is set, the playground shows the router (SGLang Model Gateway) launch command as a separate companion block and retargets the cURL modal to `router.port` (clients hit the router, not the role servers). |
|
||||
| `hicache` | Enable + storage + write policy | Model is large enough that hierarchical KV cache matters. |
|
||||
| `hisparse` | Enable + host-ratio select; whole card gated on the live PD-Disagg mode being `decode` | DSA-style model (DeepSeek-V3.2 / V4, GLM-5) that supports decode-side hierarchical sparse attention. |
|
||||
| `flagSelects` | A config-declared **list** of single-selects, each `{ id, title, stripPrefixes, options }` (option = `{ id, label, flags?, hide?, disable?, disableReason? }`); a flagless option is the "none"/accuracy-safe choice | A titled single-select that picks one value of a flag family the other axes don't model — e.g. KV-cache dtype (`--kv-cache-dtype`), mamba scheduler strategy (`--mamba-scheduler-strategy`). Generic: no engine change to add another. |
|
||||
|
||||
**Per-chip constraints**: any chip entry in any axis can be wrapped with
|
||||
`hide` / `disable` constraint objects:
|
||||
@@ -203,6 +204,15 @@ box; omit the import and the prop if the cookbook has no measured numbers
|
||||
yet. See the `_deployment.jsx` header and `deepseek-v4-benchmarks.jsx` for
|
||||
the full speed/accuracy schema.
|
||||
|
||||
Each entry's `sglang_version` must be a **reproducible anchor** — a release
|
||||
tag/version (`v0.5.9`), a commit hash, or (for **Day-0 support**, before the
|
||||
enabling PR merges or a release is cut) a specific PR (`PR #27944`) or commit
|
||||
you can `gh pr checkout` / `git checkout`. Never a moving ref like `"main"` /
|
||||
`"main (2026-06-11)"` (not reproducible). A spec-decoding model whose cell
|
||||
carries `--speculative-algorithm` but no `--max-running-requests` auto-shows
|
||||
an amber Deploy + Playground callout (SGLang otherwise caps it at 48) — it is
|
||||
flag-driven, so no per-page prose is needed.
|
||||
|
||||
To let users *reproduce* those numbers, add a `benchmarkCommands` block to
|
||||
the config (§2.1, next to `curl`). When present alongside `benchmarks`, the
|
||||
benchmark card grows a **"⚡ Reproduce"** button that opens a modal listing
|
||||
|
||||
@@ -42,15 +42,22 @@ your dispatch prompt, or ask for it.
|
||||
(the recipe was measured with it, and fp8 KV halves KV memory —
|
||||
stripping could OOM it). See dimension-mapping.md §2 caveats.
|
||||
2. **Never invent versions or numbers.** Benchmark numbers only from the
|
||||
legacy page's measured blocks. **Speed measurements migrate ONLY when the
|
||||
legacy page pins an exact, reproducible build** (a release tag or a commit
|
||||
hash) — drifting strings like "main branch" are no version anchor: drop the
|
||||
speed numbers AND the entry's `sglang_version`, keep accuracy (far less
|
||||
build-sensitive) and `benchmarkCommands` so ⚡Reproduce guides
|
||||
re-measurement against a pinned release; confirm ambiguous strings (e.g.
|
||||
"0.5.8+") with the maintainer. When kept, `sglang_version` is the legacy
|
||||
page's string verbatim. Docker tags only the ones the legacy page pinned
|
||||
(unmapped hw falls back to `:dev`).
|
||||
legacy page's measured blocks, and a result migrates ONLY when its
|
||||
`sglang_version` is a **reproducible anchor** — the bar is reproducibility,
|
||||
not "must be a release":
|
||||
- ✅ release tag/version (`v0.5.9` / `0.5.9`), commit hash, OR — for
|
||||
**Day-0 support** (the enabling PR isn't merged and no release is cut
|
||||
yet) — a specific **PR (`PR #27944`) or commit** you can `gh pr checkout`
|
||||
/ `git checkout <sha>`. Commit is most precise; a PR pin is fine for day-0.
|
||||
- ❌ a moving ref — `"main branch"`, `"main (2026-06-11)"`, open-ended
|
||||
`"0.5.8+"` — is NOT reproducible: **drop the WHOLE result (speed AND
|
||||
accuracy)**, not just speed. Keep `benchmarkCommands` so ⚡Reproduce still
|
||||
guides re-measurement against a pinned build.
|
||||
**Never inherit cross-model numbers** — measurements the legacy page
|
||||
attributes to a *different model* (e.g. a K2.6 page carrying K2.5-measured
|
||||
speed) are dropped regardless of version. When kept, `sglang_version` is the
|
||||
legacy page's string verbatim. Docker tags only the ones the legacy page
|
||||
pinned (unmapped hw falls back to `:dev`).
|
||||
3. **Verified policy (strictest tier).** `verified: true` ONLY when (a) the
|
||||
legacy page has concrete measured data for that exact 5-dim combo AND
|
||||
(b) the cell's flags equal the deployment command used for that measurement
|
||||
@@ -64,11 +71,14 @@ your dispatch prompt, or ask for it.
|
||||
them in the PR body for the re-verification track.
|
||||
4. **Engines are read-only.** `_deployment.jsx` / `_playground.jsx` must not
|
||||
change in a migration PR. Model-specific features are config DATA consumed
|
||||
by generic axis handlers (MegaMoE precedent), so they need NO engine
|
||||
change — only a genuinely new control *shape* does, and then as a one-time
|
||||
generic primitive (never a model-named handler) on a separate prior PR
|
||||
(engine-axis.md). KV Cache DType / mamba-cache select trigger this once for
|
||||
the whole round; afterwards both are config.
|
||||
by generic axis handlers (MegaMoE precedent), so they need NO engine change.
|
||||
A **titled single-select that strips a flag family** — KV Cache DType
|
||||
(`--kv-cache-dtype`), mamba (`--mamba-scheduler-strategy`), … — is already
|
||||
covered by the merged generic **`flagSelects`** axis: declare it in the
|
||||
config (a list of `{ id, title, stripPrefixes, options }`; see the Qwen3.5
|
||||
mamba example), **no engine PR**. Only a genuinely new control *shape* that
|
||||
`flagSelects` can't express would need a one-time generic primitive (never a
|
||||
model-named handler) on a separate prior PR (engine-axis.md).
|
||||
5. **`github.cookbookModel` must be set** (`<hf-org>/<page-slug>`, e.g.
|
||||
`qwen/qwen3.5`) and the block never pruned — without it Submit ↗ mislabels
|
||||
as deepseek-v4. The issue template itself needs NO edits (free-form input).
|
||||
@@ -117,12 +127,13 @@ Playground-only (DSv4 convention). **Every legacy control survives as an
|
||||
interactive control** — a dimension or a Playground axis, never a tips-only
|
||||
mention — and a model-specific control is **config data, not engine code**
|
||||
(MegaMoE W4A4 is all DSv4 config on the existing `moe` axis). It's pure
|
||||
config whenever it fits an existing axis's data schema. Only a genuinely new
|
||||
*shape* (Nemotron3's "KV Cache DType" — a titled single-select stripping a
|
||||
flag family no axis manages) needs the engine, and then as a ONE-TIME
|
||||
generic config-parameterized primitive (never a model-named handler) on a
|
||||
separate PRIOR engine PR, keeping the migration PR data-only (hard rule 4,
|
||||
engine-axis.md). The strategy count follows the page's
|
||||
config whenever it fits an existing axis's data schema. A **titled
|
||||
single-select that strips a flag family** (Nemotron3's "KV Cache DType",
|
||||
mamba `--mamba-scheduler-strategy`, …) fits the merged generic **`flagSelects`**
|
||||
axis — so it too is config-only (declare a `flagSelects` list). Only a control
|
||||
whose *shape* `flagSelects` still can't express would need a ONE-TIME generic
|
||||
primitive (never a model-named handler) on a separate PRIOR engine PR, keeping
|
||||
the migration PR data-only (hard rule 4, engine-axis.md). The strategy count follows the page's
|
||||
operating points: **one recipe → a single `balanced`; two → `low-latency` +
|
||||
`high-throughput`; three → the full trio (the ideal)**. The tiers apply per
|
||||
(hw × variant × quant) combination — a single-recipe combination on a
|
||||
|
||||
@@ -13,9 +13,9 @@ this file is about the *mapping decisions*.
|
||||
| model-size / model-name radio | `variants` | One variant per deployable checkpoint family; single `{id:"default"}` when there's no variant axis (then `modelNames` keys drop the variant half). |
|
||||
| quantization radio | `quantizations` | Real precision ids (`bf16`/`fp8`/`fp4`/`int4`/…). One `fp4` id even when checkpoints differ per vendor — route via `hw\|variant\|quant` triple keys in `modelNames` (NVFP4 on Blackwell vs AMD MXFP4 is the precedent); per-hw greying falls out of which cells exist. |
|
||||
| toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--dp N --enable-dp-attention` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). |
|
||||
| toggle that only adds/removes its own flags | Playground axis (+ bake, EXCEPT parsers and accuracy-degrading flags) | **Parsers (`--reasoning-parser` / `--tool-call-parser`) are NEVER baked into cells** — Deployment commands ship without them regardless of the legacy default or the measured command; the `parsers` axis adds them on top (DSv4 convention; cells mirror the legacy generator's parsers-OFF output). Accuracy-degrading toggles are never baked either — §2 caveats (axis-only, accuracy-safe cells). Other flag-only toggles: legacy default ON → bake into cells AND declare the axis so users can strip (red strikethrough); default OFF → keep cells clean, axis preset only. MTP/EAGLE presets → `speculative` axis; dp-attention → a strategy when the legacy page labels it as the operating-point split or when coupled (see the row above), else `attention.dpAttn`. **EVERY legacy control survives as an interactive control** (a dimension or a Playground axis), never a tips-only mention — but a model-specific control is **config DATA, not engine code**: the axis handler reads options/flags/env/gating straight from `config.playgroundFeatures` (MegaMoE W4A4 is entirely DSv4 config data on the existing `moe` axis — no per-model engine edit). A control that fits an existing axis's data schema is therefore pure config, full stop. Only when the *shape* is genuinely new — a titled single-select that strips a flag family no axis manages, e.g. Nemotron3's "KV Cache DType" (`--kv-cache-dtype`) — does the engine need that shape, and then you add it ONCE as a **generic config-parameterized primitive** (title + strip-prefixes + options all from config), **never** a model-named `kvcache` handler; afterwards this feature and every future one of its shape are pure config. Such a primitive is backward-compatible — runtime is opt-in per key (`if (!fc) continue`), and a model-specific axis is NOT added to the opt-out general set — so it churns zero existing configs (engine-axis.md). |
|
||||
| toggle that only adds/removes its own flags | Playground axis (+ bake, EXCEPT parsers and accuracy-degrading flags) | **Parsers (`--reasoning-parser` / `--tool-call-parser`) are NEVER baked into cells** — Deployment commands ship without them regardless of the legacy default or the measured command; the `parsers` axis adds them on top (DSv4 convention; cells mirror the legacy generator's parsers-OFF output). Accuracy-degrading toggles are never baked either — §2 caveats (axis-only, accuracy-safe cells). Other flag-only toggles: legacy default ON → bake into cells AND declare the axis so users can strip (red strikethrough); default OFF → keep cells clean, axis preset only. MTP/EAGLE presets → `speculative` axis; dp-attention → a strategy when the legacy page labels it as the operating-point split or when coupled (see the row above), else `attention.dpAttn`. **EVERY legacy control survives as an interactive control** (a dimension or a Playground axis), never a tips-only mention — but a model-specific control is **config DATA, not engine code**: the axis handler reads options/flags/env/gating straight from `config.playgroundFeatures` (MegaMoE W4A4 is entirely DSv4 config data on the existing `moe` axis — no per-model engine edit). A control that fits an existing axis's data schema is therefore pure config, full stop. A **titled single-select that strips a flag family** (e.g. Nemotron3's "KV Cache DType" `--kv-cache-dtype`) is covered by the merged generic **`flagSelects`** axis → **config-only**: declare a `flagSelects` list of `{ id, title, stripPrefixes, options }` (see the Qwen3.5 mamba example), **no engine PR**. Only a control whose *shape* `flagSelects` still can't express would need a new ONE-TIME generic primitive (never a model-named handler) on a prior engine PR; the backward-compat reasoning (opt-in per key, not in the opt-out set) is in engine-axis.md. |
|
||||
| per-combo hidden option (e.g. spec hidden on Xeon) | absent cells | Don't create cells for combos the legacy widget couldn't produce; the engine greys them automatically. `# Error:` pseudo-commands → no cell + explanation in §2 tips and/or a chip `disable`/`disableReason`. |
|
||||
| coupled secondary knob (e.g. mamba cache V1/V2) | cells + Playground axis | Bake the correct value per cell following the legacy coupling (Qwen3.5: MTP ⇒ `--mamba-scheduler-strategy extra_buffer` on NVIDIA; AMD/Xeon ⇒ V1/no flag) and document the coupling in §2 tips — AND surface the knob as a Playground axis like every other legacy feature (row above; add the axis when none fits). Baking alone is NOT enough — the every-feature rule supersedes the pilot's cells+prose-only treatment of Qwen3.5's mamba knob (retrofit pending). The mamba knob is the same single-select shape as KV Cache DType, so it rides the SAME generic primitive (row above) — once that lands, Qwen3.6 / Qwen3-Coder-Next declare it purely in config. |
|
||||
| coupled secondary knob (e.g. mamba cache V1/V2) | cells + Playground axis | Bake the correct value per cell following the legacy coupling (Qwen3.5: MTP ⇒ `--mamba-scheduler-strategy extra_buffer` on NVIDIA; AMD/Xeon ⇒ V1/no flag) and document the coupling in §2 tips — AND surface the knob as a Playground axis like every other legacy feature (row above; add the axis when none fits). Baking alone is NOT enough — the every-feature rule supersedes the pilot's cells+prose-only treatment of Qwen3.5's mamba knob (retrofit pending). The mamba knob is the same single-select shape as KV Cache DType, so it rides the merged generic **`flagSelects`** axis — Qwen3.6 / Qwen3-Coder-Next declare it purely in config (a `flagSelects` block), **no engine PR**. |
|
||||
|
||||
## 2. Command rewrite table (the ONLY allowed normalizations)
|
||||
|
||||
@@ -25,6 +25,7 @@ this file is about the *mapping decisions*.
|
||||
| `--model X` / `--model-path X` | `--model-path {{MODEL_NAME}}` + `modelNames` key |
|
||||
| `--tp-size N` | `--tp N` |
|
||||
| `--speculative-algo X` (abbreviated) | `--speculative-algorithm X` — the Playground spec axis strips/derives by the full first token only; an abbreviated alias would survive toggles and double up |
|
||||
| `--speculative-algorithm NEXTN` | `--speculative-algorithm EAGLE` — **NEXTN is an alias of EAGLE** (same algorithm). Normalize cells + presets to EAGLE; never expose both NEXTN and EAGLE as separate `speculative` presets (they'd be duplicate chips). Keep a one-line "the bench reported NEXTN, an alias of EAGLE" provenance note where the measured command used it. |
|
||||
| `--expert-parallel-size N` | `--ep N` — the Playground EP knob recognizes/strips only `--ep`; the long form would survive toggles and double up |
|
||||
| (absent) | append `--host {{HOST_IP}}`, `--port {{PORT}}` to every cell |
|
||||
| `--nnodes N --node-rank … --dist-init-addr …` literals | delete; `match.nodes: "multi-N"` + `nodesOptions` entry — the engine injects the trio after the last parallelism anchor plus the multi-node header comment |
|
||||
@@ -46,12 +47,10 @@ Caveats discovered in the pilot:
|
||||
cells mirror the accuracy-safe side (even if the legacy default was the
|
||||
lossy side). The option itself **must survive as a Playground control**
|
||||
— the user's choice may not degrade to a tips mention. Express it as
|
||||
config data on the fitting axis (DSv4 gates W4A4 behind `megamoeQuant`);
|
||||
when no axis models its shape — e.g. Nemotron3-Ultra's "KV Cache DType"
|
||||
radio (None default / fp8_e4m3 / bf16) is a titled single-select the
|
||||
playground has no shape for today — that shape is added ONCE as a
|
||||
generic config-parameterized primitive (row above / engine-axis.md),
|
||||
then this config and every future one declare it as data;
|
||||
config data on the fitting axis (DSv4 gates W4A4 behind `megamoeQuant`;
|
||||
a single-select like Nemotron3-Ultra's "KV Cache DType" radio
|
||||
None/fp8_e4m3/bf16 rides the merged generic **`flagSelects`** axis — declare
|
||||
a `flagSelects` block, config-only, no engine PR);
|
||||
- baked into the recipe's **unconditional/default command** → keep it
|
||||
verbatim. The legacy measurements ran with it, and fp8 KV halves KV
|
||||
memory — stripping could OOM the recipe. Expect this pattern: legacy
|
||||
@@ -74,9 +73,16 @@ variants/hw get `disable` + `disableReason` (per-chip constraints), not
|
||||
removal — e.g. MoE backend/EP greyed out on dense variants.
|
||||
|
||||
`speculative` presets must include every algorithm that actually appears on
|
||||
the page — including the measured command's algorithm when it differs from
|
||||
the generator default (Qwen3.5 ships both NEXTN and EAGLE) — otherwise the
|
||||
verified cell's baseline can't be re-applied after a strip.
|
||||
the page (otherwise a stripped cell's baseline can't be re-applied) — but
|
||||
**collapse aliases**: NEXTN is an alias of EAGLE (§2 rewrite table), so a page
|
||||
benchmarked with NEXTN ships a single `eagle` preset, not both. (Pilot history:
|
||||
Qwen3.5 once shipped both; corrected to EAGLE-only.)
|
||||
|
||||
**MTP `--max-running-requests` hint (engine, automatic):** when a cell's
|
||||
command turns speculative decoding on (`--speculative-algorithm` present)
|
||||
without `--max-running-requests`, the Deploy panel + Playground auto-render an
|
||||
amber callout (SGLang otherwise caps it at 48). It is FLAG-driven, not
|
||||
strategy-driven — nothing to author per page; do NOT duplicate it in §2 prose.
|
||||
|
||||
The `parsers` axis is **add-only**: `--reasoning-parser` /
|
||||
`--tool-call-parser` are never part of any Deployment cell (see §1) — the
|
||||
@@ -171,7 +177,7 @@ model-specific note — never toggle-/migration-centric explanations.
|
||||
| Family | strategies | Notes |
|
||||
|---|---|---|
|
||||
| Gemma4 | `low-latency` (MTP on — the legacy toggle's own "Lower Latency" subtitle) / `high-throughput` (MTP off); mi300x hides the toggle → its single recipe → `balanced` (trio union, Qwen3.5 Xeon pattern) | variants = e2b/e4b/12b/31b/26b-a4b; checkpoint radio Standard(BF16)/QAT(q4_0) → quant ids via `modelNames`; §3.3 prose carries AMD recipes beyond the widget's mi300x — maintainer call on cells-from-prose vs tips; vision/audio invocation prose carries over (deployment matrix is text-standard); "gemma4 branch" version → speed drops, MMLU/GSM8K accuracy keeps (mind the few-shot vs run_eval harness footnote); dedicated multi-arch dev images verbatim |
|
||||
| Nemotron3-Ultra | dpattention carries "Low latency"/"High throughput" subtitles (naming rule) but THREE perf controls stack — multi-value DP-Attention (2/4/8) × MTP × EP — design the tier mapping via the step-2 table; maintainer sign-off required | NVIDIA-only (h100→gb300) with a per-quant verified-hw SUPPORT matrix → absent cells; "Model" radio = the quant dim (BF16 / NVFP4 Blackwell-only); TP radio 8/16 — TP=16 is 2-node → `nodes` dim; **kvcache radio (None/fp8_e4m3/bf16) → NEW Playground axis, engine PR FIRST** (every-feature rule §1); `launch_server` + spec-V2 env prefix verbatim; dedicated `dev-nemotron3-ultra(+cu13)` images verbatim ("not in any stable release"); "main branch" version → speed drops, GSM8K accuracy keeps |
|
||||
| Nemotron3-Ultra | dpattention carries "Low latency"/"High throughput" subtitles (naming rule) but THREE perf controls stack — multi-value DP-Attention (2/4/8) × MTP × EP — design the tier mapping via the step-2 table; maintainer sign-off required | NVIDIA-only (h100→gb300) with a per-quant verified-hw SUPPORT matrix → absent cells; "Model" radio = the quant dim (BF16 / NVFP4 Blackwell-only); TP radio 8/16 — TP=16 is 2-node → `nodes` dim; **kvcache radio (None/fp8_e4m3/bf16) → `flagSelects` axis, config-only** (the generic primitive merged in #28128 — NO engine PR); `launch_server` + spec-V2 env prefix verbatim; dedicated `dev-nemotron3-ultra(+cu13)` images verbatim ("not in any stable release"); **"main branch" version is non-reproducible → drop the WHOLE measured result (speed AND accuracy)** unless it can be pinned to the support PR/commit (day-0 rule, §hard-rule-2) |
|
||||
| GLM-4.5, GLM-4.6 | `low-latency` (TP, + MTP from the legacy checkbox) / `high-throughput` (TP+DP+EP) | |
|
||||
| GLM-4.7 | `low-latency`(2 GPUs) / `balanced`(4) / `high-throughput`(8) — gpus 2/4/8 + SUPPORT matrix; confirm naming, tiers are GPU budgets | measured-best B200 TP=2 NVFP4 → the verified cell |
|
||||
| GLM-4.7-Flash | `low-latency` (tp1 + MTP from the legacy checkbox) / `high-throughput` (DP) | derive from the legacy dp/mtp checkboxes |
|
||||
@@ -202,11 +208,12 @@ Decisions log, in the order they came up:
|
||||
Result: 186 cells = 87 low-latency + 87 high-throughput + 12 balanced; the
|
||||
page ships the full trio and the engine greys unused chips per selection.
|
||||
2. **Verified cell follows the measurement**: H200/397B/BF16/low-latency =
|
||||
`SGLANG_USE_CUDA_IPC_TRANSPORT=1` env + `--speculative-algorithm NEXTN`
|
||||
(normalized spelling) + measured flag set **minus the parser flags** (the
|
||||
measured run had both parsers on; cells never carry them — noted in the
|
||||
benchmarks header). All other cells = the generator's parsers-OFF output
|
||||
verbatim with `EAGLE`. Both spec presets exposed on the speculative axis.
|
||||
`SGLANG_USE_CUDA_IPC_TRANSPORT=1` env + `--speculative-algorithm EAGLE`
|
||||
(the bench reported NEXTN, an alias of EAGLE — normalized to EAGLE, §2) +
|
||||
measured flag set **minus the parser flags** (the measured run had both
|
||||
parsers on; cells never carry them — noted in the benchmarks header). All
|
||||
other cells = the generator's parsers-OFF output verbatim. A single `eagle`
|
||||
spec preset on the speculative axis (the duplicate NEXTN preset was dropped).
|
||||
3. **FP4 single quant id** with `hw|variant|quant` modelNames keys →
|
||||
`nvidia/...NVFP4` (b200/b300) vs `amd/...MXFP4` (mi355x).
|
||||
4. **Xeon** as `config.hardware` `vendor:"intel"`; cells carry
|
||||
|
||||
@@ -128,6 +128,15 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
color: isDark ? "#e5e7eb" : "#374151",
|
||||
whiteSpace: "pre-wrap", overflowX: "auto", margin: 0,
|
||||
},
|
||||
// Amber callout under the command when speculative decoding (MTP) is on
|
||||
// but --max-running-requests isn't set (SGLang then caps it at 48).
|
||||
mtpWarn: {
|
||||
margin: "8px 0 0", padding: "8px 12px", borderRadius: "8px",
|
||||
fontSize: "12px", lineHeight: "1.45",
|
||||
background: isDark ? "#78350f" : "#fef3c7",
|
||||
color: isDark ? "#fde68a" : "#92400e",
|
||||
border: `1px solid ${isDark ? "#92400e" : "#fcd34d"}`,
|
||||
},
|
||||
badge: (verified) => ({
|
||||
display: "inline-flex", alignItems: "center", gap: "6px",
|
||||
padding: "2px 8px", borderRadius: "10px",
|
||||
@@ -904,6 +913,13 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
const s = makeStyles(isDark);
|
||||
const cell = findCell(config.cells, sel);
|
||||
const command = renderCommand(cell, sel, env, runMode);
|
||||
// MTP hint: fire on the actual command (speculative decoding ON) — NOT on
|
||||
// strategy=low-latency, since a low-latency cell may not enable MTP. SGLang
|
||||
// resets --max-running-requests to 48 when spec is on and it's unset.
|
||||
const mtpHint =
|
||||
!!cell &&
|
||||
(cell.flags || []).some((f) => f.split(/[\s=]/)[0] === "--speculative-algorithm") &&
|
||||
!(cell.flags || []).some((f) => f.split(/[\s=]/)[0] === "--max-running-requests");
|
||||
const modelName = resolveModelName(sel);
|
||||
const curlText = interpolate(config.curl || "", env, modelName);
|
||||
const hwGroups = buildHardwareGroups();
|
||||
@@ -1042,6 +1058,11 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
</div>
|
||||
</div>
|
||||
<pre style={s.commandPre}>{command}</pre>
|
||||
{mtpHint && (
|
||||
<div style={s.mtpWarn}>
|
||||
⚠️ Speculative decoding (MTP) is on — SGLang resets <code>--max-running-requests</code> to <strong>48</strong> when it isn't set. Add <code>--max-running-requests <N></code> sized for your target concurrency.
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -1306,6 +1306,16 @@ export const Playground = ({ config }) => {
|
||||
color: isDark ? "#e5e7eb" : "#374151",
|
||||
whiteSpace: "pre-wrap", overflowX: "auto", margin: 0,
|
||||
},
|
||||
// Amber callout under the playground command when the effective (post-
|
||||
// override) command turns speculative decoding on without setting
|
||||
// --max-running-requests (SGLang then caps it at 48).
|
||||
mtpWarn: {
|
||||
margin: "8px 0 0", padding: "8px 12px", borderRadius: "8px",
|
||||
fontSize: "12px", lineHeight: "1.45",
|
||||
background: isDark ? "#78350f" : "#fef3c7",
|
||||
color: isDark ? "#fde68a" : "#92400e",
|
||||
border: `1px solid ${isDark ? "#92400e" : "#fcd34d"}`,
|
||||
},
|
||||
diffLineUnchanged: { display: "block" },
|
||||
diffLineAdded: {
|
||||
display: "block",
|
||||
@@ -1652,6 +1662,12 @@ export const Playground = ({ config }) => {
|
||||
const playgroundVerified = !!(matchedCell && matchedCell.verified);
|
||||
const matchedSiblingCell = (matchedCell && matchedCell !== baseCell)
|
||||
? matchedCell : null;
|
||||
// MTP hint on the EFFECTIVE (post-override) command — fires when the user
|
||||
// toggles speculative decoding on without setting --max-running-requests
|
||||
// (NOT keyed on strategy). Mirrors the Deploy panel's hint.
|
||||
const pgMtpHint =
|
||||
pgFlagsLatest.some((f) => f.split(/[\s=]/)[0] === "--speculative-algorithm") &&
|
||||
!pgFlagsLatest.some((f) => f.split(/[\s=]/)[0] === "--max-running-requests");
|
||||
|
||||
// Submission snippets: proposed cell + existing cell at the same match.
|
||||
const proposedCellSnippet = baseCell
|
||||
@@ -1901,6 +1917,11 @@ export const Playground = ({ config }) => {
|
||||
</span>
|
||||
)) : "# No verified base cell at the current Deployment selection.\n# Pick a supported hardware/variant in the Deployment panel to populate the playground base."}
|
||||
</pre>
|
||||
{pgMtpHint && (
|
||||
<div style={s.mtpWarn}>
|
||||
⚠️ Speculative decoding (MTP) is on — SGLang resets <code>--max-running-requests</code> to <strong>48</strong> when it isn't set. Add <code>--max-running-requests <N></code> sized for your target concurrency.
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
Reference in New Issue
Block a user