docs(cookbook): add the Kimi-K3 serving cookbook (#32542)
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai> Co-authored-by: Zijie Xia <zijie.xia@radixark.ai> Co-authored-by: ispobock <ispobaoke@gmail.com> Co-authored-by: Mick <mickjagger19@icloud.com> Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com> Co-authored-by: thomawan <thomawan@amd.com> Co-authored-by: BBuf <1182563586@qq.com>
This commit is contained in:
Liangsheng Yin
co-authored by
kpham-sgl
Zijie Xia
ispobock
Mick
Baizhou Zhang
thomawan
BBuf
kpham-sgl
Zijie Xia
ispobock
Mick
Baizhou Zhang
thomawan
BBuf
parent
8d6549bc40
commit
7dafacca49
@@ -4,7 +4,7 @@ Loaded on demand by the `cookbook-add-model` skill. This is the field-by-field
|
||||
contract for when the clone needs more than a rename. The two engine files are
|
||||
the canonical specs — read their headers first:
|
||||
|
||||
- [`_deployment.jsx`](../../../../docs_new/src/snippets/_deployment.jsx) — the 5-dim matrix widget; lists every config field.
|
||||
- [`_deployment.jsx`](../../../../docs_new/src/snippets/_deployment.jsx) — the matrix widget; its header lists every config field. Dimensions are the legacy fixed five by default, or config-declared via `matchDims` / `overlayDims` (§2.1b).
|
||||
- [`_playground.jsx`](../../../../docs_new/src/snippets/_playground.jsx) — the diff-based override widget; lists the `playgroundFeatures` axes + the `AXIS_HANDLERS` interface.
|
||||
|
||||
Engine extension (adding a new playground axis) lives in [engine-axis.md](engine-axis.md).
|
||||
@@ -34,7 +34,7 @@ the full contract):
|
||||
| `quantizations` | `{id, label}[]` | 3rd-dim option list. |
|
||||
| `strategies` | `{id, label}[]` | 4th-dim option list. Canonical ids: `low-latency` / `balanced` / `high-throughput` (never model-specific ids like `mtp`). **The count follows the page's operating points**: one recipe → a single `balanced`; two → `low-latency` + `high-throughput`; three → the full trio (the ideal). Tiers apply per (hw × variant × quant) combination — a single-recipe combination parks under its semantically honest tier (clear slant → that tier, e.g. DSv4's RTX 6000 → `low-latency`; no slant → `balanced`, e.g. Qwen3.5's Xeon); the page's list is the union and the engine greys unused chips per selection. Never invent a recipe just to fill chips. When two recipes differ by MTP / speculative decoding, the assignment is deterministic: spec ON → `low-latency`, spec OFF → `high-throughput` (at saturation the draft+verify overhead outweighs the speedup — same reason DSv4's high-throughput recipes disable MTP). The recurring markers in the other direction: dp-attention ON (MLA-attention models) and EP / DP+EP ON (MoE models) → `high-throughput`. |
|
||||
| `nodesOptions` | `{id, label}[]` | 5th-dim option list. The `id` MUST be `single` or `multi-N` — the engine parses N from the id for `--nnodes`. |
|
||||
| `cells` | `{match, verified?, env, flags}[]` | One per supported (hw × variant × quant × strategy × nodes) combination. See §2.2. |
|
||||
| `cells` | `{match, verified?, nnodes?, env, flags}[]` | One per supported (hw × match-dim) combination. See §2.2. `nnodes` supplies the node count when the config declares no `nodes` dim (default 1). |
|
||||
| `modelNames` | `{[key]: string}` | HF slug lookup. Keys are either `hw\|variant\|quant` (most specific) or `variant\|quant` (fallback). |
|
||||
| `placeholders` | `{[key]: {target, label, default?}}` | `{{KEY}}` interpolation map for command + curl. `target` is `'command'` or `'curl'`. Editable through the Env modal. |
|
||||
| `curl` | string | cURL template. Uses `{{MODEL_NAME}}` + placeholder keys. |
|
||||
@@ -52,7 +52,42 @@ the full contract):
|
||||
| `latencyPercentile` | `"Mean" \| "P50"` | Optional, **temporary**; the percentile the benchmark TTFT/TPOT values are. **Default `"P50"`** — the card renders `TTFT (<pct>)` / `TPOT (<pct>)`. Set `"Mean"` only for legacy data recorded as Mean (being re-measured to P50). A benchmarks entry may carry its own `latencyPercentile` to override the page value per cell (entry → config → `"P50"`). `tokens_per_sec_per_gpu` is stored as **total (in+out)/GPU** = `output tok/s/GPU × (isl+osl)/osl`, shown by the card as-is. |
|
||||
| `github` | `{owner?, repo?, issueTemplate?, cookbookModel?}` | Overrides for the "Submit verified cell" CTA in the playground. Defaults: `sgl-project/sglang` + `3-playground-verified-cell.yml` + `"deepseek-ai/deepseek-v4"`. Set `cookbookModel` to the model's HF id (`<hf-org>/<model-slug>`); it prefills the issue template's free-form `model` input when the issue opens. **Don't prune this block** — without it the engine falls back to `deepseek-ai/deepseek-v4` and submissions from your page get mislabeled. |
|
||||
|
||||
## 2.2 Author the 5-dim matrix (`cells[]`)
|
||||
## 2.1b Custom dimensions (`matchDims` / `overlayDims`)
|
||||
|
||||
The legacy shape above is a fixed five dimensions. A model whose axes don't fit
|
||||
(no variant axis, a deployment-shape axis, an orthogonal feature toggle) declares
|
||||
its own instead. Declaring `matchDims` replaces `variants`/`quantizations`/
|
||||
`strategies`/`nodesOptions` wholesale; omitting it keeps the legacy behaviour, so
|
||||
existing pages need no change.
|
||||
|
||||
| Field | Shape | Notes |
|
||||
|---|---|---|
|
||||
| `matchDims` | `{id, title, options}[]` | Rows that key the cell lookup. `hw` is always the implicit first dim, so cells match on (hw × these ids). Order is priority: lower rows adapt to higher ones. |
|
||||
| `overlayDims` | `{id, title, default?, showWhen?, options}[]` | Rows that do NOT key the lookup — the picked option layers onto whichever cell matched. Use this for a knob that is orthogonal to the grid (speculative decoding, hierarchical cache): as a match dim it would multiply the cell count, as an overlay it costs nothing. |
|
||||
|
||||
Option shape, both kinds:
|
||||
|
||||
| Key | Meaning |
|
||||
|---|---|
|
||||
| `id`, `label` | as usual |
|
||||
| `showWhen(sel)` | option is hidden unless the predicate accepts the current selection — this is how one row shows a different option set per mode |
|
||||
| `disabled`, `disableReason` | `true` or a predicate over the selection; greys the option out and supplies the tooltip. Use it for a combination the server rejects, so the reader learns why instead of hitting a startup error |
|
||||
|
||||
Overlay options additionally take `flags`, `env` and `hints` — each a literal array
|
||||
or a function of the whole selection (so an "auto" value can resolve against another
|
||||
row). `hints` render as `# ...` comment lines above the command, for setup the
|
||||
launch line cannot express on its own.
|
||||
|
||||
`_playground.jsx` sees the overlay dims in its `base`, so a playground axis can gate
|
||||
on them with its own `showWhen(base)` — an axis whose feature the Deploy panel never
|
||||
switched on is not rendered at all. Changing the Deploy selection resets every
|
||||
playground axis back to inherit-from-base.
|
||||
|
||||
> The overlay resolution rule is written in BOTH engines (snippets can't import each
|
||||
> other); each copy is marked `MIRROR`. Change both or neither, or the Deploy command
|
||||
> and the playground base silently disagree.
|
||||
|
||||
## 2.2 Author the matrix (`cells[]`)
|
||||
|
||||
Each cell describes one verified (or auto-estimated) launch recipe.
|
||||
|
||||
|
||||
@@ -3,7 +3,6 @@ title: Kimi-K2.7-Code
|
||||
description: "Deploy Kimi-K2.7-Code with SGLang for coding-focused agentic workflows, thinking output, tool calling, and multimodal input."
|
||||
metatags:
|
||||
description: "Deploy Kimi-K2.7-Code native multimodal agentic model with SGLang - reasoning, tool calling, and multimodal capabilities."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
@@ -0,0 +1,403 @@
|
||||
---
|
||||
title: Kimi-K3
|
||||
description: "Deploy Moonshot AI's Kimi-K3 with SGLang — a 2.8T-parameter hybrid Mixture-of-Experts vision-language model (Kimi Delta Attention + MLA, 16/896 active experts) with NVIDIA and AMD recipes."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install).
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:kimi-k3 # CUDA13
|
||||
docker pull lmsysorg/sglang:kimi-k3-cu12 # CUDA12
|
||||
docker pull lmsysorg/sglang-rocm:rocm720-mi35x-k3-20260727 # ROCM
|
||||
```
|
||||
|
||||
These tags publish with the public K3 launch; until then, build from the Dockerfiles linked below.
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
If you do not want to use a Docker image, reproduce the dependency installation steps from the [CUDA 13 Dockerfile](https://github.com/DarkSharpness/sglang-kimi/blob/kimi-k3/docker/kimi_k3/kimi_k3_cu13.Dockerfile) or [CUDA 12 Dockerfile](https://github.com/DarkSharpness/sglang-kimi/blob/kimi-k3/docker/kimi_k3/kimi_k3_cu12.Dockerfile).
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware, then the deployment shape and operating point. Node count follows the hardware recipe (B200 2×8, GB200 4×4, H100 4×8, B300 1×8, H200 2×8, GB300 2×4, MI350X/MI355X 1×8), so it is not a separate choice.
|
||||
|
||||
**PD Mode** — `Unified` serves prefill and decode together. `Prefill` / `Decode` split them into dedicated pools (see [PD disaggregation](#pd-disaggregation)); `Prefill` ships two strategies on the TP8 platforms, both chunked at 16k: `Default` (TP8) and `Long-Context` (`--pp-size 8 --tp-size 1`, see [Deep PP](#deep-pp-for-long-context-prefill)).
|
||||
|
||||
**Strategy** — the operating point within that shape:
|
||||
|
||||
- **Low-Latency** — plain TP, no DCP. For chat.
|
||||
- **Balanced** — the accuracy-preserving default: TP16/DCP16 on B200/GB200, TP8/DCP8 on B300/GB300, TP8 ROCm/AITER on MI35x.
|
||||
- **High-Throughput** — the large-scale lane: pick a **Cluster Size** and **Large-Scale Preset** in the Playground ([details](#large-scale-presets)). On H100/H200: Balanced plus `extra_buffer_lazy`.
|
||||
- **Long-Context** — B200 only: TP8/PP2 splits KV and KDA state across two pipeline stages.
|
||||
|
||||
**Spec Decode** — independent of the strategy: DSPARK layers onto any `pp_size == 1` recipe, proposing 7 draft tokens per step (tune in the Playground); DFLASH has no published draft checkpoint. Its win is largest on short interactive traffic and fades as the prompt grows.
|
||||
|
||||
<Note>
|
||||
`--mamba-full-memory-ratio` is the one sizing flag, computed live: set your average request length in the [Mamba ratio calculator](#mamba-ratio-calculator); everything else follows the panels, and the result is pinned into the command.
|
||||
</Note>
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/moonshotai/kimi-k3.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/moonshotai/kimi-k3-benchmarks.jsx";
|
||||
import { KimiK3MambaRatioCalculator } from "/src/snippets/_kimi_k3_mamba_ratio_calculator.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
### Mamba ratio calculator
|
||||
|
||||
<KimiK3MambaRatioCalculator />
|
||||
|
||||
<Accordion title="How --mamba-full-memory-ratio is calculated">
|
||||
|
||||
`--mamba-full-memory-ratio` is the ratio between the KDA state pool and the MLA KV pool. Every parameter below except `L` is read live from the Deploy panel and Playground selection; the balanced value is the per-request cost ratio:
|
||||
|
||||
```text
|
||||
ratio = (S + D) x state_bytes / (L x (mla_kv_bytes / DCP + draft_kv_bytes))
|
||||
```
|
||||
|
||||
- `S` — KDA state slots per request: `extra_buffer=5`, `extra_buffer_lazy=4`, `no_buffer=3`, disabled radix cache `=1`. `SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK` frees one slot on the extra-buffer strategies; with the overlap scheduler off (or `pp > 1`, which disables it) the track buffer costs one slot instead of two.
|
||||
- `D` — verify intermediate states under speculative decoding: `0` when disabled, otherwise DSPARK block size + 1 (`8` at the default 7). ReplaySSM (`--enable-linear-replayssm-spec`) folds them into a per-slot ring, returning `D` to `0`.
|
||||
- `state_bytes` — one state slot's bytes, from K3's fixed geometry, the attention-TP width, and the SSM dtype.
|
||||
- `mla_kv_bytes` — one token's MLA latent KV bytes (KV-dtype dependent); DCP shards it across its ranks. The DSPARK draft model's KV (~1.4 KB per token) is replicated on every rank, so it enters flat — negligible without DCP, the same order as the sharded MLA share under DCP8.
|
||||
- `L` — average total request length in tokens: input + output.
|
||||
|
||||
</Accordion>
|
||||
|
||||
<a id="playground" style={{ scrollMarginTop: "96px" }} />
|
||||
|
||||
## Advanced Features Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the deployment matrix**. The Deploy panel above emits the recipes the SGLang team is converging on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**Kimi-K3** is Moonshot AI's flagship hybrid MoE vision-language model: **2.8 trillion parameters**, **16 of 896 experts** active per token, roughly **2.5× the scaling efficiency of Kimi-K2**. The backbone interleaves **Kimi Delta Attention (KDA)** with MLA across 93 layers (plus Attention Residuals and Stable LatentMoE); serving supports image input and a **1M-token** window with prefix caching. Weights ship in **MXFP4**: the FlashInfer MXFP4 (trtllm-gen SiTU) runner serves them on Blackwell, Marlin (W4A16) elsewhere, MegaMoE for short-context batch throughput.
|
||||
|
||||
K3 **always runs with thinking enabled**, with reasoning depth controlled by `reasoning_effort` (`low` / `high` / `max`; default `max`).
|
||||
|
||||
<Note>
|
||||
Kimi-K3 is Moonshot AI's first open-source model in the trillion-plus class; **full model weights
|
||||
are scheduled to release by July 27, 2026**. The recipes on this page were validated on the
|
||||
[`DarkSharpness/sglang-kimi`](https://github.com/DarkSharpness/sglang-kimi) fork — the HuggingFace
|
||||
repository (`moonshotai/Kimi-K3`) and a public `lmsysorg/sglang` image with K3 support will be
|
||||
available at launch.
|
||||
|
||||
Every cell on this page is currently marked **Not Verified**: the recipes run, but none has a
|
||||
serving round on the final weights and current code behind it. Re-measure throughput and accuracy
|
||||
before you rely on any of them.
|
||||
</Note>
|
||||
|
||||
**Recommended generation:** `temperature=1.0`, `top_p=0.95`, `presence_penalty=0`, `frequency_penalty=0` (fixed by the model; informational — do not hardcode in sample code).
|
||||
|
||||
**Resources:** [HuggingFace](https://huggingface.co/moonshotai/Kimi-K3) · [Kimi-K3 Quickstart](https://platform.kimi.ai/docs/guide/kimi-k3-quickstart).
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
**Memory: two pools, one flag.** K3 splits static memory into a worst-case-reserved **KDA state pool** (it sets the concurrency ceiling) and a paged **MLA KV pool**, divided by `--mamba-full-memory-ratio`. The command panel pins that flag to the [calculator](#mamba-ratio-calculator)'s output — set your average request length there; every other calculator input follows the panels. After boot, read back `max_total_num_tokens` (the KV side) and the admitted-request cap (the state side).
|
||||
|
||||
Capacity levers, all in the Playground. Each trades precision or cache behavior for capacity — re-verify accuracy on your workload:
|
||||
|
||||
| Lever | Effect |
|
||||
|---|---|
|
||||
| `--mamba-radix-cache-strategy extra_buffer_lazy` | 4 state slots per request instead of 5 |
|
||||
| `--mamba-ssm-dtype bfloat16` | ~halves state bytes; with spec on, KDA verification falls back from the fused kernel to Triton |
|
||||
| `--kv-cache-dtype fp8_e4m3` | halves KV bytes per token; under PD both roles must match at connect |
|
||||
| `--mem-fraction-static` 0.90–0.92 | cheapest first win when the boot log shows a large idle `avail mem` |
|
||||
| `SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1` | frees one more slot per request (experimental, under validation) |
|
||||
|
||||
Speculation: DSPARK holds block size + 1 (= 8) intermediate states per request — the calculator folds this in — and an unset `--max-running-requests` resets to 48 under spec (the command panel reminds you; set it explicitly to raise).
|
||||
|
||||
**MoE runner.** Leave `--moe-runner-backend` unset on Blackwell: FlashInfer MXFP4 (W4A8, prebuilt trtllm-gen SiTU kernels) when the cubin pool is installed, Marlin (W4A16) otherwise; H100/H200 pin Marlin. The published Docker images already provision the **SiTU cubin pool**; to install it independently, run the same flow as the Dockerfile:
|
||||
|
||||
```bash
|
||||
wget https://github.com/sgl-project/whl/releases/download/trtllm_gen_moe_cubin_20260617/trtllm_gen_moe_cubin_pool_20260617_v0613rc1.zip
|
||||
sudo mkdir -p /opt/trtllm_gen_moe_cubin_pool
|
||||
sudo unzip -q trtllm_gen_moe_cubin_pool_20260617_v0613rc1.zip -d /opt/trtllm_gen_moe_cubin_pool
|
||||
export SGLANG_TRTLLM_GEN_MOE_CUBIN_POOL=/opt/trtllm_gen_moe_cubin_pool/trtllm_gen_moe_cubin_pool_20260617_v0613rc1
|
||||
```
|
||||
|
||||
Remaining kernel sources JIT once from the public `flashinfer` wheel (a few minutes, cached).
|
||||
|
||||
**Attention backend.** Leave all three attention knobs unset on Blackwell: K3 resolves prefill, decode, and — under DSPARK — verification as a set (`trtllm_mla` across the board; `cutedsl_mla` takes decode and verification under DCP). On the non-DCP recipes, setting any one of the three cancels the auto-resolution for the others. H100/H200 pin `flashmla` for decode.
|
||||
|
||||
**Context length.** `--context-length` bounds the longest accepted request plus some context-scaled buffers; it does not size the KV pool. For long context the lever that adds capacity is `fp8_e4m3` KV.
|
||||
|
||||
**DSPARK.** Adds `--speculative-algorithm DSPARK` plus the draft checkpoint on top of the showing strategy. Leave `--speculative-draft-attention-backend` unset. No serving round on the final draft checkpoint has landed — measure against the same recipe running NOSPEC before adopting.
|
||||
|
||||
**Per-platform notes:**
|
||||
|
||||
| Platform | Topology | Notes |
|
||||
|---|---|---|
|
||||
| B300 1×8 | TP8 (+DCP8) | accuracy-first defaults on Low-Latency and Balanced |
|
||||
| GB300 2×4 | TP8/DCP8 | MNNVL transport and cuMem auto-detected |
|
||||
| B200 2×8 | TP16 (+DCP16); Long-Context TP8/PP2, 128K ctx, 8192 chunked prefill | DSPARK off on Long-Context (`pp_size == 1` required) |
|
||||
| GB200 4×4 | TP16/DCP16 | MNNVL auto-detected |
|
||||
| H200 2×8 | TP16/EP16 + symm-mem | same block on both ranks; export the cross-node NIC (`GLOO_SOCKET_IFNAME` / `NCCL_SOCKET_IFNAME`, `SGLANG_HOST_IP`); keep `NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1` |
|
||||
| H100 4×8 | TP32/EP32, Marlin + FlashMLA | SM90a build of the K3 image; pin NCCL/Gloo to the same NIC on all nodes; least post-weight headroom (80 GB) |
|
||||
| MI350X/MI355X 1×8 | TP8 ROCm/AITER | AITER A8W4 FlyDSL MoE, Triton attention, graph bs up to 256; DSPARK supported |
|
||||
|
||||
**DCP notes** (Blackwell Balanced / High-Throughput):
|
||||
|
||||
- DCP is the only axis that shards the TP-replicated MLA KV; Low-Latency skips it.
|
||||
- Leave `--dcp-comm-backend` unset (fabric-resolved: `fi_a2a` on GB200/GB300, `a2a` on B200/B300).
|
||||
- No `--enable-symm-mem` under DCP (force-disabled for decode-graph correctness).
|
||||
- Explicit `tokenspeed_mla` force-rewrites `--kv-cache-dtype` to fp8; the default `cutedsl_mla` serves either dtype.
|
||||
- Calculator ratios run well above 1 here (`r > 1` is legal): `bfloat16` state buys admission, `fp8` KV buys context.
|
||||
- Don't add EP — its a2a buffers reclaim the KV that DCP buys. Compose only to measure.
|
||||
|
||||
No cell has a serving round in this exact shape — treat them as starting points to verify.
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 Reasoning
|
||||
|
||||
K3 always thinks; the `kimi_k3` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) separates that thinking from the final answer — thinking lands in `message.reasoning_content`, the answer in `message.content`. Control the reasoning depth with `reasoning_effort` (`low` / `high` / `max`; default `max`).
|
||||
|
||||
<Accordion title="Reasoning Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
resp = client.chat.completions.create(
|
||||
model="moonshotai/Kimi-K3",
|
||||
messages=[{"role": "user", "content": "What is 15% of 240?"}],
|
||||
reasoning_effort="high", # "low" | "high" | "max" (default max)
|
||||
)
|
||||
msg = resp.choices[0].message
|
||||
print("Reasoning:", getattr(msg, "reasoning_content", None))
|
||||
print("Answer:", msg.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
Pending update...
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.2 Tool Calling
|
||||
|
||||
Enable the `kimi_k3` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. Because K3 is a thinking model, the follow-up turn may put text in `reasoning_content` as well as `content` — print both.
|
||||
|
||||
<Accordion title="Tool Calling Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
tools = [{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a city",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"city": {"type": "string"}},
|
||||
"required": ["city"],
|
||||
},
|
||||
},
|
||||
}]
|
||||
resp = client.chat.completions.create(
|
||||
model="moonshotai/Kimi-K3",
|
||||
messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
|
||||
tools=tools,
|
||||
)
|
||||
msg = resp.choices[0].message
|
||||
print("Reasoning:", getattr(msg, "reasoning_content", None))
|
||||
print("Tool calls:", msg.tool_calls)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
Pending update...
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.3 HiCache (Hierarchical KV Caching)
|
||||
|
||||
K3's hybrid HiCache tiers the paged MLA KV **and** the KDA/mamba state across L1 (GPU) / L2 (host) / L3 (Mooncake) — enable it from the **HiCache** card in the [Playground above](#playground) for long multi-turn workloads.
|
||||
|
||||
- On the DCP recipes (Blackwell Balanced / High-Throughput), the host tiers are not fully DCP-aware yet: **L3 always, and L1+L2 with Spec Decode on, drop the DCP flags and run plain TP** (the command hints call it out — per-request KV capacity shrinks accordingly). L1+L2 NOSPEC keeps DCP.
|
||||
- Low-Latency and the Hopper recipes take all tiers unchanged.
|
||||
|
||||
### 3.4 PD Disaggregation
|
||||
|
||||
PD splits prefill and decode into separate server groups; because K3 is hybrid, the transfer moves **both** the paged MLA KV and the KDA recurrent state.
|
||||
|
||||
- **Transfer**: the cells emit **NiXL** (RDMA); Mooncake stays selectable in the Playground.
|
||||
- **Ports**: prefill `30000`, decode `30100` (derived ZMQ/dist ranges must not collide on a shared host). The positional `8998` after `--prefill` must match `--disaggregation-bootstrap-port`, or only the decode worker registers.
|
||||
- **Decode state pool**: chunk cache — one slot per request; `--mamba-radix-cache-strategy` is inert. Keep `--disaggregation-decode-extra-slots` pinned: unpinned it defaults to twice the batch below 32 requests and **zero** above.
|
||||
|
||||
#### Deep PP for long-context prefill
|
||||
|
||||
The `Long-Context` prefill strategy is `--pp-size 8 --tp-size 1`: pipeline P2P overlaps the next microbatch's compute, unlike TP/EP collectives, and each stage owns whole layers (a clean slice of KV and state).
|
||||
|
||||
- Use all eight stages; a shallow split still pays the in-stage all-reduce and can lose to TP8.
|
||||
- Pays only with several requests in flight (hence `Default` = TP8).
|
||||
- DSPARK off (`pp_size == 1` required).
|
||||
- Fan one prefill role out to several decode roles; budget for in-transfer KV on the decode side.
|
||||
|
||||
<Accordion title="Router">
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--prefill http://<prefill-host>:30000 8998 \
|
||||
--decode http://<decode-host>:30100 \
|
||||
--host 0.0.0.0 --port 8000 \
|
||||
--disable-circuit-breaker \
|
||||
--health-check-interval-secs 999999
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
Clients then send requests to the router (`:8000`) instead of an individual role server.
|
||||
|
||||
### 3.5 VLM Serving Profiles
|
||||
|
||||
The open-source K3 serving contract currently supports **image input only** — its
|
||||
processor rejects video and audio input.
|
||||
|
||||
#### Recommended high-speed VLM
|
||||
|
||||
The command panel now opens on the **B300 · Unified · Balanced**
|
||||
recipe below. It makes the VLM-specific performance choices explicit:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--trust-remote-code \
|
||||
--model-path moonshotai/Kimi-K3 \
|
||||
--tp-size 8 \
|
||||
--dcp-size 8 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--mm-feature-transport cuda_ipc \
|
||||
--mm-processor-worker-num 2 \
|
||||
--mm-io-worker-num 16 \
|
||||
--reasoning-parser kimi_k3 \
|
||||
--tool-call-parser kimi_k3 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
- `--mm-feature-transport cuda_ipc` — single-node only: skips the CPU round trip, bounded pool (per-tensor CPU fallback when full), reserves up to `SGLANG_MM_FEATURE_CACHE_MB` on the base GPU. Multi-node recipes use CPU transport.
|
||||
- 2 processor / 16 I/O workers are the measured defaults; more adds contention.
|
||||
- Leave `--mm-attention-backend` unset — auto-selected, with a correctness fallback.
|
||||
- Don't add `--mm-enable-dp-encoder`; K3 already shards images across TP ranks.
|
||||
|
||||
#### Should ViT BCG be enabled?
|
||||
|
||||
Keep ViT BCG **off** for general serving; enable `SGLANG_VIT_ENABLE_CUDA_GRAPH=1` only for ViT-only / EPD encoder workloads with recurring image shapes and spare HBM.
|
||||
|
||||
- The win is confined to the encoder — no reliable end-to-end TTFT/TPOT gain in full-model serving.
|
||||
- Each captured graph retains HBM (graph + per-entry metadata); measure on your own shapes.
|
||||
- The default cache captures after two hits and falls back to eager above 6,144 tokens; do not enlarge it without measuring.
|
||||
|
||||
#### Low-HBM VLM
|
||||
|
||||
Use this profile when keeping HBM headroom matters more than peak concurrency.
|
||||
It removes the 1 GiB CUDA IPC pool, keeps ViT BCG disabled, halves the context
|
||||
window, caps concurrency, and lowers the static-memory target:
|
||||
|
||||
```bash Command
|
||||
SGLANG_VIT_ENABLE_CUDA_GRAPH=0 \
|
||||
sglang serve \
|
||||
--trust-remote-code \
|
||||
--model-path moonshotai/Kimi-K3 \
|
||||
--tp-size 8 \
|
||||
--context-length 65536 \
|
||||
--enable-symm-mem \
|
||||
--mem-fraction-static 0.82 \
|
||||
--mm-feature-transport cpu \
|
||||
--mm-processor-worker-num 2 \
|
||||
--mm-io-worker-num 16 \
|
||||
--reasoning-parser kimi_k3 \
|
||||
--tool-call-parser kimi_k3 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
`--mem-fraction-static 0.82` is a conservative B300 starting point, not a portable minimum: raise it toward `0.85` if startup reports insufficient memory; if HBM must go back to other workloads, reduce context/concurrency first. The precision levers (`fp8_e4m3` KV, `bfloat16` SSM state) save far more but stay accuracy-gated.
|
||||
|
||||
<a id="large-scale-presets" />
|
||||
|
||||
### 3.6 Large-Scale Serving Presets (16–64 GPUs, Blackwell)
|
||||
|
||||
**The KDA state pool is the concurrency ceiling** — DP, EP, and DCP do not shard it; only attention-TP width, SSM dtype, and cache strategy change the per-GPU bill. The MLA KV is cheap to shrink (fp8) or deduplicate (DCP).
|
||||
|
||||
Two presets come out of this, at `N = 8k` GPUs:
|
||||
|
||||
| Preset | What it trades | Pick it for |
|
||||
|---|---|---|
|
||||
| **Peak Throughput** — `dp = k`, attention-TP 8 | State shards 8-way. The per-step KDA all-reduce stays within one 8-GPU B200/B300 node, or spans two 4-GPU GB200/GB300 nodes over MNNVL. `--kv-cache-dtype fp8_e4m3` is load-bearing — bf16 KV does not fit 128 requests per replica. | Maximum sustained TPS — the default large-scale shape. |
|
||||
| **Peak Capacity (+DCP8)** — `dp = k` + `--dcp-size 8` | Deduplicates the attention-TP group's MLA KV: concurrency ceiling +72% at the same engine throughput, ~1.8× ITL. | Context ≥ ~16K, or per-replica concurrency past 128. |
|
||||
|
||||
- **Radix cache** is independent of the preset: for prefix-free traffic (offline batch, evals) switch it off (Playground's **Prefix Cache** card) — one state slot per request instead of 4–5.
|
||||
- The fully data-parallel extreme (`--dp-size` = GPU count, attention-TP 1) — the shape behind the 64-GPU sweep's ~3K tok/s per GPU — is not a preset: 288 GB GPUs only, radix forced off, no head-to-head against the preset shape.
|
||||
|
||||
The Peak Throughput preset at 32 GPUs on B200/B300 (4 nodes × 8; every node runs the same command with its own `--node-rank`). On GB200/GB300 the same 32-GPU shape uses 8 nodes × 4, and the Playground emits `--nnodes 8`:
|
||||
|
||||
```bash Command
|
||||
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=20480 \
|
||||
sglang serve \
|
||||
--trust-remote-code \
|
||||
--model-path moonshotai/Kimi-K3 \
|
||||
--tp-size 32 --ep-size 32 \
|
||||
--enable-dp-attention --dp-size 4 --enable-dp-lm-head \
|
||||
--nnodes 4 --node-rank <rank> --dist-init-addr <node0-ip>:20000 \
|
||||
--moe-a2a-backend megamoe --moe-runner-backend deep_gemm \
|
||||
--kv-cache-dtype fp8_e4m3 \
|
||||
--mamba-ssm-dtype bfloat16 \
|
||||
--mamba-radix-cache-strategy extra_buffer_lazy \
|
||||
--mem-fraction-static 0.92 \
|
||||
--reasoning-parser kimi_k3 --tool-call-parser kimi_k3 \
|
||||
--host 0.0.0.0 --port 30000
|
||||
```
|
||||
|
||||
Scale by holding the per-replica shape fixed and moving only the replica count; pool sizing rides the calculator-driven `--mamba-full-memory-ratio`, which folds in DP, DCP, precision, and speculation:
|
||||
|
||||
| GPUs | B200/B300 nodes | GB200/GB300 nodes | `--tp-size` / `--ep-size` | `--dp-size` |
|
||||
|---|---|---|---|---|
|
||||
| 16 | 2×8 | 4×4 | 16 | 2 |
|
||||
| 32 | 4×8 | 8×4 | 32 | 4 |
|
||||
| 64 | 8×8 | 16×4 | 64 | 8 |
|
||||
|
||||
For Peak Capacity, add `--dcp-size 8` and re-derive the pool split with the [Mamba ratio calculator](#mamba-ratio-calculator).
|
||||
|
||||
Both presets are one click away in the [Playground above](#playground): pick a **Cluster Size** and a **Large-Scale Preset** and the full command composes onto whichever cell is showing.
|
||||
|
||||
Decisions the preset already makes:
|
||||
|
||||
- **MegaMoE on `deep_gemm`** — the fastest a2a backend; needs the SiTU cubin pool ([§2](#2-configuration-tips)).
|
||||
- **SP-MoE and shared-expert overlap** engage automatically under EP a2a; the K3 all-reduce fusion does not.
|
||||
- **Spec Decode follows the Deploy knob.** Acceptance thins at large batch; spec × EP × DP-attention is validated only at 8-GPU EP8 × DP2 (full GSM8K) — experimental at these scales.
|
||||
|
||||
<Note>
|
||||
No preset has a full serving round on final weights; the constants derive from measured single- and dual-node rounds plus a 64-GPU sweep. Validate throughput and accuracy on your workload before committing a fleet.
|
||||
</Note>
|
||||
@@ -70,7 +70,7 @@ metatags:
|
||||
<Card
|
||||
title="Moonshotai"
|
||||
mode="card"
|
||||
href="/cookbook/autoregressive/Moonshotai/Kimi-K2.7-Code"
|
||||
href="/cookbook/autoregressive/Moonshotai/Kimi-K3"
|
||||
img="/cards/logos/moonshotai.png"
|
||||
/>
|
||||
<Card
|
||||
|
||||
@@ -1060,6 +1060,7 @@
|
||||
{
|
||||
"group": "Moonshotai",
|
||||
"pages": [
|
||||
"cookbook/autoregressive/Moonshotai/Kimi-K3",
|
||||
"cookbook/autoregressive/Moonshotai/Kimi-K2.7-Code",
|
||||
"cookbook/autoregressive/Moonshotai/Kimi-K2.6",
|
||||
"cookbook/autoregressive/Moonshotai/Kimi-K2.5",
|
||||
|
||||
@@ -9,6 +9,185 @@ keywords:
|
||||
mode: wide
|
||||
---
|
||||
|
||||
<div className="not-prose">
|
||||
<div
|
||||
style={{
|
||||
position: "relative",
|
||||
overflow: "hidden",
|
||||
marginBottom: "1.5rem",
|
||||
padding: "clamp(1.5rem, 4vw, 2.5rem)",
|
||||
border: "1px solid rgba(251, 146, 60, 0.35)",
|
||||
borderRadius: "1rem",
|
||||
background:
|
||||
"linear-gradient(135deg, #111827 0%, #31202f 58%, #9a3412 100%)",
|
||||
boxShadow: "0 20px 45px rgba(17, 24, 39, 0.18)",
|
||||
color: "#ffffff",
|
||||
}}
|
||||
>
|
||||
<div
|
||||
aria-hidden="true"
|
||||
style={{
|
||||
position: "absolute",
|
||||
top: "-7rem",
|
||||
right: "-5rem",
|
||||
width: "18rem",
|
||||
height: "18rem",
|
||||
borderRadius: "999px",
|
||||
background: "rgba(251, 146, 60, 0.18)",
|
||||
filter: "blur(2px)",
|
||||
}}
|
||||
/>
|
||||
<div
|
||||
style={{
|
||||
position: "relative",
|
||||
zIndex: 1,
|
||||
display: "flex",
|
||||
flexWrap: "wrap",
|
||||
alignItems: "center",
|
||||
gap: "clamp(1.25rem, 3vw, 2rem)",
|
||||
}}
|
||||
>
|
||||
<div style={{ flex: "1 1 24rem", minWidth: 0 }}>
|
||||
<div
|
||||
style={{
|
||||
display: "inline-flex",
|
||||
alignItems: "center",
|
||||
gap: "0.45rem",
|
||||
marginBottom: "0.9rem",
|
||||
padding: "0.35rem 0.7rem",
|
||||
border: "1px solid rgba(255, 255, 255, 0.28)",
|
||||
borderRadius: "999px",
|
||||
background: "rgba(255, 255, 255, 0.1)",
|
||||
fontSize: "0.72rem",
|
||||
fontWeight: 750,
|
||||
letterSpacing: "0.08em",
|
||||
textTransform: "uppercase",
|
||||
}}
|
||||
>
|
||||
<span aria-hidden="true">✦</span>
|
||||
Featured model · New
|
||||
</div>
|
||||
<a
|
||||
href="/cookbook/autoregressive/Moonshotai/Kimi-K3"
|
||||
style={{
|
||||
display: "block",
|
||||
margin: 0,
|
||||
color: "#ffffff",
|
||||
fontSize: "clamp(1.75rem, 4vw, 2.65rem)",
|
||||
fontWeight: 750,
|
||||
lineHeight: 1.08,
|
||||
letterSpacing: "-0.035em",
|
||||
textDecoration: "none",
|
||||
}}
|
||||
>
|
||||
Meet Kimi-K3 on SGLang
|
||||
</a>
|
||||
<p
|
||||
style={{
|
||||
maxWidth: "48rem",
|
||||
margin: "1rem 0 0",
|
||||
color: "rgba(255, 255, 255, 0.82)",
|
||||
fontSize: "1rem",
|
||||
lineHeight: 1.65,
|
||||
}}
|
||||
>
|
||||
SGLang natively implements and deeply optimizes K3's new architecture
|
||||
with fused KDA decode kernels, DP attention, MTP, PD disaggregation,
|
||||
and KDA-aware prefix caching. Kimi-K3 is supported on both NVIDIA and
|
||||
AMD GPUs.
|
||||
</p>
|
||||
<div
|
||||
style={{
|
||||
display: "flex",
|
||||
flexWrap: "wrap",
|
||||
gap: "0.5rem",
|
||||
marginTop: "1.15rem",
|
||||
}}
|
||||
>
|
||||
{[
|
||||
"2.8T parameters",
|
||||
"Fused KDA decode",
|
||||
"NVIDIA + AMD",
|
||||
].map((item) => (
|
||||
<span
|
||||
key={item}
|
||||
style={{
|
||||
padding: "0.35rem 0.65rem",
|
||||
borderRadius: "999px",
|
||||
background: "rgba(255, 255, 255, 0.1)",
|
||||
color: "rgba(255, 255, 255, 0.9)",
|
||||
fontSize: "0.78rem",
|
||||
fontWeight: 650,
|
||||
}}
|
||||
>
|
||||
{item}
|
||||
</span>
|
||||
))}
|
||||
</div>
|
||||
<a
|
||||
href="/cookbook/autoregressive/Moonshotai/Kimi-K3"
|
||||
style={{
|
||||
display: "inline-flex",
|
||||
alignItems: "center",
|
||||
marginTop: "1.35rem",
|
||||
padding: "0.7rem 1rem",
|
||||
borderRadius: "0.55rem",
|
||||
background: "#ffffff",
|
||||
color: "#7c2d12",
|
||||
fontSize: "0.88rem",
|
||||
fontWeight: 750,
|
||||
textDecoration: "none",
|
||||
}}
|
||||
>
|
||||
Open the Kimi-K3 cookbook →
|
||||
</a>
|
||||
</div>
|
||||
<a
|
||||
href="/cookbook/autoregressive/Moonshotai/Kimi-K3"
|
||||
aria-label="Open the Kimi-K3 cookbook"
|
||||
style={{
|
||||
flex: "0 1 12rem",
|
||||
minWidth: "10rem",
|
||||
padding: "0.8rem",
|
||||
border: "1px solid rgba(255, 255, 255, 0.22)",
|
||||
borderRadius: "0.9rem",
|
||||
background: "rgba(255, 255, 255, 0.96)",
|
||||
boxShadow: "0 16px 35px rgba(0, 0, 0, 0.22)",
|
||||
textDecoration: "none",
|
||||
}}
|
||||
>
|
||||
<div
|
||||
role="img"
|
||||
aria-label="Moonshot AI"
|
||||
style={{
|
||||
width: "100%",
|
||||
aspectRatio: "16 / 9",
|
||||
borderRadius: "0.45rem",
|
||||
backgroundColor: "#ffffff",
|
||||
backgroundImage: "url('/cards/logos/moonshotai.png')",
|
||||
backgroundPosition: "center",
|
||||
backgroundRepeat: "no-repeat",
|
||||
backgroundSize: "cover",
|
||||
}}
|
||||
/>
|
||||
<div
|
||||
style={{
|
||||
padding: "0.65rem 0.35rem 0.2rem",
|
||||
color: "#111827",
|
||||
textAlign: "center",
|
||||
fontSize: "0.78rem",
|
||||
fontWeight: 750,
|
||||
letterSpacing: "0.06em",
|
||||
textTransform: "uppercase",
|
||||
}}
|
||||
>
|
||||
Kimi-K3 deployment guide
|
||||
</div>
|
||||
</a>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<a
|
||||
class="github-button"
|
||||
href="https://github.com/sgl-project/sglang"
|
||||
|
||||
Executable
+192
@@ -0,0 +1,192 @@
|
||||
#!/usr/bin/env node
|
||||
// Static guard for the cookbook deployment/playground engines and their configs.
|
||||
// Zero dependencies, no browser, no Mintlify — plain `node`.
|
||||
//
|
||||
// node docs_new/scripts/check_cookbook_configs.mjs
|
||||
//
|
||||
// What it protects, in order of how expensive the bug is to find by hand:
|
||||
//
|
||||
// 1. MIRROR drift. The overlay-resolution rule is written in both engines
|
||||
// because Mintlify snippets cannot import each other. If the copies drift,
|
||||
// the Deploy command and the playground's base disagree and the reader sees
|
||||
// phantom +/- lines in the diff — with no error anywhere.
|
||||
// 2. Sibling identity. Overlay resolution clones the base cell, so sibling
|
||||
// detection must compare match dimensions rather than object references.
|
||||
// 3. Config/engine contract. A cell keyed on a dimension the config no longer
|
||||
// declares silently stops matching; the panel just shows a different cell.
|
||||
// 4. Predicate safety. showWhen / disabled / flags run against selections the
|
||||
// author never clicked through; a throw there blanks the whole widget.
|
||||
|
||||
import { readFileSync, readdirSync } from "node:fs";
|
||||
import { dirname, join, relative } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const SNIPPETS = join(dirname(fileURLToPath(import.meta.url)), "..", "src", "snippets");
|
||||
const CONFIGS = join(SNIPPETS, "configs");
|
||||
const LEGACY_DIMS = ["variants", "quantizations", "strategies", "nodesOptions"];
|
||||
|
||||
const failures = [];
|
||||
const fail = (where, msg) => failures.push(`${where}: ${msg}`);
|
||||
|
||||
// ---------------------------------------------------------------- 1. MIRROR
|
||||
// Compare the marked blocks with comments and whitespace normalized away, so
|
||||
// wording may differ per file but the rule may not.
|
||||
const mirrorBody = (file) => {
|
||||
const src = readFileSync(join(SNIPPETS, file), "utf8");
|
||||
const start = src.indexOf("==== MIRROR");
|
||||
const end = src.indexOf("==== end MIRROR");
|
||||
if (start === -1 || end === -1) return null;
|
||||
return src
|
||||
.slice(src.indexOf("\n", start), end)
|
||||
.split("\n")
|
||||
.map((l) => l.trim())
|
||||
.filter((l) => l && !l.startsWith("//"))
|
||||
.join(" ")
|
||||
.replace(/\s+/g, " ");
|
||||
};
|
||||
|
||||
const a = mirrorBody("_deployment.jsx");
|
||||
const b = mirrorBody("_playground.jsx");
|
||||
if (a === null) fail("_deployment.jsx", "MIRROR markers missing");
|
||||
if (b === null) fail("_playground.jsx", "MIRROR markers missing");
|
||||
if (a && b && a !== b) {
|
||||
fail("MIRROR", "overlay resolution has drifted between the two engines");
|
||||
const [la, lb] = [a.split(" "), b.split(" ")];
|
||||
const i = la.findIndex((t, k) => t !== lb[k]);
|
||||
fail("MIRROR", `first divergence near token ${i}: `
|
||||
+ `_deployment "${la.slice(i, i + 8).join(" ")}" vs `
|
||||
+ `_playground "${lb.slice(i, i + 8).join(" ")}"`);
|
||||
}
|
||||
|
||||
// `withOverlay` returns a clone, so object identity can never distinguish the
|
||||
// current base cell from a true sibling. This previously made every cookbook
|
||||
// show a spurious "matches … / switch base" hint before the reader changed
|
||||
// anything.
|
||||
const playgroundSource = readFileSync(join(SNIPPETS, "_playground.jsx"), "utf8");
|
||||
if (/\bmatchedCell\s*!==\s*baseCell\b/.test(playgroundSource)) {
|
||||
fail("_playground.jsx", "sibling detection compares cloned cells by object identity");
|
||||
}
|
||||
|
||||
// --------------------------------------------------------------- 3/4. Configs
|
||||
// Configs are .jsx with a single `export const config` literal; import them
|
||||
// through a data: URL so no temp file is needed.
|
||||
const loadConfig = async (path) => {
|
||||
const src = readFileSync(path, "utf8");
|
||||
const mod = await import(
|
||||
"data:text/javascript," + encodeURIComponent(src)
|
||||
);
|
||||
return mod.config;
|
||||
};
|
||||
|
||||
// Every combination of match dims + overlay dims the reader can produce.
|
||||
const selectionSpace = (config) => {
|
||||
const dims = [
|
||||
{ id: "hw", options: (config.supportedHardware || []).map((id) => ({ id })) },
|
||||
...(config.matchDims || []),
|
||||
...(config.overlayDims || []),
|
||||
];
|
||||
let space = [{}];
|
||||
for (const d of dims) {
|
||||
const next = [];
|
||||
for (const partial of space) {
|
||||
for (const opt of (d.options || [])) next.push({ ...partial, [d.id]: opt.id });
|
||||
}
|
||||
space = next.length ? next : space;
|
||||
if (space.length > 20000) return space.slice(0, 20000); // cheap blow-up guard
|
||||
}
|
||||
return space;
|
||||
};
|
||||
|
||||
const walk = (dir) => readdirSync(dir, { withFileTypes: true }).flatMap((e) =>
|
||||
e.isDirectory() ? walk(join(dir, e.name))
|
||||
: (e.name.endsWith(".jsx") && !e.name.includes("benchmark") ? [join(dir, e.name)] : []));
|
||||
|
||||
for (const path of walk(CONFIGS)) {
|
||||
const where = relative(join(SNIPPETS, ".."), path);
|
||||
let config;
|
||||
try {
|
||||
config = await loadConfig(path);
|
||||
} catch (e) {
|
||||
fail(where, `does not parse as a module: ${e.message}`);
|
||||
continue;
|
||||
}
|
||||
if (!config) { fail(where, "no `export const config`"); continue; }
|
||||
|
||||
const custom = Array.isArray(config.matchDims);
|
||||
|
||||
// A config either declares its own dims or carries the full legacy set —
|
||||
// half of each means the engine silently renders a dimension nobody authored.
|
||||
if (!custom) {
|
||||
for (const k of LEGACY_DIMS) {
|
||||
if (!Array.isArray(config[k])) fail(where, `legacy config is missing \`${k}\``);
|
||||
}
|
||||
}
|
||||
|
||||
const matchIds = ["hw", ...(custom
|
||||
? config.matchDims.map((d) => d.id)
|
||||
: LEGACY_DIMS.map((k) => ({ variants: "variant", quantizations: "quant",
|
||||
strategies: "strategy", nodesOptions: "nodes" })[k]))];
|
||||
|
||||
for (const [i, cell] of (config.cells || []).entries()) {
|
||||
const keys = Object.keys(cell.match || {}).sort();
|
||||
const want = [...matchIds].sort();
|
||||
if (keys.join(",") !== want.join(",")) {
|
||||
fail(where, `cells[${i}].match keys [${keys}] != declared dims [${want}]`);
|
||||
}
|
||||
for (const dim of (config.matchDims || [])) {
|
||||
const v = cell.match[dim.id];
|
||||
if (!(dim.options || []).some((o) => o.id === v)) {
|
||||
fail(where, `cells[${i}].match.${dim.id}="${v}" is not an option of that dim`);
|
||||
}
|
||||
}
|
||||
// Without a `nodes` dim the node count rides on the cell; a missing one
|
||||
// silently degrades a multi-node recipe to single-node.
|
||||
if (custom && !matchIds.includes("nodes") && cell.nnodes === undefined) {
|
||||
fail(where, `cells[${i}] has no \`nnodes\` and the config declares no nodes dim`);
|
||||
}
|
||||
}
|
||||
|
||||
for (const dim of (config.overlayDims || [])) {
|
||||
const ids = (dim.options || []).map((o) => o.id);
|
||||
if (dim.default !== undefined && !ids.includes(dim.default)) {
|
||||
fail(where, `overlayDims.${dim.id}.default="${dim.default}" is not one of [${ids}]`);
|
||||
}
|
||||
}
|
||||
|
||||
// Predicates and flag builders must survive every reachable selection.
|
||||
const space = selectionSpace(config);
|
||||
const probe = (fn, label) => {
|
||||
for (const sel of space) {
|
||||
try { fn(sel); } catch (e) {
|
||||
fail(where, `${label} throws on ${JSON.stringify(sel)}: ${e.message}`);
|
||||
return;
|
||||
}
|
||||
}
|
||||
};
|
||||
for (const dim of [...(config.matchDims || []), ...(config.overlayDims || [])]) {
|
||||
if (typeof dim.showWhen === "function") probe(dim.showWhen, `${dim.id}.showWhen`);
|
||||
for (const opt of (dim.options || [])) {
|
||||
const tag = `${dim.id}.${opt.id}`;
|
||||
if (typeof opt.showWhen === "function") probe(opt.showWhen, `${tag}.showWhen`);
|
||||
if (typeof opt.disabled === "function") probe(opt.disabled, `${tag}.disabled`);
|
||||
for (const key of ["flags", "env", "hints"]) {
|
||||
if (typeof opt[key] !== "function") continue;
|
||||
probe((sel) => {
|
||||
const out = opt[key](sel);
|
||||
if (out !== undefined && !Array.isArray(out)) throw new Error(`${key} returned ${typeof out}, expected an array`);
|
||||
for (const f of (out || [])) {
|
||||
if (typeof f !== "string") throw new Error(`${key} yielded a non-string entry`);
|
||||
if (/undefined|NaN/.test(f)) throw new Error(`${key} produced "${f}"`);
|
||||
}
|
||||
}, `${tag}.${key}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (failures.length) {
|
||||
console.error(`FAIL (${failures.length})`);
|
||||
for (const f of failures) console.error(" - " + f);
|
||||
process.exit(1);
|
||||
}
|
||||
console.log("cookbook config check: OK");
|
||||
@@ -8,12 +8,29 @@
|
||||
// supportedHardware hw ids shown in the catalog (subset of HARDWARE_CATALOG ∪ config.hardware)
|
||||
// hardware optional — per-model GPUs the shared HARDWARE_CATALOG lacks:
|
||||
// {id, label, vram, vendor}[] merged into the catalog at render
|
||||
// (so a model-specific GPU never needs an engine-catalog edit)
|
||||
// variants/quantizations/strategies/nodesOptions the 5-dim option lists
|
||||
// (nodesOptions id is `single` or `multi-N` → --nnodes N)
|
||||
// cells {match, verified?, env, flags}[] — one per
|
||||
// (hw × variant × quant × strategy × nodes); env/flags are
|
||||
// flat literals, only {{PLACEHOLDER}} subst applied
|
||||
// (so a model-specific GPU never needs an engine-catalog edit);
|
||||
// vendor picks the selector group: blackwell | hopper | amd
|
||||
// variants/quantizations/strategies/nodesOptions LEGACY 4-dim option lists,
|
||||
// used when `matchDims` is absent (nodesOptions id is
|
||||
// `single` or `multi-N` → --nnodes N)
|
||||
// matchDims optional — replaces the legacy four. {id, title, options}[]
|
||||
// where each option is {id, label, showWhen?(sel), disabled?,
|
||||
// disableReason?}. `hw` is always the implicit first dim.
|
||||
// Cells are then keyed on (hw × <these ids>).
|
||||
// overlayDims optional — rows that do NOT participate in cell lookup; the
|
||||
// picked option layers onto the matched cell, so an orthogonal
|
||||
// knob does not multiply the cell count. Same option shape plus
|
||||
// `flags` / `env` / `hints` (each a literal array or a function
|
||||
// of the whole selection), and a row-level `default` / `showWhen`.
|
||||
// `hints` render as `# ...` lines above the command.
|
||||
// cells {match, verified?, nnodes?, warn?, redirect?, env, flags}[] — one per
|
||||
// (hw × match dims); env/flags are flat literals, only
|
||||
// {{PLACEHOLDER}} subst applied. `nnodes` supplies the node
|
||||
// count for configs with no `nodes` dim (default 1). `warn`
|
||||
// renders as a ⚠️ banner under the cell's command; it may
|
||||
// embed [label](#anchor) links. `redirect: true` renders the
|
||||
// banner ALONE — no command, header, or copy buttons — for
|
||||
// cells that only point somewhere else.
|
||||
// modelNames HF slug lookup, `hw|variant|quant` then `variant|quant`
|
||||
// placeholders {{KEY}} → {target: 'command'|'curl', label, default?}
|
||||
// curl cURL template (uses {{MODEL_NAME}} + placeholders)
|
||||
@@ -31,6 +48,8 @@
|
||||
// multiNodeHints optional — {[hwId]: string[]} prepended as `# ...` lines
|
||||
// dockerImages optional — `docker run` image, keyed by `hw|quant`
|
||||
// then `hw`; falls back to `lmsysorg/sglang:dev`
|
||||
// runModes optional — command output tabs to show (`python` and/or
|
||||
// `docker`); defaults to both, in that order
|
||||
// github optional — "Submit verified cell" issue-template overrides
|
||||
// playgroundFeatures optional — consumed by _playground.jsx (see its header)
|
||||
//
|
||||
@@ -49,13 +68,15 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
// ==== 1. Hardware catalog (shared across cookbooks) ====
|
||||
// VRAM is per-GPU on-chip memory, not per-module.
|
||||
const HARDWARE_CATALOG = {
|
||||
nvidia: [
|
||||
{ id: "h100", label: "H100", vram: "80GB" },
|
||||
{ id: "h200", label: "H200", vram: "141GB" },
|
||||
{ id: "b200", label: "B200", vram: "192GB" },
|
||||
blackwell: [
|
||||
{ id: "b300", label: "B300", vram: "288GB" },
|
||||
{ id: "gb200", label: "GB200", vram: "192GB" },
|
||||
{ id: "gb300", label: "GB300", vram: "288GB" },
|
||||
{ id: "b200", label: "B200", vram: "192GB" },
|
||||
{ id: "gb200", label: "GB200", vram: "192GB" },
|
||||
],
|
||||
hopper: [
|
||||
{ id: "h200", label: "H200", vram: "141GB" },
|
||||
{ id: "h100", label: "H100", vram: "80GB" },
|
||||
],
|
||||
amd: [
|
||||
{ id: "mi300x", label: "MI300X", vram: "192GB" },
|
||||
@@ -86,10 +107,12 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
},
|
||||
title: { fontSize: "12px", fontWeight: "600", minWidth: "108px", flexShrink: 0, color: isDark ? "#e5e7eb" : "inherit" },
|
||||
vendorRow: { display: "flex", alignItems: "center", gap: "6px" },
|
||||
// Fixed width so every row's chips start at the same x regardless of the
|
||||
// group name ("BLACKWELL" is the widest).
|
||||
vendorLabel: {
|
||||
fontSize: "10px", fontWeight: "600",
|
||||
color: isDark ? "#9ca3af" : "#6b7280",
|
||||
minWidth: "38px", textTransform: "uppercase", letterSpacing: "0.04em",
|
||||
width: "68px", flexShrink: 0, textTransform: "uppercase", letterSpacing: "0.04em",
|
||||
},
|
||||
// auto-fit + a real min width: columns wrap on narrow screens instead of
|
||||
// shrinking below their label (the old minmax(0,1fr) let buttons overlap on
|
||||
@@ -365,9 +388,72 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
});
|
||||
|
||||
// ==== 3. Pure helpers (no React state) ====
|
||||
// Two kinds of selector row:
|
||||
// match dims participate in cell lookup (cell.match[dim] === sel[dim])
|
||||
// overlay dims never touch cell lookup; the picked option contributes flags
|
||||
// on top of the matched cell (so an orthogonal knob like
|
||||
// speculative decoding does not multiply the cell count)
|
||||
// A config that declares neither keeps the legacy fixed 5-dim shape, so model
|
||||
// pages written before this existed render unchanged.
|
||||
const LEGACY_MATCH_DIMS = [
|
||||
{ id: "variant", title: "Model Variant", optionsKey: "variants" },
|
||||
{ id: "quant", title: "Quantization", optionsKey: "quantizations" },
|
||||
{ id: "strategy", title: "Strategy", optionsKey: "strategies" },
|
||||
{ id: "nodes", title: "Nodes", optionsKey: "nodesOptions" },
|
||||
];
|
||||
// `hw` is always the first match dim; it has its own vendor-grouped renderer.
|
||||
const matchDimSpecs = (config.matchDims || LEGACY_MATCH_DIMS).map((d) => ({
|
||||
...d,
|
||||
options: d.options || config[d.optionsKey] || [],
|
||||
}));
|
||||
const overlayDimSpecs = config.overlayDims || [];
|
||||
// DIMENSIONS is ordered by priority — higher-index dims adapt to lower-index
|
||||
// picks, never the reverse. Drives the grey-out/snap logic below.
|
||||
const DIMENSIONS = ["hw", "variant", "quant", "strategy", "nodes"];
|
||||
const DIMENSIONS = ["hw", ...matchDimSpecs.map((d) => d.id)];
|
||||
|
||||
// An option is visible when it declares no `showWhen`, or its predicate accepts
|
||||
// the current selection. Hidden options are excluded from snapping and from the
|
||||
// grey-out scan, so a stale pick can never survive a dependent-row switch.
|
||||
// ==== MIRROR in _playground.jsx — keep the two copies identical ====
|
||||
// Snippets cannot import each other, so the overlay-resolution rule is written
|
||||
// twice. A divergence makes the Deploy command and the playground base disagree,
|
||||
// which shows up as phantom +/- lines in the diff and no error anywhere.
|
||||
// Guarded by docs_new/scripts/check_cookbook_configs.mjs.
|
||||
const optionVisible = (opt, sel) =>
|
||||
typeof opt.showWhen !== "function" || opt.showWhen(sel);
|
||||
const optionDisabled = (opt, sel) =>
|
||||
typeof opt.disabled === "function" ? opt.disabled(sel) : !!opt.disabled;
|
||||
const visibleOptions = (spec, sel) =>
|
||||
(spec.options || []).filter((o) => optionVisible(o, sel));
|
||||
const rowVisible = (spec, sel) =>
|
||||
(typeof spec.showWhen !== "function" || spec.showWhen(sel)) &&
|
||||
visibleOptions(spec, sel).length > 0;
|
||||
const overlayPick = (sel) => {
|
||||
const picked = [];
|
||||
for (const spec of (config.overlayDims || [])) {
|
||||
if (!rowVisible(spec, sel)) continue;
|
||||
const opt = (spec.options || []).find((o) => o.id === sel[spec.id]);
|
||||
if (opt && !optionDisabled(opt, sel)) picked.push(opt);
|
||||
}
|
||||
return picked;
|
||||
};
|
||||
const overlayPart = (sel, key) => {
|
||||
const out = [];
|
||||
for (const opt of overlayPick(sel)) {
|
||||
const add = typeof opt[key] === "function" ? opt[key](sel) : opt[key];
|
||||
if (add) out.push(...add);
|
||||
}
|
||||
return out;
|
||||
};
|
||||
// An overlay option may also REMOVE cell flags, declared as `stripPrefixes`
|
||||
// (a static list, or a function of the selection). L3 uses it to drop the
|
||||
// whole DCP operating point, which the server rejects with an L3 backend.
|
||||
const overlayStrip = (cellFlags, sel) => {
|
||||
const strip = overlayPart(sel, "stripPrefixes");
|
||||
if (!strip.length) return [...(cellFlags || [])];
|
||||
return (cellFlags || []).filter((f) => !strip.includes(f.split(/[\s=]/)[0]));
|
||||
};
|
||||
// ==== end MIRROR ====
|
||||
const findCell = (cells, sel) =>
|
||||
cells.find((c) => DIMENSIONS.every((d) => c.match[d] === sel[d]));
|
||||
|
||||
@@ -461,37 +547,74 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
valid[dim] = fallback ? fallback.match[dim] : want;
|
||||
}
|
||||
}
|
||||
// Overlay dims ride along: they never key cells, so snapping must not drop
|
||||
// them (it did — a strict-mode hash round-trip lost the spec default).
|
||||
// Keep the parsed value when it names a real option, else the row default.
|
||||
for (const spec of overlayDimSpecs) {
|
||||
const want = parsed[spec.id];
|
||||
const opts = spec.options || [];
|
||||
valid[spec.id] = opts.some((o) => o.id === want)
|
||||
? want
|
||||
: spec.default ?? (opts[0] && opts[0].id) ?? "";
|
||||
}
|
||||
return valid;
|
||||
};
|
||||
|
||||
// Lookup walks most-specific to least so a config that drops the variant/quant
|
||||
// dims can key its HF slug on `hw` alone, or on the single "default" entry.
|
||||
const resolveModelName = (sel) => {
|
||||
const triple = `${sel.hw}|${sel.variant}|${sel.quant}`;
|
||||
const pair = `${sel.variant}|${sel.quant}`;
|
||||
return config.modelNames[triple] ?? config.modelNames[pair] ?? "";
|
||||
const keys = [
|
||||
`${sel.hw}|${sel.variant}|${sel.quant}`,
|
||||
`${sel.variant}|${sel.quant}`,
|
||||
sel.hw,
|
||||
"default",
|
||||
];
|
||||
for (const k of keys) {
|
||||
const hit = config.modelNames[k];
|
||||
if (hit) return hit;
|
||||
}
|
||||
return "";
|
||||
};
|
||||
|
||||
const interpolate = (text, env, modelName) =>
|
||||
text.replace(/{{(\w+)}}/g, (_, key) =>
|
||||
key === "MODEL_NAME" ? modelName : (env[key] ?? `{{${key}}}`));
|
||||
|
||||
// Node count comes from the `nodes` dim when the config has one; without that
|
||||
// dim it is a property of the cell itself (`nnodes`), since the deployment
|
||||
// shape is then fixed by the hardware rather than picked by the reader.
|
||||
const parseNnodes = (id) => {
|
||||
if (id === "single") return 1;
|
||||
const m = /^multi-(\d+)$/.exec(id);
|
||||
const m = /^multi-(\d+)$/.exec(id || "");
|
||||
return m ? parseInt(m[1], 10) : 1;
|
||||
};
|
||||
const cellNnodes = (cell, sel) =>
|
||||
sel.nodes !== undefined ? parseNnodes(sel.nodes) : (cell.nnodes || 1);
|
||||
|
||||
// Role-specific serving ports for PD deployments — keep in sync with PD_PORTS
|
||||
// in _playground.jsx, which the generated router command targets. Each role
|
||||
// derives 5 ZMQ/dist ports from its --port, so the serve ports are spaced 100
|
||||
// apart to keep those ranges from overlapping on a same-host deployment.
|
||||
const PD_SERVE_PORTS = { prefill: 30000, decode: 30100 };
|
||||
|
||||
// `flags` / `env` / `hints` may each be a function of the whole selection, so an
|
||||
// "Auto" option can resolve against another row (draft tokens per strategy).
|
||||
const overlayFlags = (sel) => overlayPart(sel, "flags");
|
||||
const overlayEnv = (sel) => overlayPart(sel, "env");
|
||||
const overlayHints = (sel) => overlayPart(sel, "hints");
|
||||
|
||||
// python mode → bare `sglang serve`; docker mode → wrapped in `docker run`.
|
||||
const renderCommand = (cell, sel, envValues, mode = "python") => {
|
||||
if (!cell) return "# No command available for the current selection.";
|
||||
const modelName = resolveModelName(sel);
|
||||
const nnodes = parseNnodes(sel.nodes);
|
||||
const nnodes = cellNnodes(cell, sel);
|
||||
const multinode = nnodes > 1;
|
||||
const cellEnv = cell.env || [];
|
||||
const flags = [...(cell.flags || [])];
|
||||
const cellEnv = [...(cell.env || []), ...overlayEnv(sel)];
|
||||
const flags = [...overlayStrip(cell.flags, sel), ...overlayFlags(sel)];
|
||||
if (multinode) {
|
||||
// Insert the multi-node trio after the last parallelism flag,
|
||||
// falling back to right after --model-path.
|
||||
const PARALLELISM_ANCHORS = ["--enable-dp-attention", "--dp", "--tp"];
|
||||
const PARALLELISM_ANCHORS = ["--enable-dp-attention", "--dp", "--tp-size", "--tp"];
|
||||
let i = -1;
|
||||
for (const anchor of PARALLELISM_ANCHORS) {
|
||||
i = flags.findIndex((f) => f.split(/[\s=]/)[0] === anchor);
|
||||
@@ -504,6 +627,15 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
`--dist-init-addr {{NODE0_IP}}:20000`);
|
||||
}
|
||||
|
||||
const pdServePort = PD_SERVE_PORTS[sel.pdMode];
|
||||
if (pdServePort !== undefined) {
|
||||
for (let j = 0; j < flags.length; j++) {
|
||||
if (flags[j].split(/[\s=]/)[0] === "--port") {
|
||||
flags[j] = `--port ${pdServePort}`;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let cmd;
|
||||
if (mode === "docker") {
|
||||
// Image keyed by `hw|quant` (most specific) then `hw`; `:dev` if unmapped.
|
||||
@@ -553,8 +685,14 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
cmd = `${envBlock}sglang serve \\\n${flagBlock}`;
|
||||
}
|
||||
|
||||
if (multinode && config.multiNodeHints && config.multiNodeHints[sel.hw]) {
|
||||
const hint = config.multiNodeHints[sel.hw]
|
||||
const hintLines = [
|
||||
...overlayHints(sel),
|
||||
...(multinode && config.multiNodeHints && config.multiNodeHints[sel.hw]
|
||||
? config.multiNodeHints[sel.hw]
|
||||
: []),
|
||||
];
|
||||
if (hintLines.length) {
|
||||
const hint = hintLines
|
||||
.map((line) => (line.length ? "# " + line : "#")).join("\n");
|
||||
cmd = `${hint}\n${cmd}`;
|
||||
}
|
||||
@@ -836,13 +974,19 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
return groups;
|
||||
};
|
||||
|
||||
// Match dims seed from the first cell (authoring convention: put the flagship
|
||||
// verified cell first). Overlay dims seed from their own `default`, or the
|
||||
// first option, since no cell carries them.
|
||||
const initialSelectionFromCells = () => {
|
||||
const first = config.cells[0];
|
||||
if (!first) return Object.fromEntries(DIMENSIONS.map((d) => [d, ""]));
|
||||
return {
|
||||
hw: first.match.hw, variant: first.match.variant, quant: first.match.quant,
|
||||
strategy: first.match.strategy, nodes: first.match.nodes,
|
||||
};
|
||||
const sel = Object.fromEntries(
|
||||
DIMENSIONS.map((d) => [d, first ? first.match[d] : ""]),
|
||||
);
|
||||
for (const spec of overlayDimSpecs) {
|
||||
const opts = spec.options || [];
|
||||
sel[spec.id] = spec.default ?? (opts[0] && opts[0].id) ?? "";
|
||||
}
|
||||
return sel;
|
||||
};
|
||||
|
||||
const placeholderDefaults = (schema) => {
|
||||
@@ -888,6 +1032,8 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
};
|
||||
|
||||
const [sel, setSel] = useState(() => initialSelectionFromCells());
|
||||
const INTERNAL_HASH_STATE_KEY = "__sglangDeployInternalHash";
|
||||
const DEPLOYMENT_COMPONENT_ID = "deployment-configurator";
|
||||
useEffect(() => {
|
||||
const hydrate = () => {
|
||||
const raw = window.location.hash.replace(/^#/, "");
|
||||
@@ -902,10 +1048,15 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
if (!touched) return;
|
||||
// Snap to a real cell if the hash named an impossible combo (stale link).
|
||||
setSel(validateSelection(config.cells, parsed));
|
||||
// Scroll the Deploy section into view. Heading slugs to "deployment" or
|
||||
// "deploy"; only fires on hash navigation (not replaceState chip clicks).
|
||||
const el = document.getElementById("deployment")
|
||||
|| document.getElementById("deploy");
|
||||
const historyState = window.history.state;
|
||||
const isInternalHash =
|
||||
historyState &&
|
||||
typeof historyState === "object" &&
|
||||
historyState[INTERNAL_HASH_STATE_KEY] === `#${raw}`;
|
||||
if (isInternalHash) return;
|
||||
// External selection hashes land on the interactive configurator. Hashes
|
||||
// written internally while initializing or changing chips do not scroll.
|
||||
const el = document.getElementById(DEPLOYMENT_COMPONENT_ID);
|
||||
if (el) el.scrollIntoView({ behavior: "smooth", block: "start" });
|
||||
};
|
||||
hydrate();
|
||||
@@ -917,7 +1068,15 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
useEffect(() => {
|
||||
const target = "#" + new URLSearchParams(sel).toString();
|
||||
if (window.location.hash !== target) {
|
||||
window.history.replaceState(null, "", target);
|
||||
const historyState =
|
||||
window.history.state && typeof window.history.state === "object"
|
||||
? window.history.state
|
||||
: {};
|
||||
window.history.replaceState(
|
||||
{ ...historyState, [INTERNAL_HASH_STATE_KEY]: target },
|
||||
"",
|
||||
target
|
||||
);
|
||||
}
|
||||
window.dispatchEvent(new CustomEvent("sglang-deploy-sel", { detail: sel }));
|
||||
}, [sel]);
|
||||
@@ -943,29 +1102,123 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
const [benchConc, setBenchConc] = useState(null);
|
||||
const [benchAcc, setBenchAcc] = useState(null);
|
||||
const [benchCopied, setBenchCopied] = useState(null);
|
||||
const [runMode, setRunMode] = useState("python"); // "python" | "docker"
|
||||
const runModes = config.runModes || ["python", "docker"];
|
||||
const [runMode, setRunMode] = useState(runModes[0]); // "python" | "docker"
|
||||
useEffect(() => { if (modal === "env") setEnvDraft(env); }, [modal, env]);
|
||||
|
||||
// Live --mamba-full-memory-ratio from the ratio calculator (K3 pages):
|
||||
// pool sizing is consolidated into this one flag, computed from the
|
||||
// calculator's request length plus the current panel selection.
|
||||
const [mambaRatio, setMambaRatio] = useState(null);
|
||||
useEffect(() => {
|
||||
// Deploy shows base flags only, so it takes the base-config ratio (the
|
||||
// effective one belongs to the playground's composed command).
|
||||
const onRatio = (e) =>
|
||||
setMambaRatio((e.detail && (e.detail.baseRatio || e.detail.ratio)) || null);
|
||||
window.addEventListener("sglang-k3-mamba-ratio", onRatio);
|
||||
return () => window.removeEventListener("sglang-k3-mamba-ratio", onRatio);
|
||||
}, []);
|
||||
|
||||
// ==== 5. Derived values ====
|
||||
const s = makeStyles(isDark);
|
||||
const cell = findCell(config.cells, sel);
|
||||
const command = renderCommand(cell, sel, env, runMode);
|
||||
// MTP hint: fire on the actual command (speculative decoding ON) — NOT on
|
||||
// strategy=low-latency, since a low-latency cell may not enable MTP. SGLang
|
||||
// resets --max-running-requests to 48 when spec is on and it's unset.
|
||||
// Pin the calculator-computed ratio into the rendered command (before the
|
||||
// host/port tail); cells themselves stay ratio-free.
|
||||
const cellWithRatio = (() => {
|
||||
if (!cell || !mambaRatio) return cell;
|
||||
if (cell.flags.some((f) => f.startsWith("--mamba-full-memory-ratio"))) return cell;
|
||||
const flags = [...cell.flags];
|
||||
const line = `--mamba-full-memory-ratio ${mambaRatio}`;
|
||||
const i = flags.findIndex((f) => f.startsWith("--host"));
|
||||
if (i >= 0) flags.splice(i, 0, line);
|
||||
else flags.push(line);
|
||||
return { ...cell, flags };
|
||||
})();
|
||||
const command = renderCommand(cellWithRatio, sel, env, runMode);
|
||||
// MTP hint on the EFFECTIVE flags — speculation arrives via the Spec Decode
|
||||
// overlay, never the cell. SGLang resets --max-running-requests to 48 when
|
||||
// spec is on and it's unset.
|
||||
const effFlags = cell ? [...overlayStrip(cell.flags, sel), ...overlayFlags(sel)] : [];
|
||||
const mtpHint =
|
||||
!!cell &&
|
||||
(cell.flags || []).some((f) => f.split(/[\s=]/)[0] === "--speculative-algorithm") &&
|
||||
!(cell.flags || []).some((f) => f.split(/[\s=]/)[0] === "--max-running-requests");
|
||||
effFlags.some((f) => f.split(/[\s=]/)[0] === "--speculative-algorithm") &&
|
||||
!effFlags.some((f) => f.split(/[\s=]/)[0] === "--max-running-requests");
|
||||
// cell.warn may embed [label](#anchor) links — rendered as scrollIntoView
|
||||
// buttons, not hrefs, so the hash (which carries the selection) isn't overwritten.
|
||||
const renderWarn = (text) => {
|
||||
const out = [];
|
||||
const re = /\[([^\]]+)\]\(#([^)]+)\)/g;
|
||||
let last = 0;
|
||||
for (let m; (m = re.exec(text)); last = m.index + m[0].length) {
|
||||
if (m.index > last) out.push(text.slice(last, m.index));
|
||||
const anchor = m[2];
|
||||
out.push(
|
||||
<button
|
||||
key={m.index}
|
||||
type="button"
|
||||
onClick={() => {
|
||||
const el = document.getElementById(anchor);
|
||||
if (el) el.scrollIntoView({ behavior: "smooth", block: "start" });
|
||||
}}
|
||||
style={{
|
||||
background: "transparent",
|
||||
border: "none",
|
||||
padding: 0,
|
||||
color: isDark ? "#FDBA74" : "#C2410C",
|
||||
cursor: "pointer",
|
||||
font: "inherit",
|
||||
fontWeight: 600,
|
||||
textDecoration: "underline",
|
||||
textUnderlineOffset: "2px",
|
||||
}}
|
||||
>
|
||||
{m[1]}
|
||||
</button>
|
||||
);
|
||||
}
|
||||
if (last < text.length) out.push(text.slice(last));
|
||||
return out;
|
||||
};
|
||||
const modelName = resolveModelName(sel);
|
||||
const curlText = interpolate(config.curl || "", env, modelName);
|
||||
const hwGroups = buildHardwareGroups();
|
||||
const benchEntry = benchmarks ? findBenchmark(benchmarks, sel) : null;
|
||||
|
||||
const isEnabled = (dim, value) => isOptionAvailable(config.cells, sel, dim, value);
|
||||
// Overlay dims have no cells to constrain them, so an option is selectable
|
||||
// unless it says otherwise; only match dims get the grey-out scan.
|
||||
const isOverlayDim = (dim) => overlayDimSpecs.some((d) => d.id === dim);
|
||||
const findOption = (dim, value) => {
|
||||
const spec = [...matchDimSpecs, ...overlayDimSpecs].find((d) => d.id === dim);
|
||||
return spec && (spec.options || []).find((o) => o.id === value);
|
||||
};
|
||||
const isEnabled = (dim, value) => {
|
||||
const opt = findOption(dim, value);
|
||||
if (opt && optionDisabled(opt, sel)) return false;
|
||||
return isOverlayDim(dim) || isOptionAvailable(config.cells, sel, dim, value);
|
||||
};
|
||||
|
||||
// Switching a match dim can hide the option a dependent row currently holds
|
||||
// (Strategy's option set differs per PD mode). Re-seat any overlay/match pick
|
||||
// that just became invisible onto the first visible option of its row.
|
||||
const reseatHiddenPicks = (next) => {
|
||||
let out = next;
|
||||
for (const spec of [...matchDimSpecs, ...overlayDimSpecs]) {
|
||||
const opts = visibleOptions(spec, out).filter((o) => !optionDisabled(o, out));
|
||||
if (!opts.length) continue;
|
||||
if (!opts.some((o) => o.id === out[spec.id])) {
|
||||
out = { ...out, [spec.id]: opts[0].id };
|
||||
}
|
||||
}
|
||||
return out;
|
||||
};
|
||||
|
||||
const handleSelect = (dim, value) => {
|
||||
setSel((prev) => snapToValidCell(config.cells, prev, dim, value));
|
||||
setSel((prev) =>
|
||||
reseatHiddenPicks(
|
||||
isOverlayDim(dim)
|
||||
? { ...prev, [dim]: value }
|
||||
: snapToValidCell(config.cells, prev, dim, value),
|
||||
),
|
||||
);
|
||||
};
|
||||
|
||||
const handleCopy = () => {
|
||||
@@ -1006,7 +1259,11 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
...(checked ? s.checked : {}),
|
||||
...(disabled ? s.disabled : {}),
|
||||
}}
|
||||
title={disabled ? "Not supported for current selection" : ""}
|
||||
title={
|
||||
disabled
|
||||
? item.disableReason || "Not supported for current selection"
|
||||
: ""
|
||||
}
|
||||
onClick={(e) => {
|
||||
if (disabled) { e.preventDefault(); return; }
|
||||
handleSelect(dim, item.id);
|
||||
@@ -1035,7 +1292,11 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
const maxHwCols = Math.max(...hwGroups.map((x) => x.items.length));
|
||||
|
||||
return (
|
||||
<div style={s.container} className="not-prose">
|
||||
<div
|
||||
id={DEPLOYMENT_COMPONENT_ID}
|
||||
style={{ ...s.container, scrollMarginTop: "104px" }}
|
||||
className="not-prose"
|
||||
>
|
||||
{/* Hardware section (2 vendor rows in one card, equal-width grid) */}
|
||||
<div style={s.cardColumn}>
|
||||
<div style={{ ...s.title, marginBottom: "2px" }}>Hardware Platform</div>
|
||||
@@ -1052,54 +1313,69 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
))}
|
||||
</div>
|
||||
|
||||
{renderFlatSection("Model Variant", config.variants, "variant", sel.variant)}
|
||||
{renderFlatSection("Quantization", config.quantizations, "quant", sel.quant)}
|
||||
{renderFlatSection("Strategy", config.strategies, "strategy", sel.strategy)}
|
||||
{renderFlatSection("Nodes", config.nodesOptions, "nodes", sel.nodes)}
|
||||
{matchDimSpecs
|
||||
.filter((d) => rowVisible(d, sel))
|
||||
.map((d) => (
|
||||
<div key={d.id}>
|
||||
{renderFlatSection(d.title, visibleOptions(d, sel), d.id, sel[d.id])}
|
||||
</div>
|
||||
))}
|
||||
{overlayDimSpecs
|
||||
.filter((d) => rowVisible(d, sel))
|
||||
.map((d) => (
|
||||
<div key={d.id}>
|
||||
{renderFlatSection(d.title, visibleOptions(d, sel), d.id, sel[d.id])}
|
||||
</div>
|
||||
))}
|
||||
|
||||
{/* Command box */}
|
||||
<div style={s.card}>
|
||||
<div style={s.title}>Run this Command:</div>
|
||||
<div style={s.title}>Command:</div>
|
||||
<div style={s.commandWrap}>
|
||||
<div style={s.commandHeader}>
|
||||
<div style={s.headerLeft}>
|
||||
<div style={s.badge(Boolean(cell && cell.verified))}>
|
||||
<span style={s.badgeDot(Boolean(cell && cell.verified))} />
|
||||
{cell && cell.verified ? "Verified" : "Not Verified"}
|
||||
{cell && cell.redirect ? (
|
||||
cell.warn && <div style={s.mtpWarn}>⚠️ {renderWarn(cell.warn)}</div>
|
||||
) : (<>
|
||||
<div style={s.commandHeader}>
|
||||
<div style={s.headerLeft}>
|
||||
<div style={s.badge(Boolean(cell && cell.verified))}>
|
||||
<span style={s.badgeDot(Boolean(cell && cell.verified))} />
|
||||
{cell && cell.verified ? "Verified" : "Not Verified"}
|
||||
</div>
|
||||
<div style={s.runModeWrap} role="tablist" aria-label="Output format">
|
||||
{runModes.map((mode, index) => (
|
||||
<span
|
||||
key={mode}
|
||||
style={{
|
||||
...(index === runModes.length - 1
|
||||
? s.runModeChipLast(runMode === mode)
|
||||
: s.runModeChip(runMode === mode)),
|
||||
...(runModes.length === 1 ? { borderRadius: 7 } : {}),
|
||||
}}
|
||||
onClick={() => setRunMode(mode)}
|
||||
role="tab"
|
||||
aria-selected={runMode === mode}
|
||||
>
|
||||
{mode === "docker" ? "Docker" : "Python"}
|
||||
</span>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
<div style={s.runModeWrap} role="tablist" aria-label="Output format">
|
||||
<span
|
||||
style={s.runModeChip(runMode === "python")}
|
||||
onClick={() => setRunMode("python")}
|
||||
role="tab"
|
||||
aria-selected={runMode === "python"}
|
||||
>
|
||||
Python
|
||||
</span>
|
||||
<span
|
||||
style={s.runModeChipLast(runMode === "docker")}
|
||||
onClick={() => setRunMode("docker")}
|
||||
role="tab"
|
||||
aria-selected={runMode === "docker"}
|
||||
>
|
||||
Docker
|
||||
</span>
|
||||
<div style={s.iconRow}>
|
||||
<button style={s.iconButton} onClick={handleCopy}>
|
||||
{copied ? "✓ Copied" : "⧉ Copy"}
|
||||
</button>
|
||||
<button style={s.iconButton} onClick={() => setModal("curl")}>$ cURL</button>
|
||||
<button style={s.iconButton} onClick={() => setModal("env")}>⚙ Env</button>
|
||||
</div>
|
||||
</div>
|
||||
<div style={s.iconRow}>
|
||||
<button style={s.iconButton} onClick={handleCopy}>
|
||||
{copied ? "✓ Copied" : "⧉ Copy"}
|
||||
</button>
|
||||
<button style={s.iconButton} onClick={() => setModal("curl")}>$ cURL</button>
|
||||
<button style={s.iconButton} onClick={() => setModal("env")}>⚙ Env</button>
|
||||
</div>
|
||||
</div>
|
||||
<pre style={s.commandPre}>{command}</pre>
|
||||
{mtpHint && (
|
||||
<div style={s.mtpWarn}>
|
||||
⚠️ Speculative decoding (MTP) is on — SGLang resets <code>--max-running-requests</code> to <strong>48</strong> when it isn't set. Add <code>--max-running-requests <N></code> sized for your target concurrency.
|
||||
</div>
|
||||
)}
|
||||
<pre style={s.commandPre}>{command}</pre>
|
||||
{cell && cell.warn && <div style={s.mtpWarn}>⚠️ {renderWarn(cell.warn)}</div>}
|
||||
{mtpHint && (
|
||||
<div style={s.mtpWarn}>
|
||||
⚠️ Speculative decoding (MTP) is on — SGLang resets <code>--max-running-requests</code> to <strong>48</strong> when it isn't set. Add <code>--max-running-requests <N></code> sized for your target concurrency.
|
||||
</div>
|
||||
)}
|
||||
</>)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -0,0 +1,331 @@
|
||||
// Kimi-K3-only calculator, live-coupled to the Deploy panel: every parameter
|
||||
// except the average request length is derived from the effective config the
|
||||
// Playground broadcasts (base cell + Deploy overlays + Playground overrides),
|
||||
// and the computed --mamba-full-memory-ratio is broadcast back for the Deploy
|
||||
// panel to pin into its command.
|
||||
|
||||
export const KimiK3MambaRatioCalculator = () => {
|
||||
const [isDark, setIsDark] = useState(false);
|
||||
const [requestLength, setRequestLength] = useState("11264");
|
||||
const [copied, setCopied] = useState(false);
|
||||
// Effective serving config; empty until the Playground's first broadcast
|
||||
// (the parse below then falls back to the stock defaults: tp8, bf16 KV,
|
||||
// fp32 state, extra_buffer, NOSPEC, no DCP).
|
||||
const [cfg, setCfg] = useState({ flags: [], env: [], baseFlags: [], baseEnv: [] });
|
||||
|
||||
useEffect(() => {
|
||||
const checkTheme = () => {
|
||||
const html = document.documentElement;
|
||||
setIsDark(
|
||||
html.classList.contains("dark") ||
|
||||
html.getAttribute("data-theme") === "dark" ||
|
||||
html.style.colorScheme === "dark"
|
||||
);
|
||||
};
|
||||
checkTheme();
|
||||
const observer = new MutationObserver(checkTheme);
|
||||
observer.observe(document.documentElement, {
|
||||
attributes: true,
|
||||
attributeFilter: ["class", "data-theme", "style"],
|
||||
});
|
||||
return () => observer.disconnect();
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
const onCfg = (e) =>
|
||||
setCfg({
|
||||
flags: (e.detail && e.detail.flags) || [],
|
||||
env: (e.detail && e.detail.env) || [],
|
||||
baseFlags: (e.detail && e.detail.baseFlags) || [],
|
||||
baseEnv: (e.detail && e.detail.baseEnv) || [],
|
||||
});
|
||||
window.addEventListener("sglang-k3-effective-config", onCfg);
|
||||
return () => window.removeEventListener("sglang-k3-effective-config", onCfg);
|
||||
}, []);
|
||||
|
||||
const length = Number.parseFloat(requestLength);
|
||||
|
||||
// Derive the serving parameters from a flag/env list and evaluate the balance
|
||||
// formula (measured dual-pool balance), written as a per-request cost ratio:
|
||||
//
|
||||
// r = (S + D) x state_bytes / (L x per_token_kv_bytes)
|
||||
//
|
||||
// The state side (S main slots plus D verify intermediates) is per-GPU and
|
||||
// never DCP-sharded. The KV side is per-GPU per logical token: DCP shards the
|
||||
// MLA latent KV across its ranks, while the DSPARK draft model's own KV is
|
||||
// replicated on every rank, so it stays a flat term. Without DCP the draft
|
||||
// term is ~10% noise; once DCP shards MLA it is the same order as the MLA
|
||||
// share, which is why it cannot be folded into a plain x dcp factor.
|
||||
const derive = (flags, env) => {
|
||||
const flagArg = (name) => {
|
||||
for (const f of flags) {
|
||||
const parts = f.split(/\s+/);
|
||||
if (parts[0] === name) return parts[1];
|
||||
}
|
||||
return null;
|
||||
};
|
||||
const hasFlag = (name) => flags.some((f) => f.split(/[\s=]/)[0] === name);
|
||||
|
||||
const tp = Number(flagArg("--tp-size")) || 8;
|
||||
const dp = hasFlag("--enable-dp-attention") ? Number(flagArg("--dp-size")) || 1 : 1;
|
||||
// KDA state and the replicated MLA KV both live per attention-TP group.
|
||||
// PP needs no term: it splits the layers of both pools equally.
|
||||
const attnTp = Math.max(1, Math.round(tp / dp));
|
||||
const dcp = Number(flagArg("--dcp-size")) || 1;
|
||||
const kvDtype = flagArg("--kv-cache-dtype") === "fp8_e4m3" ? "fp8_e4m3" : "bfloat16";
|
||||
const specOn = hasFlag("--speculative-algorithm");
|
||||
const replaySpec = hasFlag("--enable-linear-replayssm-spec");
|
||||
// ReplaySSM removes the D intermediate states but does NOT pin the state
|
||||
// dtype: an unset --mamba-ssm-dtype defaults to fp32 either way, and an
|
||||
// explicit 16-bit state is accepted (warned for drift), so read the flag.
|
||||
const ssmFlag = flagArg("--mamba-ssm-dtype");
|
||||
const ssmDtype =
|
||||
ssmFlag === "bfloat16" || ssmFlag === "float16" ? ssmFlag : "float32";
|
||||
const radixOff = hasFlag("--disable-radix-cache");
|
||||
// "auto" (and anything unrecognized) resolves to extra_buffer.
|
||||
const strategyFlag = flagArg("--mamba-radix-cache-strategy");
|
||||
const strategy =
|
||||
strategyFlag === "no_buffer" || strategyFlag === "extra_buffer_lazy"
|
||||
? strategyFlag
|
||||
: "extra_buffer";
|
||||
const skipLock = env.some((e) => e.startsWith("SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1"));
|
||||
const pdRole = flagArg("--disaggregation-mode");
|
||||
// Pipeline parallelism is incompatible with the overlap scheduler, so pp > 1
|
||||
// turns it off for you — and the track buffer then costs one slot, not two.
|
||||
const overlapOff =
|
||||
hasFlag("--disable-overlap-schedule") || (Number(flagArg("--pp-size")) || 1) > 1;
|
||||
// Mirrors kv_cache_configurator._calculate_mamba_ratio: base 3, minus 1 under
|
||||
// the decode-lock skip, plus the ping-pong track buffer (2 under the overlap
|
||||
// scheduler, 1 for lazy or without overlap). no_buffer has no track buffer and
|
||||
// adds the skip's drop back, so it stays 3; a disabled radix cache is 1.
|
||||
// A PD decode server runs a chunk cache: one live slot per request, and the
|
||||
// radix-strategy knobs are inert there.
|
||||
const slots = pdRole === "decode"
|
||||
? 1
|
||||
: radixOff
|
||||
? 1
|
||||
: strategy === "no_buffer"
|
||||
? 3
|
||||
: 3 -
|
||||
(skipLock ? 1 : 0) +
|
||||
(overlapOff || strategy === "extra_buffer_lazy" ? 1 : 2);
|
||||
const block = Number(flagArg("--speculative-dspark-block-size")) || 7;
|
||||
const drafts = specOn && !replaySpec && pdRole !== "prefill" ? block + 1 : 0;
|
||||
|
||||
const ssmBytes = ssmDtype === "float32" ? 4 : 2;
|
||||
const kvBytes = kvDtype === "fp8_e4m3" ? 1 : 2;
|
||||
// Fixed K3 geometry:
|
||||
// KDA: 69 layers, 96 heads, head_dim 128, conv kernel 4 (conv state always bf16).
|
||||
// MLA: 24 layers, kv_lora_rank 512, qk_rope_head_dim 64.
|
||||
const stateBytesPerSlot =
|
||||
69 * ((96 / attnTp) * 128 * 128 * ssmBytes + 3 * 3 * (96 / attnTp) * 128 * 2);
|
||||
const kvBytesPerToken = 24 * (512 + 64) * kvBytes;
|
||||
// DCP shards the MLA latent KV across its ranks; the DSPARK draft model's KV
|
||||
// is replicated on every rank (~1.4 KB/token on trtllm_mha), so it does not
|
||||
// shard and is added flat.
|
||||
const draftKvBytesPerToken = specOn ? 1400 : 0;
|
||||
const kvBytesPerTokenPerRank = kvBytesPerToken / dcp + draftKvBytesPerToken;
|
||||
const ratio =
|
||||
((slots + drafts) * stateBytesPerSlot) / (kvBytesPerTokenPerRank * length);
|
||||
return { ratio, tp, dp, attnTp, dcp, kvDtype, ssmDtype, radixOff, strategy, skipLock, slots, specOn, replaySpec, block, pdRole };
|
||||
};
|
||||
|
||||
// Two evaluations: `eff` matches the Playground's composed command, `bs`
|
||||
// matches the Deploy command (cell + overlays only).
|
||||
const eff = derive(cfg.flags, cfg.env);
|
||||
const bs = derive(cfg.baseFlags.length ? cfg.baseFlags : cfg.flags,
|
||||
cfg.baseFlags.length ? cfg.baseEnv : cfg.env);
|
||||
const { ratio, tp, dp, attnTp, dcp, kvDtype, ssmDtype, radixOff, strategy, skipLock, slots, specOn, replaySpec, block, pdRole } = eff;
|
||||
const valid = Number.isFinite(ratio) && ratio > 0 && length > 0 && 96 % attnTp === 0;
|
||||
const baseValid = Number.isFinite(bs.ratio) && bs.ratio > 0 && length > 0;
|
||||
|
||||
const formatRatio = (value) => {
|
||||
if (!Number.isFinite(value)) return "—";
|
||||
if (value >= 10) return value.toFixed(1).replace(/\.0$/, "");
|
||||
if (value >= 1) return value.toFixed(2).replace(/\.?0+$/, "");
|
||||
return Number(value.toPrecision(2)).toString();
|
||||
};
|
||||
|
||||
const result = valid ? formatRatio(ratio) : "—";
|
||||
const baseResult = baseValid ? formatRatio(bs.ratio) : "—";
|
||||
const cliFlag = valid ? `--mamba-full-memory-ratio ${result}` : "";
|
||||
|
||||
// Broadcast both results: the Deploy command takes the base-config value,
|
||||
// the Playground's composed command takes the effective one.
|
||||
useEffect(() => {
|
||||
window.dispatchEvent(
|
||||
new CustomEvent("sglang-k3-mamba-ratio", {
|
||||
detail: {
|
||||
ratio: valid ? result : null,
|
||||
baseRatio: baseValid ? baseResult : null,
|
||||
},
|
||||
})
|
||||
);
|
||||
}, [result, valid, baseResult, baseValid]);
|
||||
|
||||
const copyFlag = () => {
|
||||
if (!cliFlag || typeof navigator === "undefined" || !navigator.clipboard) return;
|
||||
navigator.clipboard.writeText(cliFlag);
|
||||
setCopied(true);
|
||||
window.setTimeout(() => setCopied(false), 1600);
|
||||
};
|
||||
|
||||
const colors = {
|
||||
border: isDark ? "#374151" : "#e5e7eb",
|
||||
panel: isDark ? "#1f2937" : "#ffffff",
|
||||
input: isDark ? "#111827" : "#f8fafc",
|
||||
text: isDark ? "#e5e7eb" : "#1f2937",
|
||||
muted: isDark ? "#9ca3af" : "#64748b",
|
||||
accent: isDark ? "#E85D4D" : "#D45D44",
|
||||
error: isDark ? "#fca5a5" : "#b91c1c",
|
||||
};
|
||||
|
||||
const inputStyle = {
|
||||
width: "100%",
|
||||
boxSizing: "border-box",
|
||||
padding: "8px 10px",
|
||||
border: `1px solid ${colors.border}`,
|
||||
borderRadius: "5px",
|
||||
background: colors.input,
|
||||
color: colors.text,
|
||||
fontSize: "13px",
|
||||
};
|
||||
const labelStyle = {
|
||||
display: "flex",
|
||||
flexDirection: "column",
|
||||
gap: "5px",
|
||||
fontSize: "12px",
|
||||
fontWeight: 600,
|
||||
};
|
||||
const chipStyle = {
|
||||
padding: "3px 9px",
|
||||
border: `1px solid ${colors.border}`,
|
||||
borderRadius: "999px",
|
||||
background: colors.input,
|
||||
color: colors.text,
|
||||
fontSize: "12px",
|
||||
whiteSpace: "nowrap",
|
||||
};
|
||||
|
||||
const specLabel = !specOn
|
||||
? "NOSPEC"
|
||||
: replaySpec
|
||||
? "DSPARK + ReplaySSM (D folded)"
|
||||
: `DSPARK (D = ${block + 1})`;
|
||||
const derivedChips = [
|
||||
// With DP attention on, show the whole topology so a large-scale preset is
|
||||
// visibly understood: total GPUs = DP replicas x attention-TP group width.
|
||||
dp > 1
|
||||
? `${tp} GPUs = DP ${dp} × Attention TP ${attnTp}`
|
||||
: `Attention TP ${attnTp}`,
|
||||
`DCP ${dcp}`,
|
||||
`KV ${kvDtype === "fp8_e4m3" ? "FP8" : "BF16"}`,
|
||||
`State ${ssmDtype === "float32" ? "FP32" : ssmDtype === "bfloat16" ? "BF16" : "FP16"}`,
|
||||
pdRole === "decode"
|
||||
? "PD decode: chunk cache (S = 1)"
|
||||
: radixOff
|
||||
? "Radix off (S = 1)"
|
||||
: `${strategy}${skipLock ? " + slot saving" : ""} (S = ${slots})`,
|
||||
pdRole === "prefill" ? "PD prefill (no verify states)" : null,
|
||||
specLabel,
|
||||
].filter(Boolean);
|
||||
|
||||
return (
|
||||
<div
|
||||
className="not-prose"
|
||||
style={{
|
||||
display: "grid",
|
||||
gap: "12px",
|
||||
padding: "14px",
|
||||
border: `1px solid ${colors.border}`,
|
||||
borderRadius: "8px",
|
||||
background: colors.panel,
|
||||
color: colors.text,
|
||||
}}
|
||||
>
|
||||
<div
|
||||
style={{
|
||||
display: "grid",
|
||||
gridTemplateColumns: "minmax(180px, 260px) 1fr",
|
||||
gap: "14px",
|
||||
alignItems: "start",
|
||||
}}
|
||||
>
|
||||
<label htmlFor="k3-ratio-length" style={labelStyle}>
|
||||
Average request length
|
||||
<input
|
||||
id="k3-ratio-length"
|
||||
type="number"
|
||||
min="1"
|
||||
step="1"
|
||||
value={requestLength}
|
||||
onChange={(event) => setRequestLength(event.target.value)}
|
||||
style={inputStyle}
|
||||
/>
|
||||
<span style={{ color: colors.muted, fontSize: "11px", fontWeight: 400 }}>
|
||||
Input + output tokens — the only free parameter
|
||||
</span>
|
||||
</label>
|
||||
|
||||
<div style={{ display: "flex", flexDirection: "column", gap: "6px" }}>
|
||||
<span style={{ fontSize: "12px", fontWeight: 600 }}>
|
||||
Serving configuration (follows the Deploy panel and Playground)
|
||||
</span>
|
||||
<div style={{ display: "flex", flexWrap: "wrap", gap: "6px" }}>
|
||||
{derivedChips.map((c) => (
|
||||
<span key={c} style={chipStyle}>{c}</span>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{!valid ? (
|
||||
<div style={{ color: colors.error, fontSize: "12px" }}>
|
||||
Enter a valid request length.
|
||||
</div>
|
||||
) : (
|
||||
<div
|
||||
style={{
|
||||
display: "flex",
|
||||
alignItems: "center",
|
||||
gap: "12px",
|
||||
paddingTop: "12px",
|
||||
borderTop: `1px solid ${colors.border}`,
|
||||
flexWrap: "wrap",
|
||||
}}
|
||||
>
|
||||
<div>
|
||||
<div style={{ color: colors.muted, fontSize: "11px" }}>
|
||||
Balanced ratio — pinned into the commands above
|
||||
</div>
|
||||
<div style={{ fontSize: "26px", fontWeight: 700 }}>{result}</div>
|
||||
{baseResult !== result ? (
|
||||
<div style={{ color: colors.muted, fontSize: "11px" }}>
|
||||
Deploy command (without Playground overrides): {baseResult}
|
||||
</div>
|
||||
) : null}
|
||||
</div>
|
||||
<code style={{ flex: 1, minWidth: "240px", color: colors.text }}>
|
||||
{cliFlag}
|
||||
</code>
|
||||
<button
|
||||
type="button"
|
||||
onClick={copyFlag}
|
||||
style={{
|
||||
padding: "7px 11px",
|
||||
border: 0,
|
||||
borderRadius: "5px",
|
||||
background: colors.accent,
|
||||
color: "#ffffff",
|
||||
fontSize: "12px",
|
||||
fontWeight: 600,
|
||||
cursor: "pointer",
|
||||
}}
|
||||
>
|
||||
{copied ? "Copied" : "Copy flag"}
|
||||
</button>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
};
|
||||
@@ -10,6 +10,17 @@
|
||||
// moe — backend (+ MegaMoE quantization sub-select) + EP
|
||||
// parsers — per-item toggle flags
|
||||
// speculative — single-select preset
|
||||
//
|
||||
// Axis-level `showWhen(base)` (any axis): the card is not rendered when the Deploy
|
||||
// panel has not switched that feature on. `base` carries the cell match dims plus
|
||||
// the Deploy panel's overlay dims, so an axis can gate on either.
|
||||
// flagSelects extras: `control: "slider"` renders the option list as a range input
|
||||
// (option order is the scale), and a per-select `showWhen(base)` gates one row.
|
||||
// A row may set `default` (initial pick instead of inherit) and options may carry
|
||||
// `env` (stripped/re-added like the MoE backend card) or `flags` as a FUNCTION
|
||||
// `(rowValues, base) => [...]` for cross-row composition (e.g. preset x cluster
|
||||
// size); a function returning null means "leave the base untouched".
|
||||
// Changing the Deploy selection resets every axis back to inherit-from-base.
|
||||
// pdDisagg — role + transfer backend + IB device + optional router
|
||||
// hicache — enable + backend + write policy
|
||||
// hisparse — enable + host ratio (decode-only)
|
||||
@@ -17,7 +28,7 @@
|
||||
// its own title + strip-prefixes + options (no per-feature code)
|
||||
//
|
||||
// Adding an axis = one entry in AXIS_HANDLERS below; nothing else switches on
|
||||
// an axis id. Each handler implements initState / revertHidden / apply /
|
||||
// an axis id. Each handler implements initState / apply /
|
||||
// render, plus optional deriveFromBase (recover state from base cell flags)
|
||||
// and getRenderHints.
|
||||
//
|
||||
@@ -34,7 +45,60 @@ export const Playground = ({ config }) => {
|
||||
// ==========================================================================
|
||||
// 1. Constants
|
||||
// ==========================================================================
|
||||
const DIMENSIONS = ["hw", "variant", "quant", "strategy", "nodes"];
|
||||
const DIMENSIONS = ["hw", ...((config.matchDims
|
||||
|| [{ id: "variant" }, { id: "quant" }, { id: "strategy" }, { id: "nodes" }])
|
||||
.map((d) => d.id))];
|
||||
|
||||
// The Deploy panel layers its overlay dims (speculation, hicache, ...) on top of
|
||||
// the matched cell, so the playground's "base" is cell + overlay — otherwise the
|
||||
// diff shows overlay flags as playground additions and deriveFromBase can't see
|
||||
// what Deploy already resolved.
|
||||
//
|
||||
// ==== MIRROR in _deployment.jsx — keep the two copies identical ====
|
||||
// Snippets cannot import each other, so the overlay-resolution rule is written
|
||||
// twice. A divergence makes the Deploy command and the playground base disagree,
|
||||
// which shows up as phantom +/- lines in the diff and no error anywhere.
|
||||
// Guarded by docs_new/scripts/check_cookbook_configs.mjs.
|
||||
const optionVisible = (opt, sel) =>
|
||||
typeof opt.showWhen !== "function" || opt.showWhen(sel);
|
||||
const optionDisabled = (opt, sel) =>
|
||||
typeof opt.disabled === "function" ? opt.disabled(sel) : !!opt.disabled;
|
||||
const visibleOptions = (spec, sel) =>
|
||||
(spec.options || []).filter((o) => optionVisible(o, sel));
|
||||
const rowVisible = (spec, sel) =>
|
||||
(typeof spec.showWhen !== "function" || spec.showWhen(sel)) &&
|
||||
visibleOptions(spec, sel).length > 0;
|
||||
const overlayPick = (sel) => {
|
||||
const picked = [];
|
||||
for (const spec of (config.overlayDims || [])) {
|
||||
if (!rowVisible(spec, sel)) continue;
|
||||
const opt = (spec.options || []).find((o) => o.id === sel[spec.id]);
|
||||
if (opt && !optionDisabled(opt, sel)) picked.push(opt);
|
||||
}
|
||||
return picked;
|
||||
};
|
||||
const overlayPart = (sel, key) => {
|
||||
const out = [];
|
||||
for (const opt of overlayPick(sel)) {
|
||||
const add = typeof opt[key] === "function" ? opt[key](sel) : opt[key];
|
||||
if (add) out.push(...add);
|
||||
}
|
||||
return out;
|
||||
};
|
||||
// An overlay option may also REMOVE cell flags, declared as `stripPrefixes`
|
||||
// (a static list, or a function of the selection). L3 uses it to drop the
|
||||
// whole DCP operating point, which the server rejects with an L3 backend.
|
||||
const overlayStrip = (cellFlags, sel) => {
|
||||
const strip = overlayPart(sel, "stripPrefixes");
|
||||
if (!strip.length) return [...(cellFlags || [])];
|
||||
return (cellFlags || []).filter((f) => !strip.includes(f.split(/[\s=]/)[0]));
|
||||
};
|
||||
// ==== end MIRROR ====
|
||||
const withOverlay = (cell, sel) => (cell && {
|
||||
...cell,
|
||||
flags: [...overlayStrip(cell.flags, sel), ...overlayPart(sel, "flags")],
|
||||
env: [...(cell.env || []), ...overlayPart(sel, "env")],
|
||||
}) || cell;
|
||||
// Shared with `_deployment.jsx` (HOST/PORT/etc. unified across the page).
|
||||
const STORAGE_KEY = "sglang-deploy-env";
|
||||
|
||||
@@ -56,9 +120,11 @@ export const Playground = ({ config }) => {
|
||||
cells.find((c) => DIMENSIONS.every((d) => c.match[d] === sel[d]));
|
||||
|
||||
// After applying overrides, the resulting (env, flags) may equal another
|
||||
// cell sharing the same (hw, variant, quant, nodes) but a different
|
||||
// strategy. flags compared ordered; env compared as a set.
|
||||
// cell sharing every match dim except strategy. Keep this generic so custom
|
||||
// grids (for example K3's PD mode) cannot match across unrelated base cells.
|
||||
// flags compared ordered; env compared as a set.
|
||||
const findMatchingCell = (cells, sel, pgEnv, pgFlags) => {
|
||||
const fixedDims = DIMENSIONS.filter((d) => d !== "strategy");
|
||||
const flagsEq = (a, b) =>
|
||||
a.length === b.length && a.every((x, i) => x === b[i]);
|
||||
const envEq = (a, b) => {
|
||||
@@ -68,10 +134,7 @@ export const Playground = ({ config }) => {
|
||||
return true;
|
||||
};
|
||||
for (const c of cells) {
|
||||
if (c.match.hw !== sel.hw) continue;
|
||||
if (c.match.variant !== sel.variant) continue;
|
||||
if (c.match.quant !== sel.quant) continue;
|
||||
if (c.match.nodes !== sel.nodes) continue;
|
||||
if (fixedDims.some((d) => c.match[d] !== sel[d])) continue;
|
||||
if (flagsEq(c.flags || [], pgFlags || []) && envEq(c.env || [], pgEnv || [])) {
|
||||
return c;
|
||||
}
|
||||
@@ -236,20 +299,37 @@ export const Playground = ({ config }) => {
|
||||
return null;
|
||||
};
|
||||
|
||||
// --tp / --ep spelling families: configs write either the canonical
|
||||
// --tp-size / --ep-size or the short --tp / --ep. Parse and strip every
|
||||
// spelling; when re-emitting, keep the spelling the base already uses.
|
||||
const TP_HEADS = ["--tp-size", "--tp", "--tensor-parallel-size"];
|
||||
const EP_HEADS = ["--ep-size", "--ep", "--expert-parallel-size"];
|
||||
const parseIntFlagAny = (flags, heads) => {
|
||||
for (const head of heads) {
|
||||
const n = parseIntFlag(flags, head);
|
||||
if (n !== null) return n;
|
||||
}
|
||||
return null;
|
||||
};
|
||||
const flagSpelling = (flags, heads, fallback) =>
|
||||
heads.find((head) =>
|
||||
(flags || []).some((f) => f.split(/[\s=]/)[0] === head)) || fallback;
|
||||
|
||||
// Insertion-anchor sets (priority-ordered; each includes siblings so
|
||||
// insertion still works in partial cells).
|
||||
const ANCHOR_NEAR_MODEL_PATH = ["--model-path"];
|
||||
const ANCHOR_NEAR_TP = ["--tp", "--model-path"];
|
||||
const ANCHOR_NEAR_DP = ["--dp", "--tp", "--model-path"];
|
||||
const ANCHOR_NEAR_DPATTN = ["--enable-dp-attention", "--dp", "--tp", "--model-path"];
|
||||
const ANCHOR_NEAR_TP = ["--tp-size", "--tp", "--model-path"];
|
||||
const ANCHOR_NEAR_DP = ["--dp", "--tp-size", "--tp", "--model-path"];
|
||||
const ANCHOR_NEAR_DPATTN = ["--enable-dp-attention", "--dp", "--tp-size", "--tp", "--model-path"];
|
||||
const ANCHOR_NEAR_MOE = ["--moe-a2a-backend", "--moe-runner-backend",
|
||||
"--enable-dp-attention", "--dp", "--tp", "--model-path"];
|
||||
"--enable-dp-attention", "--dp", "--tp-size", "--tp", "--model-path"];
|
||||
|
||||
// Helper bundle passed to every axis handler.
|
||||
const helpers = {
|
||||
matchConstraint, evaluateChip, findEntry, isHidden,
|
||||
stripFlagsByFirstToken, stripEnvByPrefix, insertBeforeTail, insertAfter,
|
||||
parseIntFlag, hasFlag, findFlagArg,
|
||||
TP_HEADS, EP_HEADS, parseIntFlagAny, flagSpelling,
|
||||
ANCHOR_NEAR_MODEL_PATH, ANCHOR_NEAR_TP, ANCHOR_NEAR_DP,
|
||||
ANCHOR_NEAR_DPATTN, ANCHOR_NEAR_MOE,
|
||||
};
|
||||
@@ -288,7 +368,7 @@ export const Playground = ({ config }) => {
|
||||
// ==========================================================================
|
||||
// 5. AXIS_HANDLERS — the built-in playground axis registry
|
||||
// ==========================================================================
|
||||
// Each entry implements initState / revertHidden / apply / render (plus
|
||||
// Each entry implements initState / apply / render (plus
|
||||
// optional deriveFromBase / getRenderHints). Iterated in insertion order;
|
||||
// axes absent from config.playgroundFeatures are skipped.
|
||||
// - inherit-from-base sentinels: null / "current" / "auto" / "off" /
|
||||
@@ -322,26 +402,13 @@ export const Playground = ({ config }) => {
|
||||
else dpAttn = false;
|
||||
const cpSize = h.parseIntFlag(flags, "--attn-cp-size");
|
||||
return {
|
||||
tp: h.parseIntFlag(flags, "--tp"),
|
||||
tp: h.parseIntFlagAny(flags, h.TP_HEADS),
|
||||
cp: cpEnabledIn(flags) ? (cpSize !== null ? cpSize : 2) : null,
|
||||
cpStrategy: bakedCpStrategy(flags),
|
||||
dpAttn,
|
||||
};
|
||||
},
|
||||
|
||||
revertHidden: (value, fc, base, h) => {
|
||||
let changed = false;
|
||||
const next = { ...value };
|
||||
for (const knob of (fc.knobs || [])) {
|
||||
const cur = next[knob.id];
|
||||
if (cur !== null && cur !== undefined
|
||||
&& h.isHidden(knob.values, cur, base)) {
|
||||
next[knob.id] = null; changed = true;
|
||||
}
|
||||
}
|
||||
return changed ? next : value;
|
||||
},
|
||||
|
||||
apply: ({ flags, env, value, fc, sel, h }) => {
|
||||
// Live facts for constraint checks (same keys the render-side
|
||||
// constraintBase exposes), recomputed after each mutation.
|
||||
@@ -351,7 +418,7 @@ export const Playground = ({ config }) => {
|
||||
dpAttnOn: h.hasFlag(flags, "--enable-dp-attention"),
|
||||
cpOn: cpEnabledIn(flags),
|
||||
cpStrategy: bakedCpStrategy(flags) || "interleave",
|
||||
effTp: h.parseIntFlag(flags, "--tp"),
|
||||
effTp: h.parseIntFlagAny(flags, h.TP_HEADS),
|
||||
});
|
||||
// The runtime derives the prefill-CP size as attn_cp_size = tp/dp
|
||||
// (a mismatched --attn-cp-size is overridden), so with DP-Attention
|
||||
@@ -366,7 +433,7 @@ export const Playground = ({ config }) => {
|
||||
: (h.hasFlag(flags, "--enable-dp-attention")
|
||||
? (h.parseIntFlag(flags, "--dp") ?? 1) : false);
|
||||
if (typeof dpIntent === "number" && dpIntent > 1) return null;
|
||||
return h.parseIntFlag(flags, "--tp");
|
||||
return h.parseIntFlagAny(flags, h.TP_HEADS);
|
||||
};
|
||||
// Skip a knob whose entry or picked value is hidden/disabled under
|
||||
// the live facts — mirrors the grayed controls, so stale state never
|
||||
@@ -389,8 +456,9 @@ export const Playground = ({ config }) => {
|
||||
// below the command box) rather than banned.
|
||||
|
||||
if (value.tp !== null && !blocked("tp", value.tp)) {
|
||||
flags = h.stripFlagsByFirstToken(flags, ["--tp"]);
|
||||
flags = h.insertAfter(flags, h.ANCHOR_NEAR_MODEL_PATH, [`--tp ${value.tp}`]);
|
||||
const tpHead = h.flagSpelling(flags, h.TP_HEADS, "--tp");
|
||||
flags = h.stripFlagsByFirstToken(flags, h.TP_HEADS);
|
||||
flags = h.insertAfter(flags, h.ANCHOR_NEAR_MODEL_PATH, [`${tpHead} ${value.tp}`]);
|
||||
}
|
||||
// CP override: an explicit size pick, or a strategy-only pick on a
|
||||
// base that already carries CP. Strategy precedence: explicit knob >
|
||||
@@ -521,43 +589,29 @@ export const Playground = ({ config }) => {
|
||||
(e) => e.startsWith("SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS"));
|
||||
return {
|
||||
backend: a2a || runner || null,
|
||||
ep: h.parseIntFlag(flags, "--ep"),
|
||||
ep: h.parseIntFlagAny(flags, h.EP_HEADS),
|
||||
mmQuant: fp4Acts ? "w4a4" : "w4a8",
|
||||
};
|
||||
},
|
||||
|
||||
revertHidden: (value, fc, base, h) => {
|
||||
let changed = false;
|
||||
const next = { ...value };
|
||||
if (next.backend !== null && fc.backend?.options
|
||||
&& h.isHidden(fc.backend.options, next.backend, base)) {
|
||||
next.backend = null; changed = true;
|
||||
}
|
||||
// MegaMoE backend availability — gated by its option's requiresHw /
|
||||
// excludesStrategy (this model gates by hw only; the check is generic).
|
||||
const mmOpt = (fc.backend?.options || []).find((o) => o.id === "megamoe");
|
||||
const mmAvail = !!mmOpt
|
||||
&& (!mmOpt.requiresHw || mmOpt.requiresHw.includes(base.hw))
|
||||
&& (!mmOpt.excludesStrategy || !mmOpt.excludesStrategy.includes(base.strategy));
|
||||
if (next.backend === "megamoe" && !mmAvail) {
|
||||
next.backend = null; changed = true;
|
||||
}
|
||||
if (next.ep !== null && fc.ep?.values
|
||||
&& h.isHidden(fc.ep.values, next.ep, base)) {
|
||||
next.ep = null; changed = true;
|
||||
}
|
||||
return changed ? next : value;
|
||||
},
|
||||
|
||||
apply: ({ flags, env, value, fc, h, derived }) => {
|
||||
if (value.backend !== null) {
|
||||
flags = h.stripFlagsByFirstToken(flags, [
|
||||
"--moe-a2a-backend", "--moe-runner-backend",
|
||||
]);
|
||||
// Backend options may carry their own env (e.g. the FlashInfer MXFP4
|
||||
// cubin-pool path): strip every backend option's env keys, then
|
||||
// re-add the selected option's.
|
||||
const backendEnvKeys = [];
|
||||
for (const o of (fc.backend?.options || [])) {
|
||||
for (const e of (o.env || [])) backendEnvKeys.push(e.split("=")[0]);
|
||||
}
|
||||
if (backendEnvKeys.length) env = h.stripEnvByPrefix(env, backendEnvKeys);
|
||||
const opt = (fc.backend?.options || []).find((o) => o.id === value.backend);
|
||||
if (opt?.flags?.length) {
|
||||
flags = h.insertAfter(flags, h.ANCHOR_NEAR_DPATTN, opt.flags);
|
||||
}
|
||||
if (opt?.env?.length) env = [...env, ...opt.env];
|
||||
}
|
||||
// MegaMoE owns the MoE path: when the effective backend is megamoe, strip the
|
||||
// DeepEP dispatch + any prior megamoe env, then re-add the selected quant's
|
||||
@@ -582,9 +636,10 @@ export const Playground = ({ config }) => {
|
||||
}
|
||||
}
|
||||
if (value.ep !== null) {
|
||||
flags = h.stripFlagsByFirstToken(flags, ["--ep"]);
|
||||
const epHead = h.flagSpelling(flags, h.EP_HEADS, "--ep");
|
||||
flags = h.stripFlagsByFirstToken(flags, h.EP_HEADS);
|
||||
if (value.ep > 1) {
|
||||
flags = h.insertAfter(flags, h.ANCHOR_NEAR_MOE, [`--ep ${value.ep}`]);
|
||||
flags = h.insertAfter(flags, h.ANCHOR_NEAR_MOE, [`${epHead} ${value.ep}`]);
|
||||
}
|
||||
}
|
||||
return { flags, env };
|
||||
@@ -665,18 +720,6 @@ export const Playground = ({ config }) => {
|
||||
return out;
|
||||
},
|
||||
|
||||
revertHidden: (value, fc, base, h) => {
|
||||
let changed = false;
|
||||
const next = { ...value };
|
||||
for (const item of (fc.items || [])) {
|
||||
if (next[item.id] !== null && next[item.id] !== undefined
|
||||
&& h.evaluateChip(item, base).hidden) {
|
||||
next[item.id] = null; changed = true;
|
||||
}
|
||||
}
|
||||
return changed ? next : value;
|
||||
},
|
||||
|
||||
apply: ({ flags, env, value, fc, h, derived }) => {
|
||||
const items = fc.items || [];
|
||||
// Effective state per item: explicit > derived > false. Skip
|
||||
@@ -755,13 +798,6 @@ export const Playground = ({ config }) => {
|
||||
return "current";
|
||||
},
|
||||
|
||||
revertHidden: (value, fc, base, h) => {
|
||||
if (value !== "current" && h.isHidden(fc.options || [], value, base)) {
|
||||
return "current";
|
||||
}
|
||||
return value;
|
||||
},
|
||||
|
||||
apply: ({ flags, env, value, fc, h, derived }) => {
|
||||
if (value === "current") return { flags, env };
|
||||
// No-op when the pick already matches base (preserves flag position).
|
||||
@@ -815,39 +851,41 @@ export const Playground = ({ config }) => {
|
||||
// Owns the `--disaggregation-*` flags (unconditional strip). A backend may
|
||||
// carry hw-gated env (transferBackends[].env + .envWhen).
|
||||
pdDisagg: {
|
||||
initState: () => ({ mode: "off", transferBackend: "mooncake", ibDevice: "auto" }),
|
||||
|
||||
revertHidden: (value, fc, base, h) => {
|
||||
let changed = false;
|
||||
const next = { ...value };
|
||||
if (next.mode !== "off" && fc.modes
|
||||
&& h.isHidden(fc.modes, next.mode, base)) {
|
||||
next.mode = "off"; changed = true;
|
||||
}
|
||||
if (next.ibDevice !== "auto" && fc.ibDevices
|
||||
&& h.isHidden(fc.ibDevices, next.ibDevice, base)) {
|
||||
next.ibDevice = "auto"; changed = true;
|
||||
}
|
||||
if (next.transferBackend !== "mooncake" && fc.transferBackends
|
||||
&& h.isHidden(fc.transferBackends, next.transferBackend, base)) {
|
||||
next.transferBackend = "mooncake"; changed = true;
|
||||
}
|
||||
return changed ? next : value;
|
||||
},
|
||||
// The transport default is the config's first entry, so a model whose
|
||||
// recipes standardize on one backend does not silently start on another.
|
||||
initState: (fc) => ({
|
||||
mode: "off",
|
||||
transferBackend: (fc && (fc.transferBackends || [])[0] || {}).id || "mooncake",
|
||||
ibDevice: "auto",
|
||||
}),
|
||||
|
||||
apply: ({ flags, env, value, sel, fc, h }) => {
|
||||
// The bootstrap port is the base cell's to choose — the router's
|
||||
// --prefill positional has to match it — so carry it across the strip
|
||||
// rather than dropping it and silently falling back to the default.
|
||||
const bootstrapPort = h.findFlagArg(flags, "--disaggregation-bootstrap-port");
|
||||
flags = h.stripFlagsByFirstToken(flags, [
|
||||
"--disaggregation-mode", "--disaggregation-transfer-backend",
|
||||
"--disaggregation-ib-device", "--disaggregation-bootstrap-port",
|
||||
]);
|
||||
const backends = fc.transferBackends || [];
|
||||
// A config that omits `modes` has the role on the Deploy panel instead;
|
||||
// this card then only tunes the transport for whatever role is selected.
|
||||
const mode = (fc.modes || []).length
|
||||
? value.mode
|
||||
: ((sel && sel.pdMode) || "off");
|
||||
|
||||
if (value.mode === "prefill" || value.mode === "decode") {
|
||||
const backend = value.transferBackend || "mooncake";
|
||||
if (mode === "prefill" || mode === "decode") {
|
||||
const backend = value.transferBackend || (backends[0] || {}).id || "mooncake";
|
||||
const adds = [
|
||||
`--disaggregation-mode ${value.mode}`,
|
||||
`--disaggregation-mode ${mode}`,
|
||||
`--disaggregation-transfer-backend ${backend}`,
|
||||
];
|
||||
// Re-emitted in the base cell's own order, so an untouched Prefill
|
||||
// recipe renders byte-identical to the Deploy panel.
|
||||
if (bootstrapPort) {
|
||||
adds.push(`--disaggregation-bootstrap-port ${bootstrapPort}`);
|
||||
}
|
||||
if (value.ibDevice && value.ibDevice !== "auto") {
|
||||
adds.push(`--disaggregation-ib-device ${value.ibDevice}`);
|
||||
}
|
||||
@@ -859,7 +897,7 @@ export const Playground = ({ config }) => {
|
||||
|
||||
// Role-specific serving port so the router's prefill / decode targets
|
||||
// line up (and prefill+decode don't collide on a single host).
|
||||
const servePort = PD_PORTS[value.mode].serve;
|
||||
const servePort = PD_PORTS[mode].serve;
|
||||
flags = flags.map((f) =>
|
||||
f.split(/[\s=]/)[0] === "--port" ? `--port ${servePort}` : f);
|
||||
|
||||
@@ -930,14 +968,6 @@ export const Playground = ({ config }) => {
|
||||
hisparse: {
|
||||
initState: (fc) => ({ enable: false, hostRatio: (fc && fc.defaultHostRatio) || null }),
|
||||
|
||||
revertHidden: (value, fc, base, h) => {
|
||||
if (value.hostRatio !== null && fc.hostRatios
|
||||
&& h.isHidden(fc.hostRatios, value.hostRatio, base)) {
|
||||
return { ...value, hostRatio: (fc && fc.defaultHostRatio) || null };
|
||||
}
|
||||
return value;
|
||||
},
|
||||
|
||||
apply: ({ flags, env, value, fc, h }) => {
|
||||
const ownedHeads = [
|
||||
"--enable-hisparse", "--hisparse-config",
|
||||
@@ -969,10 +999,12 @@ export const Playground = ({ config }) => {
|
||||
<div key={axisId} style={s.card}>
|
||||
<div style={s.compactRow}>
|
||||
<span style={s.axisTitle}>HiSparse</span>
|
||||
<span style={s.field}>
|
||||
{renderChip("Enable", value.enable, true,
|
||||
() => setSlot("enable", !value.enable))}
|
||||
</span>
|
||||
{typeof fc.showWhen !== "function" && (
|
||||
<span style={s.field}>
|
||||
{renderChip("Enable", value.enable, true,
|
||||
() => setSlot("enable", !value.enable))}
|
||||
</span>
|
||||
)}
|
||||
{hasRatios && (
|
||||
<span style={s.field}>
|
||||
<span style={s.fieldLabel}>Host ratio</span>
|
||||
@@ -992,22 +1024,23 @@ export const Playground = ({ config }) => {
|
||||
hicache: {
|
||||
initState: () => ({ enable: false, backend: null, writePolicy: "auto" }),
|
||||
|
||||
revertHidden: (value, fc, base, h) => {
|
||||
let changed = false;
|
||||
const next = { ...value };
|
||||
if (next.backend !== null && fc.backends
|
||||
&& h.isHidden(fc.backends, next.backend, base)) {
|
||||
next.backend = null; changed = true;
|
||||
}
|
||||
if (next.writePolicy !== "auto" && fc.writePolicies
|
||||
&& h.isHidden(fc.writePolicies, next.writePolicy, base)) {
|
||||
next.writePolicy = "auto"; changed = true;
|
||||
}
|
||||
return changed ? next : value;
|
||||
},
|
||||
|
||||
apply: ({ flags, env, value, fc, sel, h }) => {
|
||||
if (fc.excludesHw && sel && fc.excludesHw.includes(sel.hw)) return { flags, env };
|
||||
// When the Deploy panel owns enablement (`showWhen`), the base already
|
||||
// carries a complete, verified hicache recipe. Rebuilding it from this
|
||||
// axis's own defaults would silently swap ratio/layout/io-backend, so
|
||||
// only the two knobs this card actually exposes are touched.
|
||||
if (typeof fc.showWhen === "function") {
|
||||
const set = (name, val) => {
|
||||
flags = h.stripFlagsByFirstToken(flags, [name]);
|
||||
if (val) flags = h.insertBeforeTail(flags, [`${name} ${val}`]);
|
||||
};
|
||||
if (value.backend) set("--hicache-storage-backend", value.backend);
|
||||
if (value.writePolicy && value.writePolicy !== "auto") {
|
||||
set("--hicache-write-policy", value.writePolicy);
|
||||
}
|
||||
return { flags, env };
|
||||
}
|
||||
flags = h.stripFlagsByFirstToken(flags, [
|
||||
"--enable-hierarchical-cache", "--hicache-ratio", "--hicache-size",
|
||||
"--hicache-write-policy", "--hicache-mem-layout", "--hicache-io-backend",
|
||||
@@ -1066,10 +1099,12 @@ export const Playground = ({ config }) => {
|
||||
<div key={axisId} style={s.card}>
|
||||
<div style={s.compactRow}>
|
||||
<span style={s.axisTitle}>HiCache</span>
|
||||
<span style={s.field}>
|
||||
{renderChip("Enable", value.enable, true,
|
||||
() => setSlot("enable", !value.enable))}
|
||||
</span>
|
||||
{typeof fc.showWhen !== "function" && (
|
||||
<span style={s.field}>
|
||||
{renderChip("Enable", value.enable, true,
|
||||
() => setSlot("enable", !value.enable))}
|
||||
</span>
|
||||
)}
|
||||
{hasBackends && (
|
||||
<span style={s.field}>
|
||||
<span style={s.fieldLabel}>Storage</span>
|
||||
@@ -1092,23 +1127,29 @@ export const Playground = ({ config }) => {
|
||||
|
||||
// ---- Axis: Flag Selects (generic, config-declared) ----------------------
|
||||
// A LIST of single-selects, each declared entirely in config:
|
||||
// { id, title, stripPrefixes: [...], options: [{ id, label, flags? }] }
|
||||
// { id, title, stripPrefixes: [...], stripEnv?: [...],
|
||||
// options: [{ id, label, flags?, env? }] }
|
||||
// Same shape as `speculative` minus its hardcoded title + strip list: pick
|
||||
// an option → strip the family, splice the option's flags. A flagless
|
||||
// option is the "none" / accuracy-safe choice (matches a base carrying none
|
||||
// of the family). Model-specific controls (KV-cache dtype, mamba scheduler
|
||||
// strategy, …) live here as DATA — no per-feature engine code. Supports
|
||||
// multiple selects per page. State: { [selectId]: optionId | null }
|
||||
// (null = inherit base).
|
||||
// (null = inherit base). `default` may be an option id or a function of the
|
||||
// base selection (re-evaluated on every base change).
|
||||
flagSelects: {
|
||||
initState: (fc) => {
|
||||
initState: (fc, base) => {
|
||||
const out = {};
|
||||
for (const spec of (fc || [])) out[spec.id] = null;
|
||||
for (const spec of (fc || [])) {
|
||||
const d = typeof spec.default === "function" ? spec.default(base) : spec.default;
|
||||
out[spec.id] = d ?? null;
|
||||
}
|
||||
return out;
|
||||
},
|
||||
|
||||
// Per select: match base's family flags (first token ∈ stripPrefixes)
|
||||
// against each option's flags. A flagless option matches an empty family.
|
||||
// Function-flag options (computed from the row values) never match.
|
||||
deriveFromBase: (cell, fc) => {
|
||||
const flags = (cell && cell.flags) || [];
|
||||
const out = {};
|
||||
@@ -1117,6 +1158,7 @@ export const Playground = ({ config }) => {
|
||||
const fam = flags.filter((f) => prefixes.includes(f.split(/[\s=]/)[0]));
|
||||
let hit = null;
|
||||
for (const opt of (spec.options || [])) {
|
||||
if (typeof opt.flags === "function") continue;
|
||||
const of = opt.flags || [];
|
||||
if (of.length === fam.length && of.every((x) => fam.includes(x))) {
|
||||
hit = opt.id; break;
|
||||
@@ -1127,19 +1169,6 @@ export const Playground = ({ config }) => {
|
||||
return out;
|
||||
},
|
||||
|
||||
revertHidden: (value, fc, base, h) => {
|
||||
let changed = false;
|
||||
const next = { ...value };
|
||||
for (const spec of (fc || [])) {
|
||||
const cur = next[spec.id];
|
||||
if (cur !== null && cur !== undefined
|
||||
&& h.isHidden(spec.options, cur, base)) {
|
||||
next[spec.id] = null; changed = true;
|
||||
}
|
||||
}
|
||||
return changed ? next : value;
|
||||
},
|
||||
|
||||
apply: ({ flags, env, value, fc, sel, h, derived }) => {
|
||||
const evalBase = {
|
||||
...(sel || {}),
|
||||
@@ -1147,6 +1176,9 @@ export const Playground = ({ config }) => {
|
||||
pdMode: h.findFlagArg(flags, "--disaggregation-mode") || "off",
|
||||
};
|
||||
for (const spec of (fc || [])) {
|
||||
// Hidden rows must not emit: showWhen also receives the sibling row
|
||||
// values (explicit picks + derived), so a row can gate on another's pick.
|
||||
if (typeof spec.showWhen === "function" && !spec.showWhen(sel, value, derived)) continue;
|
||||
const v = value ? value[spec.id] : null;
|
||||
if (v === null || v === undefined) continue; // inherit base
|
||||
const d = derived ? derived[spec.id] : null;
|
||||
@@ -1154,10 +1186,51 @@ export const Playground = ({ config }) => {
|
||||
const opt = (spec.options || []).find((o) => o.id === v);
|
||||
if (!opt) continue;
|
||||
if (h.evaluateChip(opt, evalBase).disabled) continue;
|
||||
flags = h.stripFlagsByFirstToken(flags, spec.stripPrefixes || []);
|
||||
if (opt.flags && opt.flags.length) {
|
||||
flags = h.insertBeforeTail(flags, opt.flags);
|
||||
// `flags` may be a function of the whole row-value object (cross-row
|
||||
// presets, e.g. preset x cluster-size). A function returning null
|
||||
// means "leave the base untouched" (a true no-op Off).
|
||||
const optFlags = typeof opt.flags === "function"
|
||||
? opt.flags(value, evalBase)
|
||||
: (opt.flags || []);
|
||||
if (optFlags === null) continue;
|
||||
// In-place substitution keeps the rendered diff minimal: a family the
|
||||
// base already carries changes value at its original position; families
|
||||
// stripped and not re-emitted vanish in place; only net-new flags
|
||||
// append as a block before the tail.
|
||||
const strip = new Set(spec.stripPrefixes || []);
|
||||
const byTok = new Map();
|
||||
for (const f of optFlags) {
|
||||
const t = f.split(/[\s=]/)[0];
|
||||
if (!byTok.has(t)) byTok.set(t, []);
|
||||
byTok.get(t).push(f);
|
||||
}
|
||||
const consumed = new Set();
|
||||
const next = [];
|
||||
for (const f of flags) {
|
||||
const t = f.split(/[\s=]/)[0];
|
||||
if (byTok.has(t)) {
|
||||
if (!consumed.has(t)) {
|
||||
next.push(...byTok.get(t));
|
||||
consumed.add(t);
|
||||
}
|
||||
} else if (!strip.has(t)) {
|
||||
next.push(f);
|
||||
}
|
||||
}
|
||||
const fresh = [];
|
||||
for (const [t, fs] of byTok) {
|
||||
if (!consumed.has(t)) fresh.push(...fs);
|
||||
}
|
||||
flags = fresh.length ? h.insertBeforeTail(next, fresh) : next;
|
||||
// Option env (env-var toggles, preset env): strip spec.stripEnv plus
|
||||
// every option's env keys, then add the picked option's — mirrors the
|
||||
// MoE backend card's strip-then-emit shape.
|
||||
const envKeys = [...(spec.stripEnv || [])];
|
||||
for (const o of (spec.options || [])) {
|
||||
for (const e of (o.env || [])) envKeys.push(e.split("=")[0]);
|
||||
}
|
||||
if (envKeys.length) env = h.stripEnvByPrefix(env, envKeys);
|
||||
if (opt.env && opt.env.length) env = [...env, ...opt.env];
|
||||
}
|
||||
return { flags, env };
|
||||
},
|
||||
@@ -1165,6 +1238,10 @@ export const Playground = ({ config }) => {
|
||||
render: ({ axisId, value, setValue, fc, base, s, h, renderChip, derived }) => {
|
||||
const cards = [];
|
||||
for (const spec of (fc || [])) {
|
||||
// Row-level gate: a select whose whole family is meaningless under the
|
||||
// current base (e.g. draft tokens with speculation off) is not rendered.
|
||||
// Also receives (rowValues, derived) for sibling-dependent rows.
|
||||
if (typeof spec.showWhen === "function" && !spec.showWhen(base, value, derived)) continue;
|
||||
const opts = (spec.options || [])
|
||||
.map((o) => h.evaluateChip(o, base))
|
||||
.filter((c) => !c.hidden);
|
||||
@@ -1172,6 +1249,34 @@ export const Playground = ({ config }) => {
|
||||
const explicit = value ? value[spec.id] : null;
|
||||
const display = (explicit !== null && explicit !== undefined)
|
||||
? explicit : (derived ? derived[spec.id] : null);
|
||||
// `control: "slider"` renders the same option list as a range input —
|
||||
// for dense ordered scales (1..7) where chips are just noise. Option
|
||||
// ORDER is the scale; the option id is still what apply() consumes.
|
||||
if (spec.control === "slider") {
|
||||
const idx = Math.max(0, opts.findIndex((c) => c.value === display));
|
||||
const cur = opts[idx];
|
||||
cards.push(
|
||||
<div key={`${axisId}-${spec.id}`} style={s.card}>
|
||||
<div style={s.compactRow}>
|
||||
<span style={s.axisTitle}>{spec.title}</span>
|
||||
<input
|
||||
type="range"
|
||||
min={0} max={opts.length - 1} step={1}
|
||||
value={idx}
|
||||
onChange={(e) => setValue({
|
||||
...value,
|
||||
[spec.id]: opts[Number(e.target.value)].value,
|
||||
})}
|
||||
style={{ flex: 1, minWidth: "120px", accentColor: "#D45D44" }}
|
||||
/>
|
||||
<span style={{ ...s.axisTitle, minWidth: "24px", textAlign: "right" }}>
|
||||
{cur ? cur.label : "-"}
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
continue;
|
||||
}
|
||||
cards.push(
|
||||
<div key={`${axisId}-${spec.id}`} style={s.card}>
|
||||
<div style={s.compactRow}>
|
||||
@@ -1224,13 +1329,21 @@ export const Playground = ({ config }) => {
|
||||
// callers can pass a modified env (e.g. MegaMoE's stripEnv + append).
|
||||
const renderCommandLines = (cell, flags, cellEnv, sel, envValues, pdMode = null, mode = "python") => {
|
||||
const modelName = resolveModelName(sel);
|
||||
const nnodes = parseNnodes(sel.nodes);
|
||||
const multinode = nnodes > 1;
|
||||
let f = [...flags];
|
||||
// Presets may replace the base cell's topology by emitting --nnodes
|
||||
// directly (for example, B300 1-node -> large-scale 4/8-node). Use that
|
||||
// effective value for Docker networking, hints, and the command banner.
|
||||
const nnodesFlag = f.find((x) => x.split(/[\s=]/)[0] === "--nnodes");
|
||||
const nnodesMatch = nnodesFlag && /^--nnodes(?:\s+|=)(\d+)$/.exec(nnodesFlag.trim());
|
||||
const baseNnodes = sel.nodes !== undefined
|
||||
? parseNnodes(sel.nodes)
|
||||
: ((cell && cell.nnodes) || 1);
|
||||
const nnodes = nnodesMatch ? parseInt(nnodesMatch[1], 10) : baseNnodes;
|
||||
const multinode = nnodes > 1;
|
||||
if (multinode && !f.some((x) => x.startsWith("--nnodes"))) {
|
||||
// Insert the multi-node trio after the last parallelism flag (matches
|
||||
// _deployment.jsx so untouched-base output is byte-identical).
|
||||
const PARALLELISM_ANCHORS = ["--enable-dp-attention", "--dp", "--tp"];
|
||||
const PARALLELISM_ANCHORS = ["--enable-dp-attention", "--dp", "--tp-size", "--tp"];
|
||||
let at = -1;
|
||||
for (const anchor of PARALLELISM_ANCHORS) {
|
||||
at = f.findIndex((x) => x.split(/[\s=]/)[0] === anchor);
|
||||
@@ -1290,7 +1403,7 @@ export const Playground = ({ config }) => {
|
||||
&& config.playgroundFeatures.pdDisagg.router;
|
||||
const routerPort = (routerCfg && routerCfg.port) || 8000;
|
||||
const routerLine = routerCfg
|
||||
? `# then front BOTH with the Router (SGLang Model Gateway) shown below.\n`
|
||||
? `# then front BOTH with the Router shown below.\n`
|
||||
+ `# Client traffic (cURL) targets the router (:${routerPort}), not this role server.`
|
||||
: `# then front BOTH with a router; client traffic targets the router, not this role server.`;
|
||||
const hicacheCfg = config.playgroundFeatures
|
||||
@@ -1685,8 +1798,18 @@ export const Playground = ({ config }) => {
|
||||
|
||||
// Base selection — live-linked to the Deployment panel via URL hash + custom event
|
||||
// (history.replaceState doesn't fire hashchange, hence the event too).
|
||||
const overlayDefaults = () => {
|
||||
const out = {};
|
||||
for (const d of (config.overlayDims || [])) {
|
||||
const opts = d.options || [];
|
||||
out[d.id] = d.default !== undefined ? d.default : ((opts[0] && opts[0].id) || "");
|
||||
}
|
||||
return out;
|
||||
};
|
||||
const baseFallback = () => ({ ...config.cells[0].match, ...overlayDefaults() });
|
||||
|
||||
const initialBaseFromHash = () => {
|
||||
const fallback = config.cells[0].match;
|
||||
const fallback = baseFallback();
|
||||
if (typeof window === "undefined") return { ...fallback };
|
||||
const raw = window.location.hash.replace(/^#/, "");
|
||||
if (!raw) return { ...fallback };
|
||||
@@ -1699,7 +1822,7 @@ export const Playground = ({ config }) => {
|
||||
useEffect(() => {
|
||||
const onHash = () => setBase(initialBaseFromHash());
|
||||
const onSelEvent = (e) => {
|
||||
const fallback = config.cells[0].match;
|
||||
const fallback = baseFallback();
|
||||
const incoming = (e && e.detail) || {};
|
||||
const next = { ...fallback };
|
||||
for (const k of Object.keys(next)) {
|
||||
@@ -1715,35 +1838,36 @@ export const Playground = ({ config }) => {
|
||||
};
|
||||
}, []);
|
||||
|
||||
// Calculator-computed --mamba-full-memory-ratio pair (K3): `eff` matches the
|
||||
// composed command (with playground overrides), `base` matches the base view.
|
||||
// Rendered into the command views below; never part of the broadcast flags.
|
||||
const [pgRatios, setPgRatios] = useState({ eff: null, base: null });
|
||||
useEffect(() => {
|
||||
const onRatio = (e) => setPgRatios({
|
||||
eff: (e.detail && e.detail.ratio) || null,
|
||||
base: (e.detail && (e.detail.baseRatio || e.detail.ratio)) || null,
|
||||
});
|
||||
window.addEventListener("sglang-k3-mamba-ratio", onRatio);
|
||||
return () => window.removeEventListener("sglang-k3-mamba-ratio", onRatio);
|
||||
}, []);
|
||||
|
||||
// Deltas: one slot per declared axis.
|
||||
const initialDeltas = () => {
|
||||
const out = {};
|
||||
for (const [axisId, handler] of Object.entries(AXIS_HANDLERS)) {
|
||||
const fc = pgFeatures[axisId];
|
||||
if (fc) out[axisId] = handler.initState(fc);
|
||||
if (fc) out[axisId] = handler.initState(fc, base);
|
||||
}
|
||||
return out;
|
||||
};
|
||||
const [deltas, setDeltas] = useState(initialDeltas);
|
||||
|
||||
// On base change, revert any now-hidden picks to their inherit default.
|
||||
// Disabled picks are NOT reverted (soft warning).
|
||||
// Picking a different base is picking a different starting point, so the
|
||||
// playground goes back to inheriting everything. Carrying overrides across a
|
||||
// base change silently mixes knobs from a config the reader already left.
|
||||
useEffect(() => {
|
||||
setDeltas((d) => {
|
||||
let next = d;
|
||||
let mutated = false;
|
||||
for (const [axisId, handler] of Object.entries(AXIS_HANDLERS)) {
|
||||
const fc = pgFeatures[axisId];
|
||||
if (!fc || d[axisId] === undefined) continue;
|
||||
const nv = handler.revertHidden(d[axisId], fc, base, helpers);
|
||||
if (nv !== d[axisId]) {
|
||||
if (!mutated) { next = { ...d }; mutated = true; }
|
||||
next[axisId] = nv;
|
||||
}
|
||||
}
|
||||
return mutated ? next : d;
|
||||
});
|
||||
}, [base.hw, base.variant, base.quant, base.strategy, base.nodes]);
|
||||
setDeltas(initialDeltas());
|
||||
}, [Object.keys(base).sort().map((k) => `${k}=${base[k]}`).join("&")]);
|
||||
|
||||
const [modal, setModal] = useState(null); // 'curl' | 'env' | 'submit' | null
|
||||
|
||||
@@ -1800,7 +1924,7 @@ export const Playground = ({ config }) => {
|
||||
// 10. Derived values
|
||||
// ==========================================================================
|
||||
const s = makeStyles(isDark);
|
||||
const baseCell = findCell(config.cells, base);
|
||||
const baseCell = withOverlay(findCell(config.cells, base), base);
|
||||
const modelName = resolveModelName(base);
|
||||
|
||||
// Per-axis state recovered from the base cell's flags (deriveFromBase).
|
||||
@@ -1815,7 +1939,7 @@ export const Playground = ({ config }) => {
|
||||
|
||||
// Cross-axis facts folded into the `base` handed to chip-constraint
|
||||
// matching, so `hide`/`disable` can react to another axis's live state
|
||||
// (render path only; revertHidden keeps the clean 5-dim base).
|
||||
// (render path only; the raw base stays untouched).
|
||||
// dpAttnOn — effective DP-Attention resolves to "on" (positive degree
|
||||
// or true), explicit override else derived-from-base.
|
||||
// cpOn — effective prefill-CP resolves to "on" (degree > 1),
|
||||
@@ -1871,7 +1995,10 @@ export const Playground = ({ config }) => {
|
||||
? attnDelta.cpStrategy
|
||||
: (attnDerived.cpStrategy !== undefined ? attnDerived.cpStrategy : null))
|
||||
|| "interleave";
|
||||
const pdMode = (deltas.pdDisagg && deltas.pdDisagg.mode) || "off";
|
||||
const pdCardOwnsMode = ((pgFeatures.pdDisagg && pgFeatures.pdDisagg.modes) || []).length > 0;
|
||||
const pdMode = pdCardOwnsMode
|
||||
? ((deltas.pdDisagg && deltas.pdDisagg.mode) || "off")
|
||||
: (base.pdMode || "off");
|
||||
const constraintBase = {
|
||||
...base, dpAttnOn, cpOn, cpStrategy, cpSizeTarget, effTp, pdMode,
|
||||
};
|
||||
@@ -1881,14 +2008,47 @@ export const Playground = ({ config }) => {
|
||||
let diffLines = [];
|
||||
let pgFlagsLatest = [];
|
||||
let pgEnvLatest = [];
|
||||
// Render-only ratio injection (before the host/port tail); skipped if the
|
||||
// flags somehow already carry the family.
|
||||
const withRatio = (fl, value) => {
|
||||
if (!value) return fl;
|
||||
if (fl.some((f) => f.startsWith("--mamba-full-memory-ratio"))) return fl;
|
||||
const out = [...fl];
|
||||
const line = `--mamba-full-memory-ratio ${value}`;
|
||||
const i = out.findIndex((f) => f.startsWith("--host"));
|
||||
if (i >= 0) out.splice(i, 0, line);
|
||||
else out.push(line);
|
||||
return out;
|
||||
};
|
||||
if (baseCell) {
|
||||
baseCommand = renderCommandLines(baseCell, baseCell.flags, baseCell.env, base, env, null, runMode);
|
||||
baseCommand = renderCommandLines(baseCell, withRatio(baseCell.flags, pgRatios.base), baseCell.env, base, env, null, runMode);
|
||||
const { flags: pgFlags, env: pgEnv, pdMode } = applyAllDeltas(baseCell.flags, baseCell.env, deltas, base, derivedMap);
|
||||
pgFlagsLatest = pgFlags;
|
||||
pgEnvLatest = pgEnv;
|
||||
playgroundCommand = renderCommandLines(baseCell, pgFlags, pgEnv, base, env, pdMode, runMode);
|
||||
playgroundCommand = renderCommandLines(baseCell, withRatio(pgFlags, pgRatios.eff), pgEnv, base, env, pdMode, runMode);
|
||||
diffLines = computeDiff(baseCommand, playgroundCommand);
|
||||
}
|
||||
// Broadcast both configs for outside consumers (the mamba ratio calculator):
|
||||
// `baseFlags`/`baseEnv` = cell + Deploy overlays (what the Deploy command
|
||||
// shows); `flags`/`env` = that plus playground overrides. Keyed on content
|
||||
// so re-renders don't spam events.
|
||||
const effectiveKey =
|
||||
pgFlagsLatest.join("\n") + " " + pgEnvLatest.join("\n") + " " +
|
||||
(baseCell ? baseCell.flags.join("\n") + " " + baseCell.env.join("\n") : "");
|
||||
useEffect(() => {
|
||||
if (typeof window === "undefined" || !baseCell) return;
|
||||
window.dispatchEvent(
|
||||
new CustomEvent("sglang-k3-effective-config", {
|
||||
detail: {
|
||||
flags: pgFlagsLatest,
|
||||
env: pgEnvLatest,
|
||||
baseFlags: baseCell.flags,
|
||||
baseEnv: baseCell.env,
|
||||
},
|
||||
})
|
||||
);
|
||||
}, [effectiveKey]);
|
||||
|
||||
// Cross-cell verified detection: the emitted (env, flags) may match the
|
||||
// base cell itself or a sibling (different strategy). A sibling match
|
||||
// still shows Verified, plus a "switch base" link.
|
||||
@@ -1896,7 +2056,8 @@ export const Playground = ({ config }) => {
|
||||
? findMatchingCell(config.cells, base, pgEnvLatest, pgFlagsLatest)
|
||||
: null;
|
||||
const playgroundVerified = !!(matchedCell && matchedCell.verified);
|
||||
const matchedSiblingCell = (matchedCell && matchedCell !== baseCell)
|
||||
const matchedSiblingCell = (matchedCell
|
||||
&& DIMENSIONS.some((d) => matchedCell.match[d] !== base[d]))
|
||||
? matchedCell : null;
|
||||
// MTP hint on the EFFECTIVE (post-override) command — fires when the user
|
||||
// toggles speculative decoding on without setting --max-running-requests
|
||||
@@ -1969,8 +2130,13 @@ export const Playground = ({ config }) => {
|
||||
setTimeout(() => setCurlCopied(false), 1200);
|
||||
};
|
||||
|
||||
// Summarize whichever dims the Deploy panel actually has — a config may drop
|
||||
// variant/quant/nodes or add its own (PD mode, ...), so nothing is hardcoded.
|
||||
const baseSummary = baseCell
|
||||
? `${base.hw.toUpperCase()} · ${base.variant} · ${base.quant.toUpperCase()} · ${base.strategy} · ${base.nodes}`
|
||||
? Object.entries(base)
|
||||
.filter(([, v]) => v !== undefined && v !== "")
|
||||
.map(([k, v]) => (k === "hw" ? String(v).toUpperCase() : String(v)))
|
||||
.join(" · ")
|
||||
: "(no verified cell at the current Deploy selection — showing playground only)";
|
||||
|
||||
// ==========================================================================
|
||||
@@ -2071,6 +2237,9 @@ export const Playground = ({ config }) => {
|
||||
{Object.entries(AXIS_HANDLERS).map(([axisId, handler]) => {
|
||||
const fc = pgFeatures[axisId];
|
||||
if (!fc) return null;
|
||||
// An axis whose feature is not switched on in the Deploy panel has
|
||||
// nothing to tune — declared per config as `showWhen(base)`.
|
||||
if (typeof fc.showWhen === "function" && !fc.showWhen(constraintBase)) return null;
|
||||
const setValue = (next) => setDeltas((d) => ({ ...d, [axisId]: next }));
|
||||
return handler.render({
|
||||
axisId, value: deltas[axisId], setValue,
|
||||
@@ -2181,7 +2350,7 @@ export const Playground = ({ config }) => {
|
||||
{/* PD-Disagg router companion (separate block so the role diff stays pure). */}
|
||||
{pdRouter && routerText && (
|
||||
<div style={s.card}>
|
||||
<div style={s.title}>Router (SGLang Model Gateway)</div>
|
||||
<div style={s.title}>Router</div>
|
||||
<div style={{ fontSize: 11, opacity: 0.7, margin: "0 0 6px" }}>
|
||||
Run after both roles are up. Substitute <code>{"<prefill-host>"}</code> /{" "}
|
||||
<code>{"<decode-host>"}</code> with reachable hosts (both <code>127.0.0.1</code>{" "}
|
||||
|
||||
@@ -17,7 +17,7 @@ export const config = {
|
||||
// merges these in, so a model-specific GPU is config data, not an engine edit.
|
||||
// RTX PRO 6000 (SM120 / Blackwell Desktop) is a workstation card, not datacenter.
|
||||
hardware: [
|
||||
{ id: "rtx6000", label: "RTX PRO 6000", vram: "96GB", vendor: "nvidia" },
|
||||
{ id: "rtx6000", label: "RTX PRO 6000", vram: "96GB", vendor: "blackwell" },
|
||||
],
|
||||
|
||||
variants: [
|
||||
|
||||
@@ -8,7 +8,7 @@ export const config = {
|
||||
|
||||
// Model-specific GPUs the shared HARDWARE_CATALOG does not carry.
|
||||
hardware: [
|
||||
{ id: "h20", label: "H20", vram: "96GB", vendor: "nvidia" },
|
||||
{ id: "h20", label: "H20", vram: "96GB", vendor: "hopper" },
|
||||
],
|
||||
|
||||
variants: [
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
export const benchmarks = [
|
||||
{ match: { hw: "b300", pdMode: "unified", strategy: "balanced" } },
|
||||
{ match: { hw: "b300", pdMode: "unified", strategy: "low-latency" } },
|
||||
{ match: { hw: "b300", pdMode: "unified", strategy: "high-throughput" } },
|
||||
{ match: { hw: "b200", pdMode: "unified", strategy: "low-latency" } },
|
||||
{ match: { hw: "b200", pdMode: "unified", strategy: "balanced" } },
|
||||
{ match: { hw: "b200", pdMode: "unified", strategy: "high-throughput" } },
|
||||
{ match: { hw: "b200", pdMode: "unified", strategy: "long-context" } },
|
||||
{ match: { hw: "mi350x", pdMode: "unified", strategy: "balanced" } },
|
||||
{ match: { hw: "mi355x", pdMode: "unified", strategy: "balanced" } },
|
||||
{ match: { hw: "h100", pdMode: "unified", strategy: "low-latency" } },
|
||||
{ match: { hw: "h100", pdMode: "unified", strategy: "balanced" } },
|
||||
{ match: { hw: "h100", pdMode: "unified", strategy: "high-throughput" } },
|
||||
{ match: { hw: "h200", pdMode: "unified", strategy: "low-latency" } },
|
||||
{ match: { hw: "h200", pdMode: "unified", strategy: "balanced" } },
|
||||
{ match: { hw: "h200", pdMode: "unified", strategy: "high-throughput" } },
|
||||
{ match: { hw: "gb300", pdMode: "unified", strategy: "low-latency" } },
|
||||
{ match: { hw: "gb300", pdMode: "unified", strategy: "balanced" } },
|
||||
{ match: { hw: "gb300", pdMode: "unified", strategy: "high-throughput" } },
|
||||
{ match: { hw: "gb200", pdMode: "unified", strategy: "low-latency" } },
|
||||
{ match: { hw: "gb200", pdMode: "unified", strategy: "balanced" } },
|
||||
{ match: { hw: "gb200", pdMode: "unified", strategy: "high-throughput" } },
|
||||
];
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user