From 7dafacca494e2832a2f44309703db260828b979d Mon Sep 17 00:00:00 2001 From: Liangsheng Yin Date: Mon, 27 Jul 2026 08:37:30 -0700 Subject: [PATCH] docs(cookbook): add the Kimi-K3 serving cookbook (#32542) Co-authored-by: kpham-sgl Co-authored-by: Zijie Xia Co-authored-by: ispobock Co-authored-by: Mick Co-authored-by: Baizhou Zhang Co-authored-by: thomawan Co-authored-by: BBuf <1182563586@qq.com> --- .../references/authoring-reference.md | 41 +- .../Moonshotai/Kimi-K2.7-Code.mdx | 1 - .../autoregressive/Moonshotai/Kimi-K3.mdx | 403 ++++ docs_new/cookbook/autoregressive/intro.mdx | 2 +- docs_new/docs.json | 1 + docs_new/index.mdx | 179 ++ docs_new/scripts/check_cookbook_configs.mjs | 192 ++ docs_new/src/snippets/_deployment.jsx | 448 ++++- .../_kimi_k3_mamba_ratio_calculator.jsx | 331 +++ docs_new/src/snippets/_playground.jsx | 531 +++-- .../configs/deepseek-ai/deepseek-v4.jsx | 2 +- .../configs/meituan-longcat/longcat-2.0.jsx | 2 +- .../configs/moonshotai/kimi-k3-benchmarks.jsx | 23 + .../snippets/configs/moonshotai/kimi-k3.jsx | 1767 +++++++++++++++++ 14 files changed, 3649 insertions(+), 274 deletions(-) create mode 100644 docs_new/cookbook/autoregressive/Moonshotai/Kimi-K3.mdx create mode 100755 docs_new/scripts/check_cookbook_configs.mjs create mode 100644 docs_new/src/snippets/_kimi_k3_mamba_ratio_calculator.jsx create mode 100644 docs_new/src/snippets/configs/moonshotai/kimi-k3-benchmarks.jsx create mode 100644 docs_new/src/snippets/configs/moonshotai/kimi-k3.jsx diff --git a/.claude/skills/cookbook-add-model/references/authoring-reference.md b/.claude/skills/cookbook-add-model/references/authoring-reference.md index 3e70d54bc..e820aabef 100644 --- a/.claude/skills/cookbook-add-model/references/authoring-reference.md +++ b/.claude/skills/cookbook-add-model/references/authoring-reference.md @@ -4,7 +4,7 @@ Loaded on demand by the `cookbook-add-model` skill. This is the field-by-field contract for when the clone needs more than a rename. The two engine files are the canonical specs — read their headers first: -- [`_deployment.jsx`](../../../../docs_new/src/snippets/_deployment.jsx) — the 5-dim matrix widget; lists every config field. +- [`_deployment.jsx`](../../../../docs_new/src/snippets/_deployment.jsx) — the matrix widget; its header lists every config field. Dimensions are the legacy fixed five by default, or config-declared via `matchDims` / `overlayDims` (§2.1b). - [`_playground.jsx`](../../../../docs_new/src/snippets/_playground.jsx) — the diff-based override widget; lists the `playgroundFeatures` axes + the `AXIS_HANDLERS` interface. Engine extension (adding a new playground axis) lives in [engine-axis.md](engine-axis.md). @@ -34,7 +34,7 @@ the full contract): | `quantizations` | `{id, label}[]` | 3rd-dim option list. | | `strategies` | `{id, label}[]` | 4th-dim option list. Canonical ids: `low-latency` / `balanced` / `high-throughput` (never model-specific ids like `mtp`). **The count follows the page's operating points**: one recipe → a single `balanced`; two → `low-latency` + `high-throughput`; three → the full trio (the ideal). Tiers apply per (hw × variant × quant) combination — a single-recipe combination parks under its semantically honest tier (clear slant → that tier, e.g. DSv4's RTX 6000 → `low-latency`; no slant → `balanced`, e.g. Qwen3.5's Xeon); the page's list is the union and the engine greys unused chips per selection. Never invent a recipe just to fill chips. When two recipes differ by MTP / speculative decoding, the assignment is deterministic: spec ON → `low-latency`, spec OFF → `high-throughput` (at saturation the draft+verify overhead outweighs the speedup — same reason DSv4's high-throughput recipes disable MTP). The recurring markers in the other direction: dp-attention ON (MLA-attention models) and EP / DP+EP ON (MoE models) → `high-throughput`. | | `nodesOptions` | `{id, label}[]` | 5th-dim option list. The `id` MUST be `single` or `multi-N` — the engine parses N from the id for `--nnodes`. | -| `cells` | `{match, verified?, env, flags}[]` | One per supported (hw × variant × quant × strategy × nodes) combination. See §2.2. | +| `cells` | `{match, verified?, nnodes?, env, flags}[]` | One per supported (hw × match-dim) combination. See §2.2. `nnodes` supplies the node count when the config declares no `nodes` dim (default 1). | | `modelNames` | `{[key]: string}` | HF slug lookup. Keys are either `hw\|variant\|quant` (most specific) or `variant\|quant` (fallback). | | `placeholders` | `{[key]: {target, label, default?}}` | `{{KEY}}` interpolation map for command + curl. `target` is `'command'` or `'curl'`. Editable through the Env modal. | | `curl` | string | cURL template. Uses `{{MODEL_NAME}}` + placeholder keys. | @@ -52,7 +52,42 @@ the full contract): | `latencyPercentile` | `"Mean" \| "P50"` | Optional, **temporary**; the percentile the benchmark TTFT/TPOT values are. **Default `"P50"`** — the card renders `TTFT ()` / `TPOT ()`. Set `"Mean"` only for legacy data recorded as Mean (being re-measured to P50). A benchmarks entry may carry its own `latencyPercentile` to override the page value per cell (entry → config → `"P50"`). `tokens_per_sec_per_gpu` is stored as **total (in+out)/GPU** = `output tok/s/GPU × (isl+osl)/osl`, shown by the card as-is. | | `github` | `{owner?, repo?, issueTemplate?, cookbookModel?}` | Overrides for the "Submit verified cell" CTA in the playground. Defaults: `sgl-project/sglang` + `3-playground-verified-cell.yml` + `"deepseek-ai/deepseek-v4"`. Set `cookbookModel` to the model's HF id (`/`); it prefills the issue template's free-form `model` input when the issue opens. **Don't prune this block** — without it the engine falls back to `deepseek-ai/deepseek-v4` and submissions from your page get mislabeled. | -## 2.2 Author the 5-dim matrix (`cells[]`) +## 2.1b Custom dimensions (`matchDims` / `overlayDims`) + +The legacy shape above is a fixed five dimensions. A model whose axes don't fit +(no variant axis, a deployment-shape axis, an orthogonal feature toggle) declares +its own instead. Declaring `matchDims` replaces `variants`/`quantizations`/ +`strategies`/`nodesOptions` wholesale; omitting it keeps the legacy behaviour, so +existing pages need no change. + +| Field | Shape | Notes | +|---|---|---| +| `matchDims` | `{id, title, options}[]` | Rows that key the cell lookup. `hw` is always the implicit first dim, so cells match on (hw × these ids). Order is priority: lower rows adapt to higher ones. | +| `overlayDims` | `{id, title, default?, showWhen?, options}[]` | Rows that do NOT key the lookup — the picked option layers onto whichever cell matched. Use this for a knob that is orthogonal to the grid (speculative decoding, hierarchical cache): as a match dim it would multiply the cell count, as an overlay it costs nothing. | + +Option shape, both kinds: + +| Key | Meaning | +|---|---| +| `id`, `label` | as usual | +| `showWhen(sel)` | option is hidden unless the predicate accepts the current selection — this is how one row shows a different option set per mode | +| `disabled`, `disableReason` | `true` or a predicate over the selection; greys the option out and supplies the tooltip. Use it for a combination the server rejects, so the reader learns why instead of hitting a startup error | + +Overlay options additionally take `flags`, `env` and `hints` — each a literal array +or a function of the whole selection (so an "auto" value can resolve against another +row). `hints` render as `# ...` comment lines above the command, for setup the +launch line cannot express on its own. + +`_playground.jsx` sees the overlay dims in its `base`, so a playground axis can gate +on them with its own `showWhen(base)` — an axis whose feature the Deploy panel never +switched on is not rendered at all. Changing the Deploy selection resets every +playground axis back to inherit-from-base. + +> The overlay resolution rule is written in BOTH engines (snippets can't import each +> other); each copy is marked `MIRROR`. Change both or neither, or the Deploy command +> and the playground base silently disagree. + +## 2.2 Author the matrix (`cells[]`) Each cell describes one verified (or auto-estimated) launch recipe. diff --git a/docs_new/cookbook/autoregressive/Moonshotai/Kimi-K2.7-Code.mdx b/docs_new/cookbook/autoregressive/Moonshotai/Kimi-K2.7-Code.mdx index 686e019c3..cf2542662 100644 --- a/docs_new/cookbook/autoregressive/Moonshotai/Kimi-K2.7-Code.mdx +++ b/docs_new/cookbook/autoregressive/Moonshotai/Kimi-K2.7-Code.mdx @@ -3,7 +3,6 @@ title: Kimi-K2.7-Code description: "Deploy Kimi-K2.7-Code with SGLang for coding-focused agentic workflows, thinking output, tool calling, and multimodal input." metatags: description: "Deploy Kimi-K2.7-Code native multimodal agentic model with SGLang - reasoning, tool calling, and multimodal capabilities." -tag: NEW --- ## 1. Model Introduction diff --git a/docs_new/cookbook/autoregressive/Moonshotai/Kimi-K3.mdx b/docs_new/cookbook/autoregressive/Moonshotai/Kimi-K3.mdx new file mode 100644 index 000000000..84203f6f8 --- /dev/null +++ b/docs_new/cookbook/autoregressive/Moonshotai/Kimi-K3.mdx @@ -0,0 +1,403 @@ +--- +title: Kimi-K3 +description: "Deploy Moonshot AI's Kimi-K3 with SGLang — a 2.8T-parameter hybrid Mixture-of-Experts vision-language model (Kimi Delta Attention + MLA, 16/896 active experts) with NVIDIA and AMD recipes." +tag: NEW +--- + +## Deployment + + + + + +For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). + + + + + +```bash Command +docker pull lmsysorg/sglang:kimi-k3 # CUDA13 +docker pull lmsysorg/sglang:kimi-k3-cu12 # CUDA12 +docker pull lmsysorg/sglang-rocm:rocm720-mi35x-k3-20260727 # ROCM +``` + +These tags publish with the public K3 launch; until then, build from the Dockerfiles linked below. + +For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. + + + + + +If you do not want to use a Docker image, reproduce the dependency installation steps from the [CUDA 13 Dockerfile](https://github.com/DarkSharpness/sglang-kimi/blob/kimi-k3/docker/kimi_k3/kimi_k3_cu13.Dockerfile) or [CUDA 12 Dockerfile](https://github.com/DarkSharpness/sglang-kimi/blob/kimi-k3/docker/kimi_k3/kimi_k3_cu12.Dockerfile). + + + +Pick your hardware, then the deployment shape and operating point. Node count follows the hardware recipe (B200 2×8, GB200 4×4, H100 4×8, B300 1×8, H200 2×8, GB300 2×4, MI350X/MI355X 1×8), so it is not a separate choice. + +**PD Mode** — `Unified` serves prefill and decode together. `Prefill` / `Decode` split them into dedicated pools (see [PD disaggregation](#pd-disaggregation)); `Prefill` ships two strategies on the TP8 platforms, both chunked at 16k: `Default` (TP8) and `Long-Context` (`--pp-size 8 --tp-size 1`, see [Deep PP](#deep-pp-for-long-context-prefill)). + +**Strategy** — the operating point within that shape: + +- **Low-Latency** — plain TP, no DCP. For chat. +- **Balanced** — the accuracy-preserving default: TP16/DCP16 on B200/GB200, TP8/DCP8 on B300/GB300, TP8 ROCm/AITER on MI35x. +- **High-Throughput** — the large-scale lane: pick a **Cluster Size** and **Large-Scale Preset** in the Playground ([details](#large-scale-presets)). On H100/H200: Balanced plus `extra_buffer_lazy`. +- **Long-Context** — B200 only: TP8/PP2 splits KV and KDA state across two pipeline stages. + +**Spec Decode** — independent of the strategy: DSPARK layers onto any `pp_size == 1` recipe, proposing 7 draft tokens per step (tune in the Playground); DFLASH has no published draft checkpoint. Its win is largest on short interactive traffic and fades as the prompt grows. + + +`--mamba-full-memory-ratio` is the one sizing flag, computed live: set your average request length in the [Mamba ratio calculator](#mamba-ratio-calculator); everything else follows the panels, and the result is pinned into the command. + + +import { Deployment } from "/src/snippets/_deployment.jsx"; +import { config } from "/src/snippets/configs/moonshotai/kimi-k3.jsx"; +import { benchmarks } from "/src/snippets/configs/moonshotai/kimi-k3-benchmarks.jsx"; +import { KimiK3MambaRatioCalculator } from "/src/snippets/_kimi_k3_mamba_ratio_calculator.jsx"; + + + +### Mamba ratio calculator + + + + + +`--mamba-full-memory-ratio` is the ratio between the KDA state pool and the MLA KV pool. Every parameter below except `L` is read live from the Deploy panel and Playground selection; the balanced value is the per-request cost ratio: + +```text +ratio = (S + D) x state_bytes / (L x (mla_kv_bytes / DCP + draft_kv_bytes)) +``` + +- `S` — KDA state slots per request: `extra_buffer=5`, `extra_buffer_lazy=4`, `no_buffer=3`, disabled radix cache `=1`. `SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK` frees one slot on the extra-buffer strategies; with the overlap scheduler off (or `pp > 1`, which disables it) the track buffer costs one slot instead of two. +- `D` — verify intermediate states under speculative decoding: `0` when disabled, otherwise DSPARK block size + 1 (`8` at the default 7). ReplaySSM (`--enable-linear-replayssm-spec`) folds them into a per-slot ring, returning `D` to `0`. +- `state_bytes` — one state slot's bytes, from K3's fixed geometry, the attention-TP width, and the SSM dtype. +- `mla_kv_bytes` — one token's MLA latent KV bytes (KV-dtype dependent); DCP shards it across its ranks. The DSPARK draft model's KV (~1.4 KB per token) is replicated on every rank, so it enters flat — negligible without DCP, the same order as the sharded MLA share under DCP8. +- `L` — average total request length in tokens: input + output. + + + + + +## Advanced Features Playground + +The Playground is where you experiment with **SGLang features beyond the deployment matrix**. The Deploy panel above emits the recipes the SGLang team is converging on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. + +import { Playground } from "/src/snippets/_playground.jsx"; + + + +## 1. Model Introduction + +**Kimi-K3** is Moonshot AI's flagship hybrid MoE vision-language model: **2.8 trillion parameters**, **16 of 896 experts** active per token, roughly **2.5× the scaling efficiency of Kimi-K2**. The backbone interleaves **Kimi Delta Attention (KDA)** with MLA across 93 layers (plus Attention Residuals and Stable LatentMoE); serving supports image input and a **1M-token** window with prefix caching. Weights ship in **MXFP4**: the FlashInfer MXFP4 (trtllm-gen SiTU) runner serves them on Blackwell, Marlin (W4A16) elsewhere, MegaMoE for short-context batch throughput. + +K3 **always runs with thinking enabled**, with reasoning depth controlled by `reasoning_effort` (`low` / `high` / `max`; default `max`). + + +Kimi-K3 is Moonshot AI's first open-source model in the trillion-plus class; **full model weights +are scheduled to release by July 27, 2026**. The recipes on this page were validated on the +[`DarkSharpness/sglang-kimi`](https://github.com/DarkSharpness/sglang-kimi) fork — the HuggingFace +repository (`moonshotai/Kimi-K3`) and a public `lmsysorg/sglang` image with K3 support will be +available at launch. + +Every cell on this page is currently marked **Not Verified**: the recipes run, but none has a +serving round on the final weights and current code behind it. Re-measure throughput and accuracy +before you rely on any of them. + + +**Recommended generation:** `temperature=1.0`, `top_p=0.95`, `presence_penalty=0`, `frequency_penalty=0` (fixed by the model; informational — do not hardcode in sample code). + +**Resources:** [HuggingFace](https://huggingface.co/moonshotai/Kimi-K3) · [Kimi-K3 Quickstart](https://platform.kimi.ai/docs/guide/kimi-k3-quickstart). + +## 2. Configuration Tips + +**Memory: two pools, one flag.** K3 splits static memory into a worst-case-reserved **KDA state pool** (it sets the concurrency ceiling) and a paged **MLA KV pool**, divided by `--mamba-full-memory-ratio`. The command panel pins that flag to the [calculator](#mamba-ratio-calculator)'s output — set your average request length there; every other calculator input follows the panels. After boot, read back `max_total_num_tokens` (the KV side) and the admitted-request cap (the state side). + +Capacity levers, all in the Playground. Each trades precision or cache behavior for capacity — re-verify accuracy on your workload: + +| Lever | Effect | +|---|---| +| `--mamba-radix-cache-strategy extra_buffer_lazy` | 4 state slots per request instead of 5 | +| `--mamba-ssm-dtype bfloat16` | ~halves state bytes; with spec on, KDA verification falls back from the fused kernel to Triton | +| `--kv-cache-dtype fp8_e4m3` | halves KV bytes per token; under PD both roles must match at connect | +| `--mem-fraction-static` 0.90–0.92 | cheapest first win when the boot log shows a large idle `avail mem` | +| `SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1` | frees one more slot per request (experimental, under validation) | + +Speculation: DSPARK holds block size + 1 (= 8) intermediate states per request — the calculator folds this in — and an unset `--max-running-requests` resets to 48 under spec (the command panel reminds you; set it explicitly to raise). + +**MoE runner.** Leave `--moe-runner-backend` unset on Blackwell: FlashInfer MXFP4 (W4A8, prebuilt trtllm-gen SiTU kernels) when the cubin pool is installed, Marlin (W4A16) otherwise; H100/H200 pin Marlin. The published Docker images already provision the **SiTU cubin pool**; to install it independently, run the same flow as the Dockerfile: + +```bash +wget https://github.com/sgl-project/whl/releases/download/trtllm_gen_moe_cubin_20260617/trtllm_gen_moe_cubin_pool_20260617_v0613rc1.zip +sudo mkdir -p /opt/trtllm_gen_moe_cubin_pool +sudo unzip -q trtllm_gen_moe_cubin_pool_20260617_v0613rc1.zip -d /opt/trtllm_gen_moe_cubin_pool +export SGLANG_TRTLLM_GEN_MOE_CUBIN_POOL=/opt/trtllm_gen_moe_cubin_pool/trtllm_gen_moe_cubin_pool_20260617_v0613rc1 +``` + +Remaining kernel sources JIT once from the public `flashinfer` wheel (a few minutes, cached). + +**Attention backend.** Leave all three attention knobs unset on Blackwell: K3 resolves prefill, decode, and — under DSPARK — verification as a set (`trtllm_mla` across the board; `cutedsl_mla` takes decode and verification under DCP). On the non-DCP recipes, setting any one of the three cancels the auto-resolution for the others. H100/H200 pin `flashmla` for decode. + +**Context length.** `--context-length` bounds the longest accepted request plus some context-scaled buffers; it does not size the KV pool. For long context the lever that adds capacity is `fp8_e4m3` KV. + +**DSPARK.** Adds `--speculative-algorithm DSPARK` plus the draft checkpoint on top of the showing strategy. Leave `--speculative-draft-attention-backend` unset. No serving round on the final draft checkpoint has landed — measure against the same recipe running NOSPEC before adopting. + +**Per-platform notes:** + +| Platform | Topology | Notes | +|---|---|---| +| B300 1×8 | TP8 (+DCP8) | accuracy-first defaults on Low-Latency and Balanced | +| GB300 2×4 | TP8/DCP8 | MNNVL transport and cuMem auto-detected | +| B200 2×8 | TP16 (+DCP16); Long-Context TP8/PP2, 128K ctx, 8192 chunked prefill | DSPARK off on Long-Context (`pp_size == 1` required) | +| GB200 4×4 | TP16/DCP16 | MNNVL auto-detected | +| H200 2×8 | TP16/EP16 + symm-mem | same block on both ranks; export the cross-node NIC (`GLOO_SOCKET_IFNAME` / `NCCL_SOCKET_IFNAME`, `SGLANG_HOST_IP`); keep `NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1` | +| H100 4×8 | TP32/EP32, Marlin + FlashMLA | SM90a build of the K3 image; pin NCCL/Gloo to the same NIC on all nodes; least post-weight headroom (80 GB) | +| MI350X/MI355X 1×8 | TP8 ROCm/AITER | AITER A8W4 FlyDSL MoE, Triton attention, graph bs up to 256; DSPARK supported | + +**DCP notes** (Blackwell Balanced / High-Throughput): + +- DCP is the only axis that shards the TP-replicated MLA KV; Low-Latency skips it. +- Leave `--dcp-comm-backend` unset (fabric-resolved: `fi_a2a` on GB200/GB300, `a2a` on B200/B300). +- No `--enable-symm-mem` under DCP (force-disabled for decode-graph correctness). +- Explicit `tokenspeed_mla` force-rewrites `--kv-cache-dtype` to fp8; the default `cutedsl_mla` serves either dtype. +- Calculator ratios run well above 1 here (`r > 1` is legal): `bfloat16` state buys admission, `fp8` KV buys context. +- Don't add EP — its a2a buffers reclaim the KV that DCP buys. Compose only to measure. + +No cell has a serving round in this exact shape — treat them as starting points to verify. + +## 3. Advanced Usage + +### 3.1 Reasoning + +K3 always thinks; the `kimi_k3` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) separates that thinking from the final answer — thinking lands in `message.reasoning_content`, the answer in `message.content`. Control the reasoning depth with `reasoning_effort` (`low` / `high` / `max`; default `max`). + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") +resp = client.chat.completions.create( + model="moonshotai/Kimi-K3", + messages=[{"role": "user", "content": "What is 15% of 240?"}], + reasoning_effort="high", # "low" | "high" | "max" (default max) +) +msg = resp.choices[0].message +print("Reasoning:", getattr(msg, "reasoning_content", None)) +print("Answer:", msg.content) +``` + + + + + +```text Output +Pending update... +``` + + + +### 3.2 Tool Calling + +Enable the `kimi_k3` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. Because K3 is a thinking model, the follow-up turn may put text in `reasoning_content` as well as `content` — print both. + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") +tools = [{ + "type": "function", + "function": { + "name": "get_weather", + "description": "Get the current weather for a city", + "parameters": { + "type": "object", + "properties": {"city": {"type": "string"}}, + "required": ["city"], + }, + }, +}] +resp = client.chat.completions.create( + model="moonshotai/Kimi-K3", + messages=[{"role": "user", "content": "What's the weather in Beijing?"}], + tools=tools, +) +msg = resp.choices[0].message +print("Reasoning:", getattr(msg, "reasoning_content", None)) +print("Tool calls:", msg.tool_calls) +``` + + + + + +```text Output +Pending update... +``` + + + +### 3.3 HiCache (Hierarchical KV Caching) + +K3's hybrid HiCache tiers the paged MLA KV **and** the KDA/mamba state across L1 (GPU) / L2 (host) / L3 (Mooncake) — enable it from the **HiCache** card in the [Playground above](#playground) for long multi-turn workloads. + +- On the DCP recipes (Blackwell Balanced / High-Throughput), the host tiers are not fully DCP-aware yet: **L3 always, and L1+L2 with Spec Decode on, drop the DCP flags and run plain TP** (the command hints call it out — per-request KV capacity shrinks accordingly). L1+L2 NOSPEC keeps DCP. +- Low-Latency and the Hopper recipes take all tiers unchanged. + +### 3.4 PD Disaggregation + +PD splits prefill and decode into separate server groups; because K3 is hybrid, the transfer moves **both** the paged MLA KV and the KDA recurrent state. + +- **Transfer**: the cells emit **NiXL** (RDMA); Mooncake stays selectable in the Playground. +- **Ports**: prefill `30000`, decode `30100` (derived ZMQ/dist ranges must not collide on a shared host). The positional `8998` after `--prefill` must match `--disaggregation-bootstrap-port`, or only the decode worker registers. +- **Decode state pool**: chunk cache — one slot per request; `--mamba-radix-cache-strategy` is inert. Keep `--disaggregation-decode-extra-slots` pinned: unpinned it defaults to twice the batch below 32 requests and **zero** above. + +#### Deep PP for long-context prefill + +The `Long-Context` prefill strategy is `--pp-size 8 --tp-size 1`: pipeline P2P overlaps the next microbatch's compute, unlike TP/EP collectives, and each stage owns whole layers (a clean slice of KV and state). + +- Use all eight stages; a shallow split still pays the in-stage all-reduce and can lose to TP8. +- Pays only with several requests in flight (hence `Default` = TP8). +- DSPARK off (`pp_size == 1` required). +- Fan one prefill role out to several decode roles; budget for in-transfer KV on the decode side. + + + +```bash Command +python3 -m sglang_router.launch_router \ + --pd-disaggregation \ + --prefill http://:30000 8998 \ + --decode http://:30100 \ + --host 0.0.0.0 --port 8000 \ + --disable-circuit-breaker \ + --health-check-interval-secs 999999 +``` + + + +Clients then send requests to the router (`:8000`) instead of an individual role server. + +### 3.5 VLM Serving Profiles + +The open-source K3 serving contract currently supports **image input only** — its +processor rejects video and audio input. + +#### Recommended high-speed VLM + +The command panel now opens on the **B300 · Unified · Balanced** +recipe below. It makes the VLM-specific performance choices explicit: + +```bash Command +sglang serve \ + --trust-remote-code \ + --model-path moonshotai/Kimi-K3 \ + --tp-size 8 \ + --dcp-size 8 \ + --mem-fraction-static 0.85 \ + --mm-feature-transport cuda_ipc \ + --mm-processor-worker-num 2 \ + --mm-io-worker-num 16 \ + --reasoning-parser kimi_k3 \ + --tool-call-parser kimi_k3 \ + --host 0.0.0.0 \ + --port 30000 +``` + +- `--mm-feature-transport cuda_ipc` — single-node only: skips the CPU round trip, bounded pool (per-tensor CPU fallback when full), reserves up to `SGLANG_MM_FEATURE_CACHE_MB` on the base GPU. Multi-node recipes use CPU transport. +- 2 processor / 16 I/O workers are the measured defaults; more adds contention. +- Leave `--mm-attention-backend` unset — auto-selected, with a correctness fallback. +- Don't add `--mm-enable-dp-encoder`; K3 already shards images across TP ranks. + +#### Should ViT BCG be enabled? + +Keep ViT BCG **off** for general serving; enable `SGLANG_VIT_ENABLE_CUDA_GRAPH=1` only for ViT-only / EPD encoder workloads with recurring image shapes and spare HBM. + +- The win is confined to the encoder — no reliable end-to-end TTFT/TPOT gain in full-model serving. +- Each captured graph retains HBM (graph + per-entry metadata); measure on your own shapes. +- The default cache captures after two hits and falls back to eager above 6,144 tokens; do not enlarge it without measuring. + +#### Low-HBM VLM + +Use this profile when keeping HBM headroom matters more than peak concurrency. +It removes the 1 GiB CUDA IPC pool, keeps ViT BCG disabled, halves the context +window, caps concurrency, and lowers the static-memory target: + +```bash Command +SGLANG_VIT_ENABLE_CUDA_GRAPH=0 \ +sglang serve \ + --trust-remote-code \ + --model-path moonshotai/Kimi-K3 \ + --tp-size 8 \ + --context-length 65536 \ + --enable-symm-mem \ + --mem-fraction-static 0.82 \ + --mm-feature-transport cpu \ + --mm-processor-worker-num 2 \ + --mm-io-worker-num 16 \ + --reasoning-parser kimi_k3 \ + --tool-call-parser kimi_k3 \ + --host 0.0.0.0 \ + --port 30000 +``` + +`--mem-fraction-static 0.82` is a conservative B300 starting point, not a portable minimum: raise it toward `0.85` if startup reports insufficient memory; if HBM must go back to other workloads, reduce context/concurrency first. The precision levers (`fp8_e4m3` KV, `bfloat16` SSM state) save far more but stay accuracy-gated. + + + +### 3.6 Large-Scale Serving Presets (16–64 GPUs, Blackwell) + +**The KDA state pool is the concurrency ceiling** — DP, EP, and DCP do not shard it; only attention-TP width, SSM dtype, and cache strategy change the per-GPU bill. The MLA KV is cheap to shrink (fp8) or deduplicate (DCP). + +Two presets come out of this, at `N = 8k` GPUs: + +| Preset | What it trades | Pick it for | +|---|---|---| +| **Peak Throughput** — `dp = k`, attention-TP 8 | State shards 8-way. The per-step KDA all-reduce stays within one 8-GPU B200/B300 node, or spans two 4-GPU GB200/GB300 nodes over MNNVL. `--kv-cache-dtype fp8_e4m3` is load-bearing — bf16 KV does not fit 128 requests per replica. | Maximum sustained TPS — the default large-scale shape. | +| **Peak Capacity (+DCP8)** — `dp = k` + `--dcp-size 8` | Deduplicates the attention-TP group's MLA KV: concurrency ceiling +72% at the same engine throughput, ~1.8× ITL. | Context ≥ ~16K, or per-replica concurrency past 128. | + +- **Radix cache** is independent of the preset: for prefix-free traffic (offline batch, evals) switch it off (Playground's **Prefix Cache** card) — one state slot per request instead of 4–5. +- The fully data-parallel extreme (`--dp-size` = GPU count, attention-TP 1) — the shape behind the 64-GPU sweep's ~3K tok/s per GPU — is not a preset: 288 GB GPUs only, radix forced off, no head-to-head against the preset shape. + +The Peak Throughput preset at 32 GPUs on B200/B300 (4 nodes × 8; every node runs the same command with its own `--node-rank`). On GB200/GB300 the same 32-GPU shape uses 8 nodes × 4, and the Playground emits `--nnodes 8`: + +```bash Command +SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=20480 \ +sglang serve \ + --trust-remote-code \ + --model-path moonshotai/Kimi-K3 \ + --tp-size 32 --ep-size 32 \ + --enable-dp-attention --dp-size 4 --enable-dp-lm-head \ + --nnodes 4 --node-rank --dist-init-addr :20000 \ + --moe-a2a-backend megamoe --moe-runner-backend deep_gemm \ + --kv-cache-dtype fp8_e4m3 \ + --mamba-ssm-dtype bfloat16 \ + --mamba-radix-cache-strategy extra_buffer_lazy \ + --mem-fraction-static 0.92 \ + --reasoning-parser kimi_k3 --tool-call-parser kimi_k3 \ + --host 0.0.0.0 --port 30000 +``` + +Scale by holding the per-replica shape fixed and moving only the replica count; pool sizing rides the calculator-driven `--mamba-full-memory-ratio`, which folds in DP, DCP, precision, and speculation: + +| GPUs | B200/B300 nodes | GB200/GB300 nodes | `--tp-size` / `--ep-size` | `--dp-size` | +|---|---|---|---|---| +| 16 | 2×8 | 4×4 | 16 | 2 | +| 32 | 4×8 | 8×4 | 32 | 4 | +| 64 | 8×8 | 16×4 | 64 | 8 | + +For Peak Capacity, add `--dcp-size 8` and re-derive the pool split with the [Mamba ratio calculator](#mamba-ratio-calculator). + +Both presets are one click away in the [Playground above](#playground): pick a **Cluster Size** and a **Large-Scale Preset** and the full command composes onto whichever cell is showing. + +Decisions the preset already makes: + +- **MegaMoE on `deep_gemm`** — the fastest a2a backend; needs the SiTU cubin pool ([§2](#2-configuration-tips)). +- **SP-MoE and shared-expert overlap** engage automatically under EP a2a; the K3 all-reduce fusion does not. +- **Spec Decode follows the Deploy knob.** Acceptance thins at large batch; spec × EP × DP-attention is validated only at 8-GPU EP8 × DP2 (full GSM8K) — experimental at these scales. + + +No preset has a full serving round on final weights; the constants derive from measured single- and dual-node rounds plus a 64-GPU sweep. Validate throughput and accuracy on your workload before committing a fleet. + diff --git a/docs_new/cookbook/autoregressive/intro.mdx b/docs_new/cookbook/autoregressive/intro.mdx index 3731e8e3f..0e7a8993f 100644 --- a/docs_new/cookbook/autoregressive/intro.mdx +++ b/docs_new/cookbook/autoregressive/intro.mdx @@ -70,7 +70,7 @@ metatags: +
+ + failures.push(`${where}: ${msg}`); + +// ---------------------------------------------------------------- 1. MIRROR +// Compare the marked blocks with comments and whitespace normalized away, so +// wording may differ per file but the rule may not. +const mirrorBody = (file) => { + const src = readFileSync(join(SNIPPETS, file), "utf8"); + const start = src.indexOf("==== MIRROR"); + const end = src.indexOf("==== end MIRROR"); + if (start === -1 || end === -1) return null; + return src + .slice(src.indexOf("\n", start), end) + .split("\n") + .map((l) => l.trim()) + .filter((l) => l && !l.startsWith("//")) + .join(" ") + .replace(/\s+/g, " "); +}; + +const a = mirrorBody("_deployment.jsx"); +const b = mirrorBody("_playground.jsx"); +if (a === null) fail("_deployment.jsx", "MIRROR markers missing"); +if (b === null) fail("_playground.jsx", "MIRROR markers missing"); +if (a && b && a !== b) { + fail("MIRROR", "overlay resolution has drifted between the two engines"); + const [la, lb] = [a.split(" "), b.split(" ")]; + const i = la.findIndex((t, k) => t !== lb[k]); + fail("MIRROR", `first divergence near token ${i}: ` + + `_deployment "${la.slice(i, i + 8).join(" ")}" vs ` + + `_playground "${lb.slice(i, i + 8).join(" ")}"`); +} + +// `withOverlay` returns a clone, so object identity can never distinguish the +// current base cell from a true sibling. This previously made every cookbook +// show a spurious "matches … / switch base" hint before the reader changed +// anything. +const playgroundSource = readFileSync(join(SNIPPETS, "_playground.jsx"), "utf8"); +if (/\bmatchedCell\s*!==\s*baseCell\b/.test(playgroundSource)) { + fail("_playground.jsx", "sibling detection compares cloned cells by object identity"); +} + +// --------------------------------------------------------------- 3/4. Configs +// Configs are .jsx with a single `export const config` literal; import them +// through a data: URL so no temp file is needed. +const loadConfig = async (path) => { + const src = readFileSync(path, "utf8"); + const mod = await import( + "data:text/javascript," + encodeURIComponent(src) + ); + return mod.config; +}; + +// Every combination of match dims + overlay dims the reader can produce. +const selectionSpace = (config) => { + const dims = [ + { id: "hw", options: (config.supportedHardware || []).map((id) => ({ id })) }, + ...(config.matchDims || []), + ...(config.overlayDims || []), + ]; + let space = [{}]; + for (const d of dims) { + const next = []; + for (const partial of space) { + for (const opt of (d.options || [])) next.push({ ...partial, [d.id]: opt.id }); + } + space = next.length ? next : space; + if (space.length > 20000) return space.slice(0, 20000); // cheap blow-up guard + } + return space; +}; + +const walk = (dir) => readdirSync(dir, { withFileTypes: true }).flatMap((e) => + e.isDirectory() ? walk(join(dir, e.name)) + : (e.name.endsWith(".jsx") && !e.name.includes("benchmark") ? [join(dir, e.name)] : [])); + +for (const path of walk(CONFIGS)) { + const where = relative(join(SNIPPETS, ".."), path); + let config; + try { + config = await loadConfig(path); + } catch (e) { + fail(where, `does not parse as a module: ${e.message}`); + continue; + } + if (!config) { fail(where, "no `export const config`"); continue; } + + const custom = Array.isArray(config.matchDims); + + // A config either declares its own dims or carries the full legacy set — + // half of each means the engine silently renders a dimension nobody authored. + if (!custom) { + for (const k of LEGACY_DIMS) { + if (!Array.isArray(config[k])) fail(where, `legacy config is missing \`${k}\``); + } + } + + const matchIds = ["hw", ...(custom + ? config.matchDims.map((d) => d.id) + : LEGACY_DIMS.map((k) => ({ variants: "variant", quantizations: "quant", + strategies: "strategy", nodesOptions: "nodes" })[k]))]; + + for (const [i, cell] of (config.cells || []).entries()) { + const keys = Object.keys(cell.match || {}).sort(); + const want = [...matchIds].sort(); + if (keys.join(",") !== want.join(",")) { + fail(where, `cells[${i}].match keys [${keys}] != declared dims [${want}]`); + } + for (const dim of (config.matchDims || [])) { + const v = cell.match[dim.id]; + if (!(dim.options || []).some((o) => o.id === v)) { + fail(where, `cells[${i}].match.${dim.id}="${v}" is not an option of that dim`); + } + } + // Without a `nodes` dim the node count rides on the cell; a missing one + // silently degrades a multi-node recipe to single-node. + if (custom && !matchIds.includes("nodes") && cell.nnodes === undefined) { + fail(where, `cells[${i}] has no \`nnodes\` and the config declares no nodes dim`); + } + } + + for (const dim of (config.overlayDims || [])) { + const ids = (dim.options || []).map((o) => o.id); + if (dim.default !== undefined && !ids.includes(dim.default)) { + fail(where, `overlayDims.${dim.id}.default="${dim.default}" is not one of [${ids}]`); + } + } + + // Predicates and flag builders must survive every reachable selection. + const space = selectionSpace(config); + const probe = (fn, label) => { + for (const sel of space) { + try { fn(sel); } catch (e) { + fail(where, `${label} throws on ${JSON.stringify(sel)}: ${e.message}`); + return; + } + } + }; + for (const dim of [...(config.matchDims || []), ...(config.overlayDims || [])]) { + if (typeof dim.showWhen === "function") probe(dim.showWhen, `${dim.id}.showWhen`); + for (const opt of (dim.options || [])) { + const tag = `${dim.id}.${opt.id}`; + if (typeof opt.showWhen === "function") probe(opt.showWhen, `${tag}.showWhen`); + if (typeof opt.disabled === "function") probe(opt.disabled, `${tag}.disabled`); + for (const key of ["flags", "env", "hints"]) { + if (typeof opt[key] !== "function") continue; + probe((sel) => { + const out = opt[key](sel); + if (out !== undefined && !Array.isArray(out)) throw new Error(`${key} returned ${typeof out}, expected an array`); + for (const f of (out || [])) { + if (typeof f !== "string") throw new Error(`${key} yielded a non-string entry`); + if (/undefined|NaN/.test(f)) throw new Error(`${key} produced "${f}"`); + } + }, `${tag}.${key}`); + } + } + } +} + +if (failures.length) { + console.error(`FAIL (${failures.length})`); + for (const f of failures) console.error(" - " + f); + process.exit(1); +} +console.log("cookbook config check: OK"); diff --git a/docs_new/src/snippets/_deployment.jsx b/docs_new/src/snippets/_deployment.jsx index 8d86d6dc7..1dc0e145d 100644 --- a/docs_new/src/snippets/_deployment.jsx +++ b/docs_new/src/snippets/_deployment.jsx @@ -8,12 +8,29 @@ // supportedHardware hw ids shown in the catalog (subset of HARDWARE_CATALOG ∪ config.hardware) // hardware optional — per-model GPUs the shared HARDWARE_CATALOG lacks: // {id, label, vram, vendor}[] merged into the catalog at render -// (so a model-specific GPU never needs an engine-catalog edit) -// variants/quantizations/strategies/nodesOptions the 5-dim option lists -// (nodesOptions id is `single` or `multi-N` → --nnodes N) -// cells {match, verified?, env, flags}[] — one per -// (hw × variant × quant × strategy × nodes); env/flags are -// flat literals, only {{PLACEHOLDER}} subst applied +// (so a model-specific GPU never needs an engine-catalog edit); +// vendor picks the selector group: blackwell | hopper | amd +// variants/quantizations/strategies/nodesOptions LEGACY 4-dim option lists, +// used when `matchDims` is absent (nodesOptions id is +// `single` or `multi-N` → --nnodes N) +// matchDims optional — replaces the legacy four. {id, title, options}[] +// where each option is {id, label, showWhen?(sel), disabled?, +// disableReason?}. `hw` is always the implicit first dim. +// Cells are then keyed on (hw × ). +// overlayDims optional — rows that do NOT participate in cell lookup; the +// picked option layers onto the matched cell, so an orthogonal +// knob does not multiply the cell count. Same option shape plus +// `flags` / `env` / `hints` (each a literal array or a function +// of the whole selection), and a row-level `default` / `showWhen`. +// `hints` render as `# ...` lines above the command. +// cells {match, verified?, nnodes?, warn?, redirect?, env, flags}[] — one per +// (hw × match dims); env/flags are flat literals, only +// {{PLACEHOLDER}} subst applied. `nnodes` supplies the node +// count for configs with no `nodes` dim (default 1). `warn` +// renders as a ⚠️ banner under the cell's command; it may +// embed [label](#anchor) links. `redirect: true` renders the +// banner ALONE — no command, header, or copy buttons — for +// cells that only point somewhere else. // modelNames HF slug lookup, `hw|variant|quant` then `variant|quant` // placeholders {{KEY}} → {target: 'command'|'curl', label, default?} // curl cURL template (uses {{MODEL_NAME}} + placeholders) @@ -31,6 +48,8 @@ // multiNodeHints optional — {[hwId]: string[]} prepended as `# ...` lines // dockerImages optional — `docker run` image, keyed by `hw|quant` // then `hw`; falls back to `lmsysorg/sglang:dev` +// runModes optional — command output tabs to show (`python` and/or +// `docker`); defaults to both, in that order // github optional — "Submit verified cell" issue-template overrides // playgroundFeatures optional — consumed by _playground.jsx (see its header) // @@ -49,13 +68,15 @@ export const Deployment = ({ config, benchmarks }) => { // ==== 1. Hardware catalog (shared across cookbooks) ==== // VRAM is per-GPU on-chip memory, not per-module. const HARDWARE_CATALOG = { - nvidia: [ - { id: "h100", label: "H100", vram: "80GB" }, - { id: "h200", label: "H200", vram: "141GB" }, - { id: "b200", label: "B200", vram: "192GB" }, + blackwell: [ { id: "b300", label: "B300", vram: "288GB" }, - { id: "gb200", label: "GB200", vram: "192GB" }, { id: "gb300", label: "GB300", vram: "288GB" }, + { id: "b200", label: "B200", vram: "192GB" }, + { id: "gb200", label: "GB200", vram: "192GB" }, + ], + hopper: [ + { id: "h200", label: "H200", vram: "141GB" }, + { id: "h100", label: "H100", vram: "80GB" }, ], amd: [ { id: "mi300x", label: "MI300X", vram: "192GB" }, @@ -86,10 +107,12 @@ export const Deployment = ({ config, benchmarks }) => { }, title: { fontSize: "12px", fontWeight: "600", minWidth: "108px", flexShrink: 0, color: isDark ? "#e5e7eb" : "inherit" }, vendorRow: { display: "flex", alignItems: "center", gap: "6px" }, + // Fixed width so every row's chips start at the same x regardless of the + // group name ("BLACKWELL" is the widest). vendorLabel: { fontSize: "10px", fontWeight: "600", color: isDark ? "#9ca3af" : "#6b7280", - minWidth: "38px", textTransform: "uppercase", letterSpacing: "0.04em", + width: "68px", flexShrink: 0, textTransform: "uppercase", letterSpacing: "0.04em", }, // auto-fit + a real min width: columns wrap on narrow screens instead of // shrinking below their label (the old minmax(0,1fr) let buttons overlap on @@ -365,9 +388,72 @@ export const Deployment = ({ config, benchmarks }) => { }); // ==== 3. Pure helpers (no React state) ==== + // Two kinds of selector row: + // match dims participate in cell lookup (cell.match[dim] === sel[dim]) + // overlay dims never touch cell lookup; the picked option contributes flags + // on top of the matched cell (so an orthogonal knob like + // speculative decoding does not multiply the cell count) + // A config that declares neither keeps the legacy fixed 5-dim shape, so model + // pages written before this existed render unchanged. + const LEGACY_MATCH_DIMS = [ + { id: "variant", title: "Model Variant", optionsKey: "variants" }, + { id: "quant", title: "Quantization", optionsKey: "quantizations" }, + { id: "strategy", title: "Strategy", optionsKey: "strategies" }, + { id: "nodes", title: "Nodes", optionsKey: "nodesOptions" }, + ]; + // `hw` is always the first match dim; it has its own vendor-grouped renderer. + const matchDimSpecs = (config.matchDims || LEGACY_MATCH_DIMS).map((d) => ({ + ...d, + options: d.options || config[d.optionsKey] || [], + })); + const overlayDimSpecs = config.overlayDims || []; // DIMENSIONS is ordered by priority — higher-index dims adapt to lower-index // picks, never the reverse. Drives the grey-out/snap logic below. - const DIMENSIONS = ["hw", "variant", "quant", "strategy", "nodes"]; + const DIMENSIONS = ["hw", ...matchDimSpecs.map((d) => d.id)]; + + // An option is visible when it declares no `showWhen`, or its predicate accepts + // the current selection. Hidden options are excluded from snapping and from the + // grey-out scan, so a stale pick can never survive a dependent-row switch. + // ==== MIRROR in _playground.jsx — keep the two copies identical ==== + // Snippets cannot import each other, so the overlay-resolution rule is written + // twice. A divergence makes the Deploy command and the playground base disagree, + // which shows up as phantom +/- lines in the diff and no error anywhere. + // Guarded by docs_new/scripts/check_cookbook_configs.mjs. + const optionVisible = (opt, sel) => + typeof opt.showWhen !== "function" || opt.showWhen(sel); + const optionDisabled = (opt, sel) => + typeof opt.disabled === "function" ? opt.disabled(sel) : !!opt.disabled; + const visibleOptions = (spec, sel) => + (spec.options || []).filter((o) => optionVisible(o, sel)); + const rowVisible = (spec, sel) => + (typeof spec.showWhen !== "function" || spec.showWhen(sel)) && + visibleOptions(spec, sel).length > 0; + const overlayPick = (sel) => { + const picked = []; + for (const spec of (config.overlayDims || [])) { + if (!rowVisible(spec, sel)) continue; + const opt = (spec.options || []).find((o) => o.id === sel[spec.id]); + if (opt && !optionDisabled(opt, sel)) picked.push(opt); + } + return picked; + }; + const overlayPart = (sel, key) => { + const out = []; + for (const opt of overlayPick(sel)) { + const add = typeof opt[key] === "function" ? opt[key](sel) : opt[key]; + if (add) out.push(...add); + } + return out; + }; + // An overlay option may also REMOVE cell flags, declared as `stripPrefixes` + // (a static list, or a function of the selection). L3 uses it to drop the + // whole DCP operating point, which the server rejects with an L3 backend. + const overlayStrip = (cellFlags, sel) => { + const strip = overlayPart(sel, "stripPrefixes"); + if (!strip.length) return [...(cellFlags || [])]; + return (cellFlags || []).filter((f) => !strip.includes(f.split(/[\s=]/)[0])); + }; + // ==== end MIRROR ==== const findCell = (cells, sel) => cells.find((c) => DIMENSIONS.every((d) => c.match[d] === sel[d])); @@ -461,37 +547,74 @@ export const Deployment = ({ config, benchmarks }) => { valid[dim] = fallback ? fallback.match[dim] : want; } } + // Overlay dims ride along: they never key cells, so snapping must not drop + // them (it did — a strict-mode hash round-trip lost the spec default). + // Keep the parsed value when it names a real option, else the row default. + for (const spec of overlayDimSpecs) { + const want = parsed[spec.id]; + const opts = spec.options || []; + valid[spec.id] = opts.some((o) => o.id === want) + ? want + : spec.default ?? (opts[0] && opts[0].id) ?? ""; + } return valid; }; + // Lookup walks most-specific to least so a config that drops the variant/quant + // dims can key its HF slug on `hw` alone, or on the single "default" entry. const resolveModelName = (sel) => { - const triple = `${sel.hw}|${sel.variant}|${sel.quant}`; - const pair = `${sel.variant}|${sel.quant}`; - return config.modelNames[triple] ?? config.modelNames[pair] ?? ""; + const keys = [ + `${sel.hw}|${sel.variant}|${sel.quant}`, + `${sel.variant}|${sel.quant}`, + sel.hw, + "default", + ]; + for (const k of keys) { + const hit = config.modelNames[k]; + if (hit) return hit; + } + return ""; }; const interpolate = (text, env, modelName) => text.replace(/{{(\w+)}}/g, (_, key) => key === "MODEL_NAME" ? modelName : (env[key] ?? `{{${key}}}`)); + // Node count comes from the `nodes` dim when the config has one; without that + // dim it is a property of the cell itself (`nnodes`), since the deployment + // shape is then fixed by the hardware rather than picked by the reader. const parseNnodes = (id) => { if (id === "single") return 1; - const m = /^multi-(\d+)$/.exec(id); + const m = /^multi-(\d+)$/.exec(id || ""); return m ? parseInt(m[1], 10) : 1; }; + const cellNnodes = (cell, sel) => + sel.nodes !== undefined ? parseNnodes(sel.nodes) : (cell.nnodes || 1); + + // Role-specific serving ports for PD deployments — keep in sync with PD_PORTS + // in _playground.jsx, which the generated router command targets. Each role + // derives 5 ZMQ/dist ports from its --port, so the serve ports are spaced 100 + // apart to keep those ranges from overlapping on a same-host deployment. + const PD_SERVE_PORTS = { prefill: 30000, decode: 30100 }; + + // `flags` / `env` / `hints` may each be a function of the whole selection, so an + // "Auto" option can resolve against another row (draft tokens per strategy). + const overlayFlags = (sel) => overlayPart(sel, "flags"); + const overlayEnv = (sel) => overlayPart(sel, "env"); + const overlayHints = (sel) => overlayPart(sel, "hints"); // python mode → bare `sglang serve`; docker mode → wrapped in `docker run`. const renderCommand = (cell, sel, envValues, mode = "python") => { if (!cell) return "# No command available for the current selection."; const modelName = resolveModelName(sel); - const nnodes = parseNnodes(sel.nodes); + const nnodes = cellNnodes(cell, sel); const multinode = nnodes > 1; - const cellEnv = cell.env || []; - const flags = [...(cell.flags || [])]; + const cellEnv = [...(cell.env || []), ...overlayEnv(sel)]; + const flags = [...overlayStrip(cell.flags, sel), ...overlayFlags(sel)]; if (multinode) { // Insert the multi-node trio after the last parallelism flag, // falling back to right after --model-path. - const PARALLELISM_ANCHORS = ["--enable-dp-attention", "--dp", "--tp"]; + const PARALLELISM_ANCHORS = ["--enable-dp-attention", "--dp", "--tp-size", "--tp"]; let i = -1; for (const anchor of PARALLELISM_ANCHORS) { i = flags.findIndex((f) => f.split(/[\s=]/)[0] === anchor); @@ -504,6 +627,15 @@ export const Deployment = ({ config, benchmarks }) => { `--dist-init-addr {{NODE0_IP}}:20000`); } + const pdServePort = PD_SERVE_PORTS[sel.pdMode]; + if (pdServePort !== undefined) { + for (let j = 0; j < flags.length; j++) { + if (flags[j].split(/[\s=]/)[0] === "--port") { + flags[j] = `--port ${pdServePort}`; + } + } + } + let cmd; if (mode === "docker") { // Image keyed by `hw|quant` (most specific) then `hw`; `:dev` if unmapped. @@ -553,8 +685,14 @@ export const Deployment = ({ config, benchmarks }) => { cmd = `${envBlock}sglang serve \\\n${flagBlock}`; } - if (multinode && config.multiNodeHints && config.multiNodeHints[sel.hw]) { - const hint = config.multiNodeHints[sel.hw] + const hintLines = [ + ...overlayHints(sel), + ...(multinode && config.multiNodeHints && config.multiNodeHints[sel.hw] + ? config.multiNodeHints[sel.hw] + : []), + ]; + if (hintLines.length) { + const hint = hintLines .map((line) => (line.length ? "# " + line : "#")).join("\n"); cmd = `${hint}\n${cmd}`; } @@ -836,13 +974,19 @@ export const Deployment = ({ config, benchmarks }) => { return groups; }; + // Match dims seed from the first cell (authoring convention: put the flagship + // verified cell first). Overlay dims seed from their own `default`, or the + // first option, since no cell carries them. const initialSelectionFromCells = () => { const first = config.cells[0]; - if (!first) return Object.fromEntries(DIMENSIONS.map((d) => [d, ""])); - return { - hw: first.match.hw, variant: first.match.variant, quant: first.match.quant, - strategy: first.match.strategy, nodes: first.match.nodes, - }; + const sel = Object.fromEntries( + DIMENSIONS.map((d) => [d, first ? first.match[d] : ""]), + ); + for (const spec of overlayDimSpecs) { + const opts = spec.options || []; + sel[spec.id] = spec.default ?? (opts[0] && opts[0].id) ?? ""; + } + return sel; }; const placeholderDefaults = (schema) => { @@ -888,6 +1032,8 @@ export const Deployment = ({ config, benchmarks }) => { }; const [sel, setSel] = useState(() => initialSelectionFromCells()); + const INTERNAL_HASH_STATE_KEY = "__sglangDeployInternalHash"; + const DEPLOYMENT_COMPONENT_ID = "deployment-configurator"; useEffect(() => { const hydrate = () => { const raw = window.location.hash.replace(/^#/, ""); @@ -902,10 +1048,15 @@ export const Deployment = ({ config, benchmarks }) => { if (!touched) return; // Snap to a real cell if the hash named an impossible combo (stale link). setSel(validateSelection(config.cells, parsed)); - // Scroll the Deploy section into view. Heading slugs to "deployment" or - // "deploy"; only fires on hash navigation (not replaceState chip clicks). - const el = document.getElementById("deployment") - || document.getElementById("deploy"); + const historyState = window.history.state; + const isInternalHash = + historyState && + typeof historyState === "object" && + historyState[INTERNAL_HASH_STATE_KEY] === `#${raw}`; + if (isInternalHash) return; + // External selection hashes land on the interactive configurator. Hashes + // written internally while initializing or changing chips do not scroll. + const el = document.getElementById(DEPLOYMENT_COMPONENT_ID); if (el) el.scrollIntoView({ behavior: "smooth", block: "start" }); }; hydrate(); @@ -917,7 +1068,15 @@ export const Deployment = ({ config, benchmarks }) => { useEffect(() => { const target = "#" + new URLSearchParams(sel).toString(); if (window.location.hash !== target) { - window.history.replaceState(null, "", target); + const historyState = + window.history.state && typeof window.history.state === "object" + ? window.history.state + : {}; + window.history.replaceState( + { ...historyState, [INTERNAL_HASH_STATE_KEY]: target }, + "", + target + ); } window.dispatchEvent(new CustomEvent("sglang-deploy-sel", { detail: sel })); }, [sel]); @@ -943,29 +1102,123 @@ export const Deployment = ({ config, benchmarks }) => { const [benchConc, setBenchConc] = useState(null); const [benchAcc, setBenchAcc] = useState(null); const [benchCopied, setBenchCopied] = useState(null); - const [runMode, setRunMode] = useState("python"); // "python" | "docker" + const runModes = config.runModes || ["python", "docker"]; + const [runMode, setRunMode] = useState(runModes[0]); // "python" | "docker" useEffect(() => { if (modal === "env") setEnvDraft(env); }, [modal, env]); + // Live --mamba-full-memory-ratio from the ratio calculator (K3 pages): + // pool sizing is consolidated into this one flag, computed from the + // calculator's request length plus the current panel selection. + const [mambaRatio, setMambaRatio] = useState(null); + useEffect(() => { + // Deploy shows base flags only, so it takes the base-config ratio (the + // effective one belongs to the playground's composed command). + const onRatio = (e) => + setMambaRatio((e.detail && (e.detail.baseRatio || e.detail.ratio)) || null); + window.addEventListener("sglang-k3-mamba-ratio", onRatio); + return () => window.removeEventListener("sglang-k3-mamba-ratio", onRatio); + }, []); + // ==== 5. Derived values ==== const s = makeStyles(isDark); const cell = findCell(config.cells, sel); - const command = renderCommand(cell, sel, env, runMode); - // MTP hint: fire on the actual command (speculative decoding ON) — NOT on - // strategy=low-latency, since a low-latency cell may not enable MTP. SGLang - // resets --max-running-requests to 48 when spec is on and it's unset. + // Pin the calculator-computed ratio into the rendered command (before the + // host/port tail); cells themselves stay ratio-free. + const cellWithRatio = (() => { + if (!cell || !mambaRatio) return cell; + if (cell.flags.some((f) => f.startsWith("--mamba-full-memory-ratio"))) return cell; + const flags = [...cell.flags]; + const line = `--mamba-full-memory-ratio ${mambaRatio}`; + const i = flags.findIndex((f) => f.startsWith("--host")); + if (i >= 0) flags.splice(i, 0, line); + else flags.push(line); + return { ...cell, flags }; + })(); + const command = renderCommand(cellWithRatio, sel, env, runMode); + // MTP hint on the EFFECTIVE flags — speculation arrives via the Spec Decode + // overlay, never the cell. SGLang resets --max-running-requests to 48 when + // spec is on and it's unset. + const effFlags = cell ? [...overlayStrip(cell.flags, sel), ...overlayFlags(sel)] : []; const mtpHint = - !!cell && - (cell.flags || []).some((f) => f.split(/[\s=]/)[0] === "--speculative-algorithm") && - !(cell.flags || []).some((f) => f.split(/[\s=]/)[0] === "--max-running-requests"); + effFlags.some((f) => f.split(/[\s=]/)[0] === "--speculative-algorithm") && + !effFlags.some((f) => f.split(/[\s=]/)[0] === "--max-running-requests"); + // cell.warn may embed [label](#anchor) links — rendered as scrollIntoView + // buttons, not hrefs, so the hash (which carries the selection) isn't overwritten. + const renderWarn = (text) => { + const out = []; + const re = /\[([^\]]+)\]\(#([^)]+)\)/g; + let last = 0; + for (let m; (m = re.exec(text)); last = m.index + m[0].length) { + if (m.index > last) out.push(text.slice(last, m.index)); + const anchor = m[2]; + out.push( + + ); + } + if (last < text.length) out.push(text.slice(last)); + return out; + }; const modelName = resolveModelName(sel); const curlText = interpolate(config.curl || "", env, modelName); const hwGroups = buildHardwareGroups(); const benchEntry = benchmarks ? findBenchmark(benchmarks, sel) : null; - const isEnabled = (dim, value) => isOptionAvailable(config.cells, sel, dim, value); + // Overlay dims have no cells to constrain them, so an option is selectable + // unless it says otherwise; only match dims get the grey-out scan. + const isOverlayDim = (dim) => overlayDimSpecs.some((d) => d.id === dim); + const findOption = (dim, value) => { + const spec = [...matchDimSpecs, ...overlayDimSpecs].find((d) => d.id === dim); + return spec && (spec.options || []).find((o) => o.id === value); + }; + const isEnabled = (dim, value) => { + const opt = findOption(dim, value); + if (opt && optionDisabled(opt, sel)) return false; + return isOverlayDim(dim) || isOptionAvailable(config.cells, sel, dim, value); + }; + + // Switching a match dim can hide the option a dependent row currently holds + // (Strategy's option set differs per PD mode). Re-seat any overlay/match pick + // that just became invisible onto the first visible option of its row. + const reseatHiddenPicks = (next) => { + let out = next; + for (const spec of [...matchDimSpecs, ...overlayDimSpecs]) { + const opts = visibleOptions(spec, out).filter((o) => !optionDisabled(o, out)); + if (!opts.length) continue; + if (!opts.some((o) => o.id === out[spec.id])) { + out = { ...out, [spec.id]: opts[0].id }; + } + } + return out; + }; const handleSelect = (dim, value) => { - setSel((prev) => snapToValidCell(config.cells, prev, dim, value)); + setSel((prev) => + reseatHiddenPicks( + isOverlayDim(dim) + ? { ...prev, [dim]: value } + : snapToValidCell(config.cells, prev, dim, value), + ), + ); }; const handleCopy = () => { @@ -1006,7 +1259,11 @@ export const Deployment = ({ config, benchmarks }) => { ...(checked ? s.checked : {}), ...(disabled ? s.disabled : {}), }} - title={disabled ? "Not supported for current selection" : ""} + title={ + disabled + ? item.disableReason || "Not supported for current selection" + : "" + } onClick={(e) => { if (disabled) { e.preventDefault(); return; } handleSelect(dim, item.id); @@ -1035,7 +1292,11 @@ export const Deployment = ({ config, benchmarks }) => { const maxHwCols = Math.max(...hwGroups.map((x) => x.items.length)); return ( -
+
{/* Hardware section (2 vendor rows in one card, equal-width grid) */}
Hardware Platform
@@ -1052,54 +1313,69 @@ export const Deployment = ({ config, benchmarks }) => { ))}
- {renderFlatSection("Model Variant", config.variants, "variant", sel.variant)} - {renderFlatSection("Quantization", config.quantizations, "quant", sel.quant)} - {renderFlatSection("Strategy", config.strategies, "strategy", sel.strategy)} - {renderFlatSection("Nodes", config.nodesOptions, "nodes", sel.nodes)} + {matchDimSpecs + .filter((d) => rowVisible(d, sel)) + .map((d) => ( +
+ {renderFlatSection(d.title, visibleOptions(d, sel), d.id, sel[d.id])} +
+ ))} + {overlayDimSpecs + .filter((d) => rowVisible(d, sel)) + .map((d) => ( +
+ {renderFlatSection(d.title, visibleOptions(d, sel), d.id, sel[d.id])} +
+ ))} {/* Command box */}
-
Run this Command:
+
Command:
-
-
-
- - {cell && cell.verified ? "Verified" : "Not Verified"} + {cell && cell.redirect ? ( + cell.warn &&
⚠️ {renderWarn(cell.warn)}
+ ) : (<> +
+
+
+ + {cell && cell.verified ? "Verified" : "Not Verified"} +
+
+ {runModes.map((mode, index) => ( + setRunMode(mode)} + role="tab" + aria-selected={runMode === mode} + > + {mode === "docker" ? "Docker" : "Python"} + + ))} +
-
- setRunMode("python")} - role="tab" - aria-selected={runMode === "python"} - > - Python - - setRunMode("docker")} - role="tab" - aria-selected={runMode === "docker"} - > - Docker - +
+ + +
-
- - - -
-
- 
{command}
- {mtpHint && ( -
- ⚠️ Speculative decoding (MTP) is on — SGLang resets --max-running-requests to 48 when it isn't set. Add --max-running-requests <N> sized for your target concurrency. -
- )} + 
{command}
+ {cell && cell.warn &&
⚠️ {renderWarn(cell.warn)}
} + {mtpHint && ( +
+ ⚠️ Speculative decoding (MTP) is on — SGLang resets --max-running-requests to 48 when it isn't set. Add --max-running-requests <N> sized for your target concurrency. +
+ )} + )}
diff --git a/docs_new/src/snippets/_kimi_k3_mamba_ratio_calculator.jsx b/docs_new/src/snippets/_kimi_k3_mamba_ratio_calculator.jsx new file mode 100644 index 000000000..41088c1bb --- /dev/null +++ b/docs_new/src/snippets/_kimi_k3_mamba_ratio_calculator.jsx @@ -0,0 +1,331 @@ +// Kimi-K3-only calculator, live-coupled to the Deploy panel: every parameter +// except the average request length is derived from the effective config the +// Playground broadcasts (base cell + Deploy overlays + Playground overrides), +// and the computed --mamba-full-memory-ratio is broadcast back for the Deploy +// panel to pin into its command. + +export const KimiK3MambaRatioCalculator = () => { + const [isDark, setIsDark] = useState(false); + const [requestLength, setRequestLength] = useState("11264"); + const [copied, setCopied] = useState(false); + // Effective serving config; empty until the Playground's first broadcast + // (the parse below then falls back to the stock defaults: tp8, bf16 KV, + // fp32 state, extra_buffer, NOSPEC, no DCP). + const [cfg, setCfg] = useState({ flags: [], env: [], baseFlags: [], baseEnv: [] }); + + useEffect(() => { + const checkTheme = () => { + const html = document.documentElement; + setIsDark( + html.classList.contains("dark") || + html.getAttribute("data-theme") === "dark" || + html.style.colorScheme === "dark" + ); + }; + checkTheme(); + const observer = new MutationObserver(checkTheme); + observer.observe(document.documentElement, { + attributes: true, + attributeFilter: ["class", "data-theme", "style"], + }); + return () => observer.disconnect(); + }, []); + + useEffect(() => { + const onCfg = (e) => + setCfg({ + flags: (e.detail && e.detail.flags) || [], + env: (e.detail && e.detail.env) || [], + baseFlags: (e.detail && e.detail.baseFlags) || [], + baseEnv: (e.detail && e.detail.baseEnv) || [], + }); + window.addEventListener("sglang-k3-effective-config", onCfg); + return () => window.removeEventListener("sglang-k3-effective-config", onCfg); + }, []); + + const length = Number.parseFloat(requestLength); + + // Derive the serving parameters from a flag/env list and evaluate the balance + // formula (measured dual-pool balance), written as a per-request cost ratio: + // + // r = (S + D) x state_bytes / (L x per_token_kv_bytes) + // + // The state side (S main slots plus D verify intermediates) is per-GPU and + // never DCP-sharded. The KV side is per-GPU per logical token: DCP shards the + // MLA latent KV across its ranks, while the DSPARK draft model's own KV is + // replicated on every rank, so it stays a flat term. Without DCP the draft + // term is ~10% noise; once DCP shards MLA it is the same order as the MLA + // share, which is why it cannot be folded into a plain x dcp factor. + const derive = (flags, env) => { + const flagArg = (name) => { + for (const f of flags) { + const parts = f.split(/\s+/); + if (parts[0] === name) return parts[1]; + } + return null; + }; + const hasFlag = (name) => flags.some((f) => f.split(/[\s=]/)[0] === name); + + const tp = Number(flagArg("--tp-size")) || 8; + const dp = hasFlag("--enable-dp-attention") ? Number(flagArg("--dp-size")) || 1 : 1; + // KDA state and the replicated MLA KV both live per attention-TP group. + // PP needs no term: it splits the layers of both pools equally. + const attnTp = Math.max(1, Math.round(tp / dp)); + const dcp = Number(flagArg("--dcp-size")) || 1; + const kvDtype = flagArg("--kv-cache-dtype") === "fp8_e4m3" ? "fp8_e4m3" : "bfloat16"; + const specOn = hasFlag("--speculative-algorithm"); + const replaySpec = hasFlag("--enable-linear-replayssm-spec"); + // ReplaySSM removes the D intermediate states but does NOT pin the state + // dtype: an unset --mamba-ssm-dtype defaults to fp32 either way, and an + // explicit 16-bit state is accepted (warned for drift), so read the flag. + const ssmFlag = flagArg("--mamba-ssm-dtype"); + const ssmDtype = + ssmFlag === "bfloat16" || ssmFlag === "float16" ? ssmFlag : "float32"; + const radixOff = hasFlag("--disable-radix-cache"); + // "auto" (and anything unrecognized) resolves to extra_buffer. + const strategyFlag = flagArg("--mamba-radix-cache-strategy"); + const strategy = + strategyFlag === "no_buffer" || strategyFlag === "extra_buffer_lazy" + ? strategyFlag + : "extra_buffer"; + const skipLock = env.some((e) => e.startsWith("SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1")); + const pdRole = flagArg("--disaggregation-mode"); + // Pipeline parallelism is incompatible with the overlap scheduler, so pp > 1 + // turns it off for you — and the track buffer then costs one slot, not two. + const overlapOff = + hasFlag("--disable-overlap-schedule") || (Number(flagArg("--pp-size")) || 1) > 1; + // Mirrors kv_cache_configurator._calculate_mamba_ratio: base 3, minus 1 under + // the decode-lock skip, plus the ping-pong track buffer (2 under the overlap + // scheduler, 1 for lazy or without overlap). no_buffer has no track buffer and + // adds the skip's drop back, so it stays 3; a disabled radix cache is 1. + // A PD decode server runs a chunk cache: one live slot per request, and the + // radix-strategy knobs are inert there. + const slots = pdRole === "decode" + ? 1 + : radixOff + ? 1 + : strategy === "no_buffer" + ? 3 + : 3 - + (skipLock ? 1 : 0) + + (overlapOff || strategy === "extra_buffer_lazy" ? 1 : 2); + const block = Number(flagArg("--speculative-dspark-block-size")) || 7; + const drafts = specOn && !replaySpec && pdRole !== "prefill" ? block + 1 : 0; + + const ssmBytes = ssmDtype === "float32" ? 4 : 2; + const kvBytes = kvDtype === "fp8_e4m3" ? 1 : 2; + // Fixed K3 geometry: + // KDA: 69 layers, 96 heads, head_dim 128, conv kernel 4 (conv state always bf16). + // MLA: 24 layers, kv_lora_rank 512, qk_rope_head_dim 64. + const stateBytesPerSlot = + 69 * ((96 / attnTp) * 128 * 128 * ssmBytes + 3 * 3 * (96 / attnTp) * 128 * 2); + const kvBytesPerToken = 24 * (512 + 64) * kvBytes; + // DCP shards the MLA latent KV across its ranks; the DSPARK draft model's KV + // is replicated on every rank (~1.4 KB/token on trtllm_mha), so it does not + // shard and is added flat. + const draftKvBytesPerToken = specOn ? 1400 : 0; + const kvBytesPerTokenPerRank = kvBytesPerToken / dcp + draftKvBytesPerToken; + const ratio = + ((slots + drafts) * stateBytesPerSlot) / (kvBytesPerTokenPerRank * length); + return { ratio, tp, dp, attnTp, dcp, kvDtype, ssmDtype, radixOff, strategy, skipLock, slots, specOn, replaySpec, block, pdRole }; + }; + + // Two evaluations: `eff` matches the Playground's composed command, `bs` + // matches the Deploy command (cell + overlays only). + const eff = derive(cfg.flags, cfg.env); + const bs = derive(cfg.baseFlags.length ? cfg.baseFlags : cfg.flags, + cfg.baseFlags.length ? cfg.baseEnv : cfg.env); + const { ratio, tp, dp, attnTp, dcp, kvDtype, ssmDtype, radixOff, strategy, skipLock, slots, specOn, replaySpec, block, pdRole } = eff; + const valid = Number.isFinite(ratio) && ratio > 0 && length > 0 && 96 % attnTp === 0; + const baseValid = Number.isFinite(bs.ratio) && bs.ratio > 0 && length > 0; + + const formatRatio = (value) => { + if (!Number.isFinite(value)) return "—"; + if (value >= 10) return value.toFixed(1).replace(/\.0$/, ""); + if (value >= 1) return value.toFixed(2).replace(/\.?0+$/, ""); + return Number(value.toPrecision(2)).toString(); + }; + + const result = valid ? formatRatio(ratio) : "—"; + const baseResult = baseValid ? formatRatio(bs.ratio) : "—"; + const cliFlag = valid ? `--mamba-full-memory-ratio ${result}` : ""; + + // Broadcast both results: the Deploy command takes the base-config value, + // the Playground's composed command takes the effective one. + useEffect(() => { + window.dispatchEvent( + new CustomEvent("sglang-k3-mamba-ratio", { + detail: { + ratio: valid ? result : null, + baseRatio: baseValid ? baseResult : null, + }, + }) + ); + }, [result, valid, baseResult, baseValid]); + + const copyFlag = () => { + if (!cliFlag || typeof navigator === "undefined" || !navigator.clipboard) return; + navigator.clipboard.writeText(cliFlag); + setCopied(true); + window.setTimeout(() => setCopied(false), 1600); + }; + + const colors = { + border: isDark ? "#374151" : "#e5e7eb", + panel: isDark ? "#1f2937" : "#ffffff", + input: isDark ? "#111827" : "#f8fafc", + text: isDark ? "#e5e7eb" : "#1f2937", + muted: isDark ? "#9ca3af" : "#64748b", + accent: isDark ? "#E85D4D" : "#D45D44", + error: isDark ? "#fca5a5" : "#b91c1c", + }; + + const inputStyle = { + width: "100%", + boxSizing: "border-box", + padding: "8px 10px", + border: `1px solid ${colors.border}`, + borderRadius: "5px", + background: colors.input, + color: colors.text, + fontSize: "13px", + }; + const labelStyle = { + display: "flex", + flexDirection: "column", + gap: "5px", + fontSize: "12px", + fontWeight: 600, + }; + const chipStyle = { + padding: "3px 9px", + border: `1px solid ${colors.border}`, + borderRadius: "999px", + background: colors.input, + color: colors.text, + fontSize: "12px", + whiteSpace: "nowrap", + }; + + const specLabel = !specOn + ? "NOSPEC" + : replaySpec + ? "DSPARK + ReplaySSM (D folded)" + : `DSPARK (D = ${block + 1})`; + const derivedChips = [ + // With DP attention on, show the whole topology so a large-scale preset is + // visibly understood: total GPUs = DP replicas x attention-TP group width. + dp > 1 + ? `${tp} GPUs = DP ${dp} × Attention TP ${attnTp}` + : `Attention TP ${attnTp}`, + `DCP ${dcp}`, + `KV ${kvDtype === "fp8_e4m3" ? "FP8" : "BF16"}`, + `State ${ssmDtype === "float32" ? "FP32" : ssmDtype === "bfloat16" ? "BF16" : "FP16"}`, + pdRole === "decode" + ? "PD decode: chunk cache (S = 1)" + : radixOff + ? "Radix off (S = 1)" + : `${strategy}${skipLock ? " + slot saving" : ""} (S = ${slots})`, + pdRole === "prefill" ? "PD prefill (no verify states)" : null, + specLabel, + ].filter(Boolean); + + return ( +
+
+ + +
+ + Serving configuration (follows the Deploy panel and Playground) + +
+ {derivedChips.map((c) => ( + {c} + ))} +
+
+
+ + {!valid ? ( +
+ Enter a valid request length. +
+ ) : ( +
+
+
+ Balanced ratio — pinned into the commands above +
+
{result}
+ {baseResult !== result ? ( +
+ Deploy command (without Playground overrides): {baseResult} +
+ ) : null} +
+ + {cliFlag} + + +
+ )} +
+ ); +}; diff --git a/docs_new/src/snippets/_playground.jsx b/docs_new/src/snippets/_playground.jsx index be8a0a28a..af6875931 100644 --- a/docs_new/src/snippets/_playground.jsx +++ b/docs_new/src/snippets/_playground.jsx @@ -10,6 +10,17 @@ // moe — backend (+ MegaMoE quantization sub-select) + EP // parsers — per-item toggle flags // speculative — single-select preset +// +// Axis-level `showWhen(base)` (any axis): the card is not rendered when the Deploy +// panel has not switched that feature on. `base` carries the cell match dims plus +// the Deploy panel's overlay dims, so an axis can gate on either. +// flagSelects extras: `control: "slider"` renders the option list as a range input +// (option order is the scale), and a per-select `showWhen(base)` gates one row. +// A row may set `default` (initial pick instead of inherit) and options may carry +// `env` (stripped/re-added like the MoE backend card) or `flags` as a FUNCTION +// `(rowValues, base) => [...]` for cross-row composition (e.g. preset x cluster +// size); a function returning null means "leave the base untouched". +// Changing the Deploy selection resets every axis back to inherit-from-base. // pdDisagg — role + transfer backend + IB device + optional router // hicache — enable + backend + write policy // hisparse — enable + host ratio (decode-only) @@ -17,7 +28,7 @@ // its own title + strip-prefixes + options (no per-feature code) // // Adding an axis = one entry in AXIS_HANDLERS below; nothing else switches on -// an axis id. Each handler implements initState / revertHidden / apply / +// an axis id. Each handler implements initState / apply / // render, plus optional deriveFromBase (recover state from base cell flags) // and getRenderHints. // @@ -34,7 +45,60 @@ export const Playground = ({ config }) => { // ========================================================================== // 1. Constants // ========================================================================== - const DIMENSIONS = ["hw", "variant", "quant", "strategy", "nodes"]; + const DIMENSIONS = ["hw", ...((config.matchDims + || [{ id: "variant" }, { id: "quant" }, { id: "strategy" }, { id: "nodes" }]) + .map((d) => d.id))]; + + // The Deploy panel layers its overlay dims (speculation, hicache, ...) on top of + // the matched cell, so the playground's "base" is cell + overlay — otherwise the + // diff shows overlay flags as playground additions and deriveFromBase can't see + // what Deploy already resolved. + // + // ==== MIRROR in _deployment.jsx — keep the two copies identical ==== + // Snippets cannot import each other, so the overlay-resolution rule is written + // twice. A divergence makes the Deploy command and the playground base disagree, + // which shows up as phantom +/- lines in the diff and no error anywhere. + // Guarded by docs_new/scripts/check_cookbook_configs.mjs. + const optionVisible = (opt, sel) => + typeof opt.showWhen !== "function" || opt.showWhen(sel); + const optionDisabled = (opt, sel) => + typeof opt.disabled === "function" ? opt.disabled(sel) : !!opt.disabled; + const visibleOptions = (spec, sel) => + (spec.options || []).filter((o) => optionVisible(o, sel)); + const rowVisible = (spec, sel) => + (typeof spec.showWhen !== "function" || spec.showWhen(sel)) && + visibleOptions(spec, sel).length > 0; + const overlayPick = (sel) => { + const picked = []; + for (const spec of (config.overlayDims || [])) { + if (!rowVisible(spec, sel)) continue; + const opt = (spec.options || []).find((o) => o.id === sel[spec.id]); + if (opt && !optionDisabled(opt, sel)) picked.push(opt); + } + return picked; + }; + const overlayPart = (sel, key) => { + const out = []; + for (const opt of overlayPick(sel)) { + const add = typeof opt[key] === "function" ? opt[key](sel) : opt[key]; + if (add) out.push(...add); + } + return out; + }; + // An overlay option may also REMOVE cell flags, declared as `stripPrefixes` + // (a static list, or a function of the selection). L3 uses it to drop the + // whole DCP operating point, which the server rejects with an L3 backend. + const overlayStrip = (cellFlags, sel) => { + const strip = overlayPart(sel, "stripPrefixes"); + if (!strip.length) return [...(cellFlags || [])]; + return (cellFlags || []).filter((f) => !strip.includes(f.split(/[\s=]/)[0])); + }; + // ==== end MIRROR ==== + const withOverlay = (cell, sel) => (cell && { + ...cell, + flags: [...overlayStrip(cell.flags, sel), ...overlayPart(sel, "flags")], + env: [...(cell.env || []), ...overlayPart(sel, "env")], + }) || cell; // Shared with `_deployment.jsx` (HOST/PORT/etc. unified across the page). const STORAGE_KEY = "sglang-deploy-env"; @@ -56,9 +120,11 @@ export const Playground = ({ config }) => { cells.find((c) => DIMENSIONS.every((d) => c.match[d] === sel[d])); // After applying overrides, the resulting (env, flags) may equal another - // cell sharing the same (hw, variant, quant, nodes) but a different - // strategy. flags compared ordered; env compared as a set. + // cell sharing every match dim except strategy. Keep this generic so custom + // grids (for example K3's PD mode) cannot match across unrelated base cells. + // flags compared ordered; env compared as a set. const findMatchingCell = (cells, sel, pgEnv, pgFlags) => { + const fixedDims = DIMENSIONS.filter((d) => d !== "strategy"); const flagsEq = (a, b) => a.length === b.length && a.every((x, i) => x === b[i]); const envEq = (a, b) => { @@ -68,10 +134,7 @@ export const Playground = ({ config }) => { return true; }; for (const c of cells) { - if (c.match.hw !== sel.hw) continue; - if (c.match.variant !== sel.variant) continue; - if (c.match.quant !== sel.quant) continue; - if (c.match.nodes !== sel.nodes) continue; + if (fixedDims.some((d) => c.match[d] !== sel[d])) continue; if (flagsEq(c.flags || [], pgFlags || []) && envEq(c.env || [], pgEnv || [])) { return c; } @@ -236,20 +299,37 @@ export const Playground = ({ config }) => { return null; }; + // --tp / --ep spelling families: configs write either the canonical + // --tp-size / --ep-size or the short --tp / --ep. Parse and strip every + // spelling; when re-emitting, keep the spelling the base already uses. + const TP_HEADS = ["--tp-size", "--tp", "--tensor-parallel-size"]; + const EP_HEADS = ["--ep-size", "--ep", "--expert-parallel-size"]; + const parseIntFlagAny = (flags, heads) => { + for (const head of heads) { + const n = parseIntFlag(flags, head); + if (n !== null) return n; + } + return null; + }; + const flagSpelling = (flags, heads, fallback) => + heads.find((head) => + (flags || []).some((f) => f.split(/[\s=]/)[0] === head)) || fallback; + // Insertion-anchor sets (priority-ordered; each includes siblings so // insertion still works in partial cells). const ANCHOR_NEAR_MODEL_PATH = ["--model-path"]; - const ANCHOR_NEAR_TP = ["--tp", "--model-path"]; - const ANCHOR_NEAR_DP = ["--dp", "--tp", "--model-path"]; - const ANCHOR_NEAR_DPATTN = ["--enable-dp-attention", "--dp", "--tp", "--model-path"]; + const ANCHOR_NEAR_TP = ["--tp-size", "--tp", "--model-path"]; + const ANCHOR_NEAR_DP = ["--dp", "--tp-size", "--tp", "--model-path"]; + const ANCHOR_NEAR_DPATTN = ["--enable-dp-attention", "--dp", "--tp-size", "--tp", "--model-path"]; const ANCHOR_NEAR_MOE = ["--moe-a2a-backend", "--moe-runner-backend", - "--enable-dp-attention", "--dp", "--tp", "--model-path"]; + "--enable-dp-attention", "--dp", "--tp-size", "--tp", "--model-path"]; // Helper bundle passed to every axis handler. const helpers = { matchConstraint, evaluateChip, findEntry, isHidden, stripFlagsByFirstToken, stripEnvByPrefix, insertBeforeTail, insertAfter, parseIntFlag, hasFlag, findFlagArg, + TP_HEADS, EP_HEADS, parseIntFlagAny, flagSpelling, ANCHOR_NEAR_MODEL_PATH, ANCHOR_NEAR_TP, ANCHOR_NEAR_DP, ANCHOR_NEAR_DPATTN, ANCHOR_NEAR_MOE, }; @@ -288,7 +368,7 @@ export const Playground = ({ config }) => { // ========================================================================== // 5. AXIS_HANDLERS — the built-in playground axis registry // ========================================================================== - // Each entry implements initState / revertHidden / apply / render (plus + // Each entry implements initState / apply / render (plus // optional deriveFromBase / getRenderHints). Iterated in insertion order; // axes absent from config.playgroundFeatures are skipped. // - inherit-from-base sentinels: null / "current" / "auto" / "off" / @@ -322,26 +402,13 @@ export const Playground = ({ config }) => { else dpAttn = false; const cpSize = h.parseIntFlag(flags, "--attn-cp-size"); return { - tp: h.parseIntFlag(flags, "--tp"), + tp: h.parseIntFlagAny(flags, h.TP_HEADS), cp: cpEnabledIn(flags) ? (cpSize !== null ? cpSize : 2) : null, cpStrategy: bakedCpStrategy(flags), dpAttn, }; }, - revertHidden: (value, fc, base, h) => { - let changed = false; - const next = { ...value }; - for (const knob of (fc.knobs || [])) { - const cur = next[knob.id]; - if (cur !== null && cur !== undefined - && h.isHidden(knob.values, cur, base)) { - next[knob.id] = null; changed = true; - } - } - return changed ? next : value; - }, - apply: ({ flags, env, value, fc, sel, h }) => { // Live facts for constraint checks (same keys the render-side // constraintBase exposes), recomputed after each mutation. @@ -351,7 +418,7 @@ export const Playground = ({ config }) => { dpAttnOn: h.hasFlag(flags, "--enable-dp-attention"), cpOn: cpEnabledIn(flags), cpStrategy: bakedCpStrategy(flags) || "interleave", - effTp: h.parseIntFlag(flags, "--tp"), + effTp: h.parseIntFlagAny(flags, h.TP_HEADS), }); // The runtime derives the prefill-CP size as attn_cp_size = tp/dp // (a mismatched --attn-cp-size is overridden), so with DP-Attention @@ -366,7 +433,7 @@ export const Playground = ({ config }) => { : (h.hasFlag(flags, "--enable-dp-attention") ? (h.parseIntFlag(flags, "--dp") ?? 1) : false); if (typeof dpIntent === "number" && dpIntent > 1) return null; - return h.parseIntFlag(flags, "--tp"); + return h.parseIntFlagAny(flags, h.TP_HEADS); }; // Skip a knob whose entry or picked value is hidden/disabled under // the live facts — mirrors the grayed controls, so stale state never @@ -389,8 +456,9 @@ export const Playground = ({ config }) => { // below the command box) rather than banned. if (value.tp !== null && !blocked("tp", value.tp)) { - flags = h.stripFlagsByFirstToken(flags, ["--tp"]); - flags = h.insertAfter(flags, h.ANCHOR_NEAR_MODEL_PATH, [`--tp ${value.tp}`]); + const tpHead = h.flagSpelling(flags, h.TP_HEADS, "--tp"); + flags = h.stripFlagsByFirstToken(flags, h.TP_HEADS); + flags = h.insertAfter(flags, h.ANCHOR_NEAR_MODEL_PATH, [`${tpHead} ${value.tp}`]); } // CP override: an explicit size pick, or a strategy-only pick on a // base that already carries CP. Strategy precedence: explicit knob > @@ -521,43 +589,29 @@ export const Playground = ({ config }) => { (e) => e.startsWith("SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS")); return { backend: a2a || runner || null, - ep: h.parseIntFlag(flags, "--ep"), + ep: h.parseIntFlagAny(flags, h.EP_HEADS), mmQuant: fp4Acts ? "w4a4" : "w4a8", }; }, - revertHidden: (value, fc, base, h) => { - let changed = false; - const next = { ...value }; - if (next.backend !== null && fc.backend?.options - && h.isHidden(fc.backend.options, next.backend, base)) { - next.backend = null; changed = true; - } - // MegaMoE backend availability — gated by its option's requiresHw / - // excludesStrategy (this model gates by hw only; the check is generic). - const mmOpt = (fc.backend?.options || []).find((o) => o.id === "megamoe"); - const mmAvail = !!mmOpt - && (!mmOpt.requiresHw || mmOpt.requiresHw.includes(base.hw)) - && (!mmOpt.excludesStrategy || !mmOpt.excludesStrategy.includes(base.strategy)); - if (next.backend === "megamoe" && !mmAvail) { - next.backend = null; changed = true; - } - if (next.ep !== null && fc.ep?.values - && h.isHidden(fc.ep.values, next.ep, base)) { - next.ep = null; changed = true; - } - return changed ? next : value; - }, - apply: ({ flags, env, value, fc, h, derived }) => { if (value.backend !== null) { flags = h.stripFlagsByFirstToken(flags, [ "--moe-a2a-backend", "--moe-runner-backend", ]); + // Backend options may carry their own env (e.g. the FlashInfer MXFP4 + // cubin-pool path): strip every backend option's env keys, then + // re-add the selected option's. + const backendEnvKeys = []; + for (const o of (fc.backend?.options || [])) { + for (const e of (o.env || [])) backendEnvKeys.push(e.split("=")[0]); + } + if (backendEnvKeys.length) env = h.stripEnvByPrefix(env, backendEnvKeys); const opt = (fc.backend?.options || []).find((o) => o.id === value.backend); if (opt?.flags?.length) { flags = h.insertAfter(flags, h.ANCHOR_NEAR_DPATTN, opt.flags); } + if (opt?.env?.length) env = [...env, ...opt.env]; } // MegaMoE owns the MoE path: when the effective backend is megamoe, strip the // DeepEP dispatch + any prior megamoe env, then re-add the selected quant's @@ -582,9 +636,10 @@ export const Playground = ({ config }) => { } } if (value.ep !== null) { - flags = h.stripFlagsByFirstToken(flags, ["--ep"]); + const epHead = h.flagSpelling(flags, h.EP_HEADS, "--ep"); + flags = h.stripFlagsByFirstToken(flags, h.EP_HEADS); if (value.ep > 1) { - flags = h.insertAfter(flags, h.ANCHOR_NEAR_MOE, [`--ep ${value.ep}`]); + flags = h.insertAfter(flags, h.ANCHOR_NEAR_MOE, [`${epHead} ${value.ep}`]); } } return { flags, env }; @@ -665,18 +720,6 @@ export const Playground = ({ config }) => { return out; }, - revertHidden: (value, fc, base, h) => { - let changed = false; - const next = { ...value }; - for (const item of (fc.items || [])) { - if (next[item.id] !== null && next[item.id] !== undefined - && h.evaluateChip(item, base).hidden) { - next[item.id] = null; changed = true; - } - } - return changed ? next : value; - }, - apply: ({ flags, env, value, fc, h, derived }) => { const items = fc.items || []; // Effective state per item: explicit > derived > false. Skip @@ -755,13 +798,6 @@ export const Playground = ({ config }) => { return "current"; }, - revertHidden: (value, fc, base, h) => { - if (value !== "current" && h.isHidden(fc.options || [], value, base)) { - return "current"; - } - return value; - }, - apply: ({ flags, env, value, fc, h, derived }) => { if (value === "current") return { flags, env }; // No-op when the pick already matches base (preserves flag position). @@ -815,39 +851,41 @@ export const Playground = ({ config }) => { // Owns the `--disaggregation-*` flags (unconditional strip). A backend may // carry hw-gated env (transferBackends[].env + .envWhen). pdDisagg: { - initState: () => ({ mode: "off", transferBackend: "mooncake", ibDevice: "auto" }), - - revertHidden: (value, fc, base, h) => { - let changed = false; - const next = { ...value }; - if (next.mode !== "off" && fc.modes - && h.isHidden(fc.modes, next.mode, base)) { - next.mode = "off"; changed = true; - } - if (next.ibDevice !== "auto" && fc.ibDevices - && h.isHidden(fc.ibDevices, next.ibDevice, base)) { - next.ibDevice = "auto"; changed = true; - } - if (next.transferBackend !== "mooncake" && fc.transferBackends - && h.isHidden(fc.transferBackends, next.transferBackend, base)) { - next.transferBackend = "mooncake"; changed = true; - } - return changed ? next : value; - }, + // The transport default is the config's first entry, so a model whose + // recipes standardize on one backend does not silently start on another. + initState: (fc) => ({ + mode: "off", + transferBackend: (fc && (fc.transferBackends || [])[0] || {}).id || "mooncake", + ibDevice: "auto", + }), apply: ({ flags, env, value, sel, fc, h }) => { + // The bootstrap port is the base cell's to choose — the router's + // --prefill positional has to match it — so carry it across the strip + // rather than dropping it and silently falling back to the default. + const bootstrapPort = h.findFlagArg(flags, "--disaggregation-bootstrap-port"); flags = h.stripFlagsByFirstToken(flags, [ "--disaggregation-mode", "--disaggregation-transfer-backend", "--disaggregation-ib-device", "--disaggregation-bootstrap-port", ]); const backends = fc.transferBackends || []; + // A config that omits `modes` has the role on the Deploy panel instead; + // this card then only tunes the transport for whatever role is selected. + const mode = (fc.modes || []).length + ? value.mode + : ((sel && sel.pdMode) || "off"); - if (value.mode === "prefill" || value.mode === "decode") { - const backend = value.transferBackend || "mooncake"; + if (mode === "prefill" || mode === "decode") { + const backend = value.transferBackend || (backends[0] || {}).id || "mooncake"; const adds = [ - `--disaggregation-mode ${value.mode}`, + `--disaggregation-mode ${mode}`, `--disaggregation-transfer-backend ${backend}`, ]; + // Re-emitted in the base cell's own order, so an untouched Prefill + // recipe renders byte-identical to the Deploy panel. + if (bootstrapPort) { + adds.push(`--disaggregation-bootstrap-port ${bootstrapPort}`); + } if (value.ibDevice && value.ibDevice !== "auto") { adds.push(`--disaggregation-ib-device ${value.ibDevice}`); } @@ -859,7 +897,7 @@ export const Playground = ({ config }) => { // Role-specific serving port so the router's prefill / decode targets // line up (and prefill+decode don't collide on a single host). - const servePort = PD_PORTS[value.mode].serve; + const servePort = PD_PORTS[mode].serve; flags = flags.map((f) => f.split(/[\s=]/)[0] === "--port" ? `--port ${servePort}` : f); @@ -930,14 +968,6 @@ export const Playground = ({ config }) => { hisparse: { initState: (fc) => ({ enable: false, hostRatio: (fc && fc.defaultHostRatio) || null }), - revertHidden: (value, fc, base, h) => { - if (value.hostRatio !== null && fc.hostRatios - && h.isHidden(fc.hostRatios, value.hostRatio, base)) { - return { ...value, hostRatio: (fc && fc.defaultHostRatio) || null }; - } - return value; - }, - apply: ({ flags, env, value, fc, h }) => { const ownedHeads = [ "--enable-hisparse", "--hisparse-config", @@ -969,10 +999,12 @@ export const Playground = ({ config }) => {
HiSparse - - {renderChip("Enable", value.enable, true, - () => setSlot("enable", !value.enable))} - + {typeof fc.showWhen !== "function" && ( + + {renderChip("Enable", value.enable, true, + () => setSlot("enable", !value.enable))} + + )} {hasRatios && ( Host ratio @@ -992,22 +1024,23 @@ export const Playground = ({ config }) => { hicache: { initState: () => ({ enable: false, backend: null, writePolicy: "auto" }), - revertHidden: (value, fc, base, h) => { - let changed = false; - const next = { ...value }; - if (next.backend !== null && fc.backends - && h.isHidden(fc.backends, next.backend, base)) { - next.backend = null; changed = true; - } - if (next.writePolicy !== "auto" && fc.writePolicies - && h.isHidden(fc.writePolicies, next.writePolicy, base)) { - next.writePolicy = "auto"; changed = true; - } - return changed ? next : value; - }, - apply: ({ flags, env, value, fc, sel, h }) => { if (fc.excludesHw && sel && fc.excludesHw.includes(sel.hw)) return { flags, env }; + // When the Deploy panel owns enablement (`showWhen`), the base already + // carries a complete, verified hicache recipe. Rebuilding it from this + // axis's own defaults would silently swap ratio/layout/io-backend, so + // only the two knobs this card actually exposes are touched. + if (typeof fc.showWhen === "function") { + const set = (name, val) => { + flags = h.stripFlagsByFirstToken(flags, [name]); + if (val) flags = h.insertBeforeTail(flags, [`${name} ${val}`]); + }; + if (value.backend) set("--hicache-storage-backend", value.backend); + if (value.writePolicy && value.writePolicy !== "auto") { + set("--hicache-write-policy", value.writePolicy); + } + return { flags, env }; + } flags = h.stripFlagsByFirstToken(flags, [ "--enable-hierarchical-cache", "--hicache-ratio", "--hicache-size", "--hicache-write-policy", "--hicache-mem-layout", "--hicache-io-backend", @@ -1066,10 +1099,12 @@ export const Playground = ({ config }) => {
HiCache - - {renderChip("Enable", value.enable, true, - () => setSlot("enable", !value.enable))} - + {typeof fc.showWhen !== "function" && ( + + {renderChip("Enable", value.enable, true, + () => setSlot("enable", !value.enable))} + + )} {hasBackends && ( Storage @@ -1092,23 +1127,29 @@ export const Playground = ({ config }) => { // ---- Axis: Flag Selects (generic, config-declared) ---------------------- // A LIST of single-selects, each declared entirely in config: - // { id, title, stripPrefixes: [...], options: [{ id, label, flags? }] } + // { id, title, stripPrefixes: [...], stripEnv?: [...], + // options: [{ id, label, flags?, env? }] } // Same shape as `speculative` minus its hardcoded title + strip list: pick // an option → strip the family, splice the option's flags. A flagless // option is the "none" / accuracy-safe choice (matches a base carrying none // of the family). Model-specific controls (KV-cache dtype, mamba scheduler // strategy, …) live here as DATA — no per-feature engine code. Supports // multiple selects per page. State: { [selectId]: optionId | null } - // (null = inherit base). + // (null = inherit base). `default` may be an option id or a function of the + // base selection (re-evaluated on every base change). flagSelects: { - initState: (fc) => { + initState: (fc, base) => { const out = {}; - for (const spec of (fc || [])) out[spec.id] = null; + for (const spec of (fc || [])) { + const d = typeof spec.default === "function" ? spec.default(base) : spec.default; + out[spec.id] = d ?? null; + } return out; }, // Per select: match base's family flags (first token ∈ stripPrefixes) // against each option's flags. A flagless option matches an empty family. + // Function-flag options (computed from the row values) never match. deriveFromBase: (cell, fc) => { const flags = (cell && cell.flags) || []; const out = {}; @@ -1117,6 +1158,7 @@ export const Playground = ({ config }) => { const fam = flags.filter((f) => prefixes.includes(f.split(/[\s=]/)[0])); let hit = null; for (const opt of (spec.options || [])) { + if (typeof opt.flags === "function") continue; const of = opt.flags || []; if (of.length === fam.length && of.every((x) => fam.includes(x))) { hit = opt.id; break; @@ -1127,19 +1169,6 @@ export const Playground = ({ config }) => { return out; }, - revertHidden: (value, fc, base, h) => { - let changed = false; - const next = { ...value }; - for (const spec of (fc || [])) { - const cur = next[spec.id]; - if (cur !== null && cur !== undefined - && h.isHidden(spec.options, cur, base)) { - next[spec.id] = null; changed = true; - } - } - return changed ? next : value; - }, - apply: ({ flags, env, value, fc, sel, h, derived }) => { const evalBase = { ...(sel || {}), @@ -1147,6 +1176,9 @@ export const Playground = ({ config }) => { pdMode: h.findFlagArg(flags, "--disaggregation-mode") || "off", }; for (const spec of (fc || [])) { + // Hidden rows must not emit: showWhen also receives the sibling row + // values (explicit picks + derived), so a row can gate on another's pick. + if (typeof spec.showWhen === "function" && !spec.showWhen(sel, value, derived)) continue; const v = value ? value[spec.id] : null; if (v === null || v === undefined) continue; // inherit base const d = derived ? derived[spec.id] : null; @@ -1154,10 +1186,51 @@ export const Playground = ({ config }) => { const opt = (spec.options || []).find((o) => o.id === v); if (!opt) continue; if (h.evaluateChip(opt, evalBase).disabled) continue; - flags = h.stripFlagsByFirstToken(flags, spec.stripPrefixes || []); - if (opt.flags && opt.flags.length) { - flags = h.insertBeforeTail(flags, opt.flags); + // `flags` may be a function of the whole row-value object (cross-row + // presets, e.g. preset x cluster-size). A function returning null + // means "leave the base untouched" (a true no-op Off). + const optFlags = typeof opt.flags === "function" + ? opt.flags(value, evalBase) + : (opt.flags || []); + if (optFlags === null) continue; + // In-place substitution keeps the rendered diff minimal: a family the + // base already carries changes value at its original position; families + // stripped and not re-emitted vanish in place; only net-new flags + // append as a block before the tail. + const strip = new Set(spec.stripPrefixes || []); + const byTok = new Map(); + for (const f of optFlags) { + const t = f.split(/[\s=]/)[0]; + if (!byTok.has(t)) byTok.set(t, []); + byTok.get(t).push(f); } + const consumed = new Set(); + const next = []; + for (const f of flags) { + const t = f.split(/[\s=]/)[0]; + if (byTok.has(t)) { + if (!consumed.has(t)) { + next.push(...byTok.get(t)); + consumed.add(t); + } + } else if (!strip.has(t)) { + next.push(f); + } + } + const fresh = []; + for (const [t, fs] of byTok) { + if (!consumed.has(t)) fresh.push(...fs); + } + flags = fresh.length ? h.insertBeforeTail(next, fresh) : next; + // Option env (env-var toggles, preset env): strip spec.stripEnv plus + // every option's env keys, then add the picked option's — mirrors the + // MoE backend card's strip-then-emit shape. + const envKeys = [...(spec.stripEnv || [])]; + for (const o of (spec.options || [])) { + for (const e of (o.env || [])) envKeys.push(e.split("=")[0]); + } + if (envKeys.length) env = h.stripEnvByPrefix(env, envKeys); + if (opt.env && opt.env.length) env = [...env, ...opt.env]; } return { flags, env }; }, @@ -1165,6 +1238,10 @@ export const Playground = ({ config }) => { render: ({ axisId, value, setValue, fc, base, s, h, renderChip, derived }) => { const cards = []; for (const spec of (fc || [])) { + // Row-level gate: a select whose whole family is meaningless under the + // current base (e.g. draft tokens with speculation off) is not rendered. + // Also receives (rowValues, derived) for sibling-dependent rows. + if (typeof spec.showWhen === "function" && !spec.showWhen(base, value, derived)) continue; const opts = (spec.options || []) .map((o) => h.evaluateChip(o, base)) .filter((c) => !c.hidden); @@ -1172,6 +1249,34 @@ export const Playground = ({ config }) => { const explicit = value ? value[spec.id] : null; const display = (explicit !== null && explicit !== undefined) ? explicit : (derived ? derived[spec.id] : null); + // `control: "slider"` renders the same option list as a range input — + // for dense ordered scales (1..7) where chips are just noise. Option + // ORDER is the scale; the option id is still what apply() consumes. + if (spec.control === "slider") { + const idx = Math.max(0, opts.findIndex((c) => c.value === display)); + const cur = opts[idx]; + cards.push( +
+
+ {spec.title} + setValue({ + ...value, + [spec.id]: opts[Number(e.target.value)].value, + })} + style={{ flex: 1, minWidth: "120px", accentColor: "#D45D44" }} + /> + + {cur ? cur.label : "-"} + +
+
+ ); + continue; + } cards.push(
@@ -1224,13 +1329,21 @@ export const Playground = ({ config }) => { // callers can pass a modified env (e.g. MegaMoE's stripEnv + append). const renderCommandLines = (cell, flags, cellEnv, sel, envValues, pdMode = null, mode = "python") => { const modelName = resolveModelName(sel); - const nnodes = parseNnodes(sel.nodes); - const multinode = nnodes > 1; let f = [...flags]; + // Presets may replace the base cell's topology by emitting --nnodes + // directly (for example, B300 1-node -> large-scale 4/8-node). Use that + // effective value for Docker networking, hints, and the command banner. + const nnodesFlag = f.find((x) => x.split(/[\s=]/)[0] === "--nnodes"); + const nnodesMatch = nnodesFlag && /^--nnodes(?:\s+|=)(\d+)$/.exec(nnodesFlag.trim()); + const baseNnodes = sel.nodes !== undefined + ? parseNnodes(sel.nodes) + : ((cell && cell.nnodes) || 1); + const nnodes = nnodesMatch ? parseInt(nnodesMatch[1], 10) : baseNnodes; + const multinode = nnodes > 1; if (multinode && !f.some((x) => x.startsWith("--nnodes"))) { // Insert the multi-node trio after the last parallelism flag (matches // _deployment.jsx so untouched-base output is byte-identical). - const PARALLELISM_ANCHORS = ["--enable-dp-attention", "--dp", "--tp"]; + const PARALLELISM_ANCHORS = ["--enable-dp-attention", "--dp", "--tp-size", "--tp"]; let at = -1; for (const anchor of PARALLELISM_ANCHORS) { at = f.findIndex((x) => x.split(/[\s=]/)[0] === anchor); @@ -1290,7 +1403,7 @@ export const Playground = ({ config }) => { && config.playgroundFeatures.pdDisagg.router; const routerPort = (routerCfg && routerCfg.port) || 8000; const routerLine = routerCfg - ? `# then front BOTH with the Router (SGLang Model Gateway) shown below.\n` + ? `# then front BOTH with the Router shown below.\n` + `# Client traffic (cURL) targets the router (:${routerPort}), not this role server.` : `# then front BOTH with a router; client traffic targets the router, not this role server.`; const hicacheCfg = config.playgroundFeatures @@ -1685,8 +1798,18 @@ export const Playground = ({ config }) => { // Base selection — live-linked to the Deployment panel via URL hash + custom event // (history.replaceState doesn't fire hashchange, hence the event too). + const overlayDefaults = () => { + const out = {}; + for (const d of (config.overlayDims || [])) { + const opts = d.options || []; + out[d.id] = d.default !== undefined ? d.default : ((opts[0] && opts[0].id) || ""); + } + return out; + }; + const baseFallback = () => ({ ...config.cells[0].match, ...overlayDefaults() }); + const initialBaseFromHash = () => { - const fallback = config.cells[0].match; + const fallback = baseFallback(); if (typeof window === "undefined") return { ...fallback }; const raw = window.location.hash.replace(/^#/, ""); if (!raw) return { ...fallback }; @@ -1699,7 +1822,7 @@ export const Playground = ({ config }) => { useEffect(() => { const onHash = () => setBase(initialBaseFromHash()); const onSelEvent = (e) => { - const fallback = config.cells[0].match; + const fallback = baseFallback(); const incoming = (e && e.detail) || {}; const next = { ...fallback }; for (const k of Object.keys(next)) { @@ -1715,35 +1838,36 @@ export const Playground = ({ config }) => { }; }, []); + // Calculator-computed --mamba-full-memory-ratio pair (K3): `eff` matches the + // composed command (with playground overrides), `base` matches the base view. + // Rendered into the command views below; never part of the broadcast flags. + const [pgRatios, setPgRatios] = useState({ eff: null, base: null }); + useEffect(() => { + const onRatio = (e) => setPgRatios({ + eff: (e.detail && e.detail.ratio) || null, + base: (e.detail && (e.detail.baseRatio || e.detail.ratio)) || null, + }); + window.addEventListener("sglang-k3-mamba-ratio", onRatio); + return () => window.removeEventListener("sglang-k3-mamba-ratio", onRatio); + }, []); + // Deltas: one slot per declared axis. const initialDeltas = () => { const out = {}; for (const [axisId, handler] of Object.entries(AXIS_HANDLERS)) { const fc = pgFeatures[axisId]; - if (fc) out[axisId] = handler.initState(fc); + if (fc) out[axisId] = handler.initState(fc, base); } return out; }; const [deltas, setDeltas] = useState(initialDeltas); - // On base change, revert any now-hidden picks to their inherit default. - // Disabled picks are NOT reverted (soft warning). + // Picking a different base is picking a different starting point, so the + // playground goes back to inheriting everything. Carrying overrides across a + // base change silently mixes knobs from a config the reader already left. useEffect(() => { - setDeltas((d) => { - let next = d; - let mutated = false; - for (const [axisId, handler] of Object.entries(AXIS_HANDLERS)) { - const fc = pgFeatures[axisId]; - if (!fc || d[axisId] === undefined) continue; - const nv = handler.revertHidden(d[axisId], fc, base, helpers); - if (nv !== d[axisId]) { - if (!mutated) { next = { ...d }; mutated = true; } - next[axisId] = nv; - } - } - return mutated ? next : d; - }); - }, [base.hw, base.variant, base.quant, base.strategy, base.nodes]); + setDeltas(initialDeltas()); + }, [Object.keys(base).sort().map((k) => `${k}=${base[k]}`).join("&")]); const [modal, setModal] = useState(null); // 'curl' | 'env' | 'submit' | null @@ -1800,7 +1924,7 @@ export const Playground = ({ config }) => { // 10. Derived values // ========================================================================== const s = makeStyles(isDark); - const baseCell = findCell(config.cells, base); + const baseCell = withOverlay(findCell(config.cells, base), base); const modelName = resolveModelName(base); // Per-axis state recovered from the base cell's flags (deriveFromBase). @@ -1815,7 +1939,7 @@ export const Playground = ({ config }) => { // Cross-axis facts folded into the `base` handed to chip-constraint // matching, so `hide`/`disable` can react to another axis's live state - // (render path only; revertHidden keeps the clean 5-dim base). + // (render path only; the raw base stays untouched). // dpAttnOn — effective DP-Attention resolves to "on" (positive degree // or true), explicit override else derived-from-base. // cpOn — effective prefill-CP resolves to "on" (degree > 1), @@ -1871,7 +1995,10 @@ export const Playground = ({ config }) => { ? attnDelta.cpStrategy : (attnDerived.cpStrategy !== undefined ? attnDerived.cpStrategy : null)) || "interleave"; - const pdMode = (deltas.pdDisagg && deltas.pdDisagg.mode) || "off"; + const pdCardOwnsMode = ((pgFeatures.pdDisagg && pgFeatures.pdDisagg.modes) || []).length > 0; + const pdMode = pdCardOwnsMode + ? ((deltas.pdDisagg && deltas.pdDisagg.mode) || "off") + : (base.pdMode || "off"); const constraintBase = { ...base, dpAttnOn, cpOn, cpStrategy, cpSizeTarget, effTp, pdMode, }; @@ -1881,14 +2008,47 @@ export const Playground = ({ config }) => { let diffLines = []; let pgFlagsLatest = []; let pgEnvLatest = []; + // Render-only ratio injection (before the host/port tail); skipped if the + // flags somehow already carry the family. + const withRatio = (fl, value) => { + if (!value) return fl; + if (fl.some((f) => f.startsWith("--mamba-full-memory-ratio"))) return fl; + const out = [...fl]; + const line = `--mamba-full-memory-ratio ${value}`; + const i = out.findIndex((f) => f.startsWith("--host")); + if (i >= 0) out.splice(i, 0, line); + else out.push(line); + return out; + }; if (baseCell) { - baseCommand = renderCommandLines(baseCell, baseCell.flags, baseCell.env, base, env, null, runMode); + baseCommand = renderCommandLines(baseCell, withRatio(baseCell.flags, pgRatios.base), baseCell.env, base, env, null, runMode); const { flags: pgFlags, env: pgEnv, pdMode } = applyAllDeltas(baseCell.flags, baseCell.env, deltas, base, derivedMap); pgFlagsLatest = pgFlags; pgEnvLatest = pgEnv; - playgroundCommand = renderCommandLines(baseCell, pgFlags, pgEnv, base, env, pdMode, runMode); + playgroundCommand = renderCommandLines(baseCell, withRatio(pgFlags, pgRatios.eff), pgEnv, base, env, pdMode, runMode); diffLines = computeDiff(baseCommand, playgroundCommand); } + // Broadcast both configs for outside consumers (the mamba ratio calculator): + // `baseFlags`/`baseEnv` = cell + Deploy overlays (what the Deploy command + // shows); `flags`/`env` = that plus playground overrides. Keyed on content + // so re-renders don't spam events. + const effectiveKey = + pgFlagsLatest.join("\n") + " " + pgEnvLatest.join("\n") + " " + + (baseCell ? baseCell.flags.join("\n") + " " + baseCell.env.join("\n") : ""); + useEffect(() => { + if (typeof window === "undefined" || !baseCell) return; + window.dispatchEvent( + new CustomEvent("sglang-k3-effective-config", { + detail: { + flags: pgFlagsLatest, + env: pgEnvLatest, + baseFlags: baseCell.flags, + baseEnv: baseCell.env, + }, + }) + ); + }, [effectiveKey]); + // Cross-cell verified detection: the emitted (env, flags) may match the // base cell itself or a sibling (different strategy). A sibling match // still shows Verified, plus a "switch base" link. @@ -1896,7 +2056,8 @@ export const Playground = ({ config }) => { ? findMatchingCell(config.cells, base, pgEnvLatest, pgFlagsLatest) : null; const playgroundVerified = !!(matchedCell && matchedCell.verified); - const matchedSiblingCell = (matchedCell && matchedCell !== baseCell) + const matchedSiblingCell = (matchedCell + && DIMENSIONS.some((d) => matchedCell.match[d] !== base[d])) ? matchedCell : null; // MTP hint on the EFFECTIVE (post-override) command — fires when the user // toggles speculative decoding on without setting --max-running-requests @@ -1969,8 +2130,13 @@ export const Playground = ({ config }) => { setTimeout(() => setCurlCopied(false), 1200); }; + // Summarize whichever dims the Deploy panel actually has — a config may drop + // variant/quant/nodes or add its own (PD mode, ...), so nothing is hardcoded. const baseSummary = baseCell - ? `${base.hw.toUpperCase()} · ${base.variant} · ${base.quant.toUpperCase()} · ${base.strategy} · ${base.nodes}` + ? Object.entries(base) + .filter(([, v]) => v !== undefined && v !== "") + .map(([k, v]) => (k === "hw" ? String(v).toUpperCase() : String(v))) + .join(" · ") : "(no verified cell at the current Deploy selection — showing playground only)"; // ========================================================================== @@ -2071,6 +2237,9 @@ export const Playground = ({ config }) => { {Object.entries(AXIS_HANDLERS).map(([axisId, handler]) => { const fc = pgFeatures[axisId]; if (!fc) return null; + // An axis whose feature is not switched on in the Deploy panel has + // nothing to tune — declared per config as `showWhen(base)`. + if (typeof fc.showWhen === "function" && !fc.showWhen(constraintBase)) return null; const setValue = (next) => setDeltas((d) => ({ ...d, [axisId]: next })); return handler.render({ axisId, value: deltas[axisId], setValue, @@ -2181,7 +2350,7 @@ export const Playground = ({ config }) => { {/* PD-Disagg router companion (separate block so the role diff stays pure). */} {pdRouter && routerText && (
-
Router (SGLang Model Gateway)
+
Router
Run after both roles are up. Substitute {""} /{" "} {""} with reachable hosts (both 127.0.0.1{" "} diff --git a/docs_new/src/snippets/configs/deepseek-ai/deepseek-v4.jsx b/docs_new/src/snippets/configs/deepseek-ai/deepseek-v4.jsx index 3d32ed2ef..f2a11ba55 100644 --- a/docs_new/src/snippets/configs/deepseek-ai/deepseek-v4.jsx +++ b/docs_new/src/snippets/configs/deepseek-ai/deepseek-v4.jsx @@ -17,7 +17,7 @@ export const config = { // merges these in, so a model-specific GPU is config data, not an engine edit. // RTX PRO 6000 (SM120 / Blackwell Desktop) is a workstation card, not datacenter. hardware: [ - { id: "rtx6000", label: "RTX PRO 6000", vram: "96GB", vendor: "nvidia" }, + { id: "rtx6000", label: "RTX PRO 6000", vram: "96GB", vendor: "blackwell" }, ], variants: [ diff --git a/docs_new/src/snippets/configs/meituan-longcat/longcat-2.0.jsx b/docs_new/src/snippets/configs/meituan-longcat/longcat-2.0.jsx index c1227143c..a641f5507 100644 --- a/docs_new/src/snippets/configs/meituan-longcat/longcat-2.0.jsx +++ b/docs_new/src/snippets/configs/meituan-longcat/longcat-2.0.jsx @@ -8,7 +8,7 @@ export const config = { // Model-specific GPUs the shared HARDWARE_CATALOG does not carry. hardware: [ - { id: "h20", label: "H20", vram: "96GB", vendor: "nvidia" }, + { id: "h20", label: "H20", vram: "96GB", vendor: "hopper" }, ], variants: [ diff --git a/docs_new/src/snippets/configs/moonshotai/kimi-k3-benchmarks.jsx b/docs_new/src/snippets/configs/moonshotai/kimi-k3-benchmarks.jsx new file mode 100644 index 000000000..b975a276a --- /dev/null +++ b/docs_new/src/snippets/configs/moonshotai/kimi-k3-benchmarks.jsx @@ -0,0 +1,23 @@ +export const benchmarks = [ + { match: { hw: "b300", pdMode: "unified", strategy: "balanced" } }, + { match: { hw: "b300", pdMode: "unified", strategy: "low-latency" } }, + { match: { hw: "b300", pdMode: "unified", strategy: "high-throughput" } }, + { match: { hw: "b200", pdMode: "unified", strategy: "low-latency" } }, + { match: { hw: "b200", pdMode: "unified", strategy: "balanced" } }, + { match: { hw: "b200", pdMode: "unified", strategy: "high-throughput" } }, + { match: { hw: "b200", pdMode: "unified", strategy: "long-context" } }, + { match: { hw: "mi350x", pdMode: "unified", strategy: "balanced" } }, + { match: { hw: "mi355x", pdMode: "unified", strategy: "balanced" } }, + { match: { hw: "h100", pdMode: "unified", strategy: "low-latency" } }, + { match: { hw: "h100", pdMode: "unified", strategy: "balanced" } }, + { match: { hw: "h100", pdMode: "unified", strategy: "high-throughput" } }, + { match: { hw: "h200", pdMode: "unified", strategy: "low-latency" } }, + { match: { hw: "h200", pdMode: "unified", strategy: "balanced" } }, + { match: { hw: "h200", pdMode: "unified", strategy: "high-throughput" } }, + { match: { hw: "gb300", pdMode: "unified", strategy: "low-latency" } }, + { match: { hw: "gb300", pdMode: "unified", strategy: "balanced" } }, + { match: { hw: "gb300", pdMode: "unified", strategy: "high-throughput" } }, + { match: { hw: "gb200", pdMode: "unified", strategy: "low-latency" } }, + { match: { hw: "gb200", pdMode: "unified", strategy: "balanced" } }, + { match: { hw: "gb200", pdMode: "unified", strategy: "high-throughput" } }, +]; diff --git a/docs_new/src/snippets/configs/moonshotai/kimi-k3.jsx b/docs_new/src/snippets/configs/moonshotai/kimi-k3.jsx new file mode 100644 index 000000000..c9a603ad6 --- /dev/null +++ b/docs_new/src/snippets/configs/moonshotai/kimi-k3.jsx @@ -0,0 +1,1767 @@ +// Single `export const config` literal — no spreads/calls/IIFE (Mintlify re-evals at hydration). +// Cells are denormalized: no `--nnodes`/`--node-rank`/`--dist-init-addr`/`--host`/`--port` literals — engine injects them. +// +// Recipes transcribed from the K3 serving benchmark scripts +// (benchmark/H200/script/v1/launch-k3.sh, benchmark/B300/script/v1/launch-k3.sh) +// and the B200 2×8 / GB200 4×4 / H100 4×8 / MI35x 1×8 reference launches. +// Kimi-K3 is a hybrid MoE VLM: 93 layers = 69 KDA (linear) + 24 MLA, 896 routed +// experts + 1 shared. Served today from the DarkSharpness/sglang-kimi fork. + +export const config = { + modelName: "Kimi-K3", + + // B300 (1×8 TP8), GB300 (2×4 TP8 MNNVL), B200 (2×8 TP16, or TP8/PP2 for + // Long-Context), GB200 (4×4 TP16 MNNVL), H200 (2×8 TP16/EP16), H100 + // (4×8 TP32/EP32), and MI350X/MI355X (1×8 TP8) have serving recipes. + supportedHardware: ["b300", "gb300", "b200", "gb200", "h200", "h100", "mi350x", "mi355x"], + + // Single checkpoint and a single shipped quantization (MXFP4), so neither is a + // reader-facing axis. Node count is fixed by the hardware recipe (B200 2x8, + // H100 4x8, B300 1x8, H200 2x8, GB200 4x4, GB300 2x4, + // MI350X/MI355X 1x8), so it rides on the cell rather than on a selector. + matchDims: [ + { + id: "pdMode", + title: "PD Mode", + options: [ + { id: "unified", label: "Unified" }, + { id: "prefill", label: "Prefill" }, + { id: "decode", label: "Decode" }, + ], + }, + { + // Prefill nodes are sized by context length; unified and decode nodes are + // sized by the latency/throughput operating point. One row, two option sets. + id: "strategy", + title: "Strategy", + options: [ + { id: "low-latency", label: "Low-Latency", showWhen: (s) => s.pdMode !== "prefill" }, + { id: "balanced", label: "Balanced", showWhen: (s) => s.pdMode !== "prefill" }, + { id: "high-throughput", label: "High-Throughput", showWhen: (s) => s.pdMode !== "prefill" }, + { id: "default", label: "Default", showWhen: (s) => s.pdMode === "prefill" }, + { + id: "long-context", + label: "Long-Context", + showWhen: (s) => + s.pdMode === "prefill" || + (s.pdMode === "unified" && s.hw === "b200"), + }, + ], + }, + ], + + // Orthogonal to the cell grid: the picked option layers flags onto whichever + // cell is showing, so turning speculation on does not triple the cell count. + overlayDims: [ + { + id: "spec", + title: "Spec Decode", + default: "dspark", + options: [ + { id: "none", label: "Non-Spec" }, + { + id: "dspark", + label: "DSPARK", + disabled: (s) => s.strategy === "long-context", + disableReason: + "The Long-Context recipes use pipeline parallelism (--pp-size 2 on B200 Unified, --pp-size 8 on Prefill), while DSPARK currently requires pp_size == 1.", + // Every DSPARK recipe layers ReplaySSM on: it moves the per-draft + // intermediate SSM states onto a fixed ring, lifting the concurrency + // the state pool admits (needs the Triton decode kernel, the K3 + // default). Only the PD prefill role opts out — it never runs verify + // and rejects the flag at startup. + flags: (s) => [ + "--speculative-algorithm DSPARK", + "--speculative-draft-model-path RadixArk/Kimi-K3-DSpark", + "--speculative-dspark-block-size 7", + ...(s.pdMode === "prefill" ? [] : ["--enable-linear-replayssm-spec"]), + ], + }, + { + id: "dflash", + label: "DFLASH", + // Listed so the axis is complete, but not selectable: no K3 DFLASH draft + // checkpoint has been published, so there is nothing to point + // --speculative-draft-model-path at. DFLASH is also CUDA-only, rejects DP + // attention, and requires pp_size == 1. + disabled: true, + disableReason: + "No K3 DFLASH draft checkpoint published yet — DSPARK is the available speculative path.", + flags: [ + "--speculative-algorithm DFLASH", + "--speculative-draft-model-path ", + ], + }, + ], + }, + { + // Recipes transcribed from the measured HiCache rounds. The `direct` io + // backend is deliberately not offered here: measured functionally + // equivalent to `kernel` (bit-identical per-tier hit rate), so it belongs + // in the Playground, not as a deployment choice. + id: "hicache", + title: "HiCache", + default: "off", + options: [ + { id: "off", label: "Off" }, + { + id: "l2", + label: "L1+L2 (host)", + flags: [ + "--enable-hierarchical-cache", + ], + // L1/L2 host tiering IS supported under DCP, so DCP stays — except with + // speculative decoding, which HiCache under DCP rejects at startup (the + // draft host pool has no DCP index translation). There the DCP operating + // point is dropped instead of blocking the option, same as L3 does. The + // ratio calculator reads --dcp-size off this command, so dropping it also + // re-solves --mamba-full-memory-ratio for the plain-TP shape. + stripPrefixes: (s) => + ["b300", "gb300", "b200", "gb200"].includes(s.hw) && + ["balanced", "high-throughput"].includes(s.strategy) && + s.spec !== "none" + ? ["--dcp-size", "--dcp-comm-backend"] + : [], + hints: (s) => + ["b300", "gb300", "b200", "gb200"].includes(s.hw) && + ["balanced", "high-throughput"].includes(s.strategy) && + s.spec !== "none" + ? [ + "HiCache under DCP rejects speculative decoding, so this recipe drops DCP", + "and runs plain TP. Per-request context is far shorter than the DCP", + "version — DCP is what buys KV capacity. Run the cell NOSPEC to keep DCP.", + ] + : [], + }, + { + id: "l3", + label: "+ L3 (Mooncake)", + flags: [ + "--enable-hierarchical-cache", + "--hicache-storage-backend mooncake", + ], + env: ["SGLANG_HICACHE_MOONCAKE_CONFIG_PATH={{MOONCAKE_CONFIG}}"], + // L3 under DCP is rejected at startup (the rank-0 replicated-MLA backup + // and the storage keys are not dcp_rank-aware), so on the DCP recipes L3 + // drops the DCP operating point and runs plain TP instead. The ratio + // calculator reads --dcp-size off this command, so dropping it also + // re-solves --mamba-full-memory-ratio for the plain-TP shape. + stripPrefixes: (s) => + ["b300", "gb300", "b200", "gb200"].includes(s.hw) && + ["balanced", "high-throughput"].includes(s.strategy) + ? ["--dcp-size", "--dcp-comm-backend"] + : [], + hints: (s) => + [ + "L3 also needs a mooncake_master process on rank 0 and the config file", + "above present on every rank — the launch command alone is not enough.", + ...(["b300", "gb300", "b200", "gb200"].includes(s.hw) && + ["balanced", "high-throughput"].includes(s.strategy) + ? [ + "L3 storage keys are not dcp_rank-aware yet, so this recipe drops DCP", + "and runs plain TP. Concurrency lands on a similar target, but", + "per-request context is far shorter than the DCP version — DCP is", + "what buys KV capacity.", + ] + : []), + ], + }, + ], + }, + ], + + modelNames: { + default: "moonshotai/Kimi-K3", + }, + + placeholders: { + HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" }, + PORT: { target: "command", label: "Bind port", default: "30000" }, + NODE0_IP: { target: "command", label: "Head node IP", default: "" }, + NODE_RANK: { target: "command", label: "This node rank", default: "" }, + LOCAL_IP: { target: "command", label: "This node IP", default: "" }, + NETWORK_IFACE: { target: "command", label: "Cross-node NIC", default: "" }, + HF_TOKEN: { target: "command", label: "HF token (Docker)", default: "" }, + MOONCAKE_CONFIG: { target: "command", label: "Mooncake config path", default: "" }, + CURL_HOST: { target: "curl", label: "Server host", default: "localhost" }, + CURL_PORT: { target: "curl", label: "Server port", default: "30000" }, + }, + + curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\ +-H 'Content-Type: application/json' \\ +-d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"Hello"}] }'`, + + // Reproduce command for the benchmark card's "⚡ Reproduce" modal. The + // sweep is 8k-in / 1k-out random, num_prompts = 5 × concurrency, cache-cold. + benchmarkCommands: { + speed: +`python3 -m sglang.bench_serving \\ + --backend sglang \\ + --host {{CURL_HOST}} --port {{CURL_PORT}} \\ + --model {{MODEL_NAME}} \\ + --dataset-name {{DATASET}} \\ + --random-input-len {{ISL}} --random-output-len {{OSL}} --random-range-ratio 1.0 \\ + --num-prompts {{NUM_PROMPTS}} --max-concurrency {{MAX_CONCURRENCY}} \\ + --flush-cache`, + // num_prompts = 5 × concurrency (measured floor 16). + numPromptsByConc: { 1: 16, 16: 80, 64: 320, 256: 1280, 1024: 5120 }, + }, + + // Recommend the published CUDA 13 image for every NVIDIA recipe. MI35x keeps + // the ROCm 7.2 image. + dockerImages: { + h100: "lmsysorg/sglang:kimi-k3", + h200: "lmsysorg/sglang:kimi-k3", + b300: "lmsysorg/sglang:kimi-k3", + gb300: "lmsysorg/sglang:kimi-k3", + b200: "lmsysorg/sglang:kimi-k3", + gb200: "lmsysorg/sglang:kimi-k3", + mi350x: "lmsysorg/sglang:dev-rocm720-mi35x", + mi355x: "lmsysorg/sglang:dev-rocm720-mi35x", + }, + // Pre-selects the issue template's `model` field on "Submit verified cell". + github: { + cookbookModel: "moonshotai/kimi-k3", + }, + + playgroundFeatures: { + + // ----- Card: "Attention Parallelism" ----- + // DP-Attention is a combined knob: value = DP degree AND toggles `--enable-dp-attention`. + // K3's MLA latent KV is TP-replicated, so DP-attention RAISES per-GPU KV pressure — + // dp=8/attn_tp=1 OOMs on a single node; use dp=2/attn_tp=4. No CP knob: K3 uses + // decode context parallel (`--dcp-size`), a different lever from prefill `--attn-cp-size`. + attention: { + knobs: [ + { id: "tp", label: "TP", values: [ + null, 8, + { + value: 16, + disable: [ + { + when: { hw: ["b300", "gb300"] }, + reason: "TP=16 needs 16 ranks; the B300 and GB300 recipes have 8 ranks.", + }, + { + when: { hw: ["b200"], strategy: ["long-context"] }, + reason: "The B200 Long-Context recipe already uses all 16 GPUs as TP8 × PP2; changing TP to 16 would require 32 ranks.", + }, + ], + }, + ]}, + { id: "dpAttn", label: "DP-Attention", + values: [ + null, false, 2, 4, + { value: 8, disable: { hw: ["b300", "gb300"] }, + disableReason: "On an 8-rank deployment (B300 1×8, GB300 2×4) dp=8 leaves attn_tp=1, so each rank holds the full unsharded MLA KV and OOMs — prefer dp=2/attn_tp=4." }, + { + value: 16, + disable: [ + { + when: { hw: ["b300", "gb300"] }, + reason: "DP-Attention=16 needs 16 TP ranks; the B300 and GB300 recipes have 8.", + }, + { + when: { hw: ["b200"], strategy: ["long-context"] }, + reason: "The B200 Long-Context recipe uses TP8 within each PP stage, so DP-Attention cannot exceed 8.", + }, + ], + }, + ], + labels: { "auto": "Auto", "false": "Off" } }, + ], + }, + + // ----- Card: "MoE Parallelism" ----- + // K3 = 896 routed experts + 1 shared. Marlin (W4A16) is the accuracy runner; the + // MXFP4 / a2a runners (FlashInfer MXFP4, DeepEP, MegaMoE) are throughput levers. + moe: { + backend: { + options: [ + { id: null, label: "Inherited" }, + { id: "deepep", label: "DeepEP", flags: ["--moe-a2a-backend deepep"] }, + // Blackwell-only kernel-fusion path; selecting it reveals the Quantization sub-select. + { id: "megamoe", label: "MegaMoE", flags: ["--moe-a2a-backend megamoe"], + requiresHw: ["b200", "b300", "gb200", "gb300"] }, + // Blackwell-only: runs the prebuilt trtllm-gen SiTU cubins; needs the + // downloadable SiTU cubin pool unpacked and pointed to by the env var. + { id: "flashinfer_mxfp4", label: "FlashInfer (MXFP4)", flags: ["--moe-runner-backend flashinfer_mxfp4"], + env: ["SGLANG_TRTLLM_GEN_MOE_CUBIN_POOL=/path/to/trtllm_gen_moe_cubin_pool"], + requiresHw: ["b200", "b300", "gb200", "gb300"] }, + { id: "marlin", label: "Marlin (W4A16)", flags: ["--moe-runner-backend marlin"] }, + ], + }, + // MegaMoE quantization sub-select — shown only when backend === "megamoe". + megamoeQuant: { + stripEnv: ["SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK"], + options: [ + { id: "w4a8", label: "W4A8", + env: ["SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320"] }, + { id: "w4a4", label: "W4A4", + env: [ + "SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320", + "SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1", + "SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1", + ] }, + ], + }, + ep: { label: "EP", values: [ + null, 1, 2, 4, 8, + { + value: 16, + disable: [ + { + when: { hw: ["b300", "gb300"] }, + reason: "EP=16 needs 16 TP ranks; the B300 and GB300 recipes have 8.", + }, + { + when: { hw: ["b200"], strategy: ["long-context"] }, + reason: "The B200 Long-Context recipe uses TP8 within each PP stage, so EP cannot exceed 8.", + }, + ], + }, + ]}, + }, + + // ----- Card: "Parsers" ----- + parsers: { + items: [ + { id: "reasoning", label: "Reasoning Parser", flag: "--reasoning-parser kimi_k3" }, + { id: "toolCall", label: "Tool Call Parser", flag: "--tool-call-parser kimi_k3" }, + ], + }, + + // ----- Card: "PD Disaggregation" ----- (validated functionally on B300×2 mooncake) + // No `modes` list: the role is a Deploy-panel dimension (PD Mode), so this card + // only tunes the transport and stays hidden until a role is selected there. + pdDisagg: { + showWhen: (b) => b.pdMode === "prefill" || b.pdMode === "decode", + transferBackends: [ + { id: "nixl", label: "NiXL" }, + { id: "mooncake", label: "Mooncake" }, + ], + // `auto` is a sentinel (emits no --disaggregation-ib-device flag). + ibDevices: [{ id: "auto", label: "Auto" }, "mlx5_0"], + router: { + port: 8000, + // Ports come from the engine's PD_PORTS, the same source the role + // commands rewrite `--port` from, so the router always targets the + // ports the two roles actually bind. The positional after --prefill is + // the bootstrap port; it must match the prefill server's + // --disaggregation-bootstrap-port (default 8998) or only the decode + // worker registers. + command: +`python3 -m sglang_router.launch_router \\ + --pd-disaggregation \\ + --prefill http://:{{PREFILL_PORT}} 8998 \\ + --decode http://:{{DECODE_PORT}} \\ + --host 0.0.0.0 --port {{ROUTER_PORT}} \\ + --disable-circuit-breaker \\ + --health-check-interval-secs 999999`, + }, + }, + + // ----- Card: "HiCache" ----- (K3 hybrid L1/L2/L3, incl. KDA state) + hicache: { + showWhen: (b) => b.hicache !== undefined && b.hicache !== "off", + // Picking a storage backend here IS L3, and L3 under DCP is rejected at + // startup. The Deploy panel's own L3 option drops DCP first; reaching L3 + // through this card would not, so gate it on the DCP recipes unless Deploy + // already switched to L3 (hicache "off"/"l2" means DCP is still standing). + backends: [ + { id: null, label: "Auto" }, + { id: "file", label: "File", + disable: [{ when: { hw: ["b300", "gb300", "b200", "gb200"], strategy: ["balanced", "high-throughput"], hicache: ["l2"], spec: ["none"] }, + reason: "This recipe runs DCP, and a storage backend (L3) under DCP is rejected at startup. Switch HiCache to L3 in the Deploy panel (that drops DCP), or stay on L1+L2." }] }, + { id: "mooncake", label: "Mooncake", + disable: [{ when: { hw: ["b300", "gb300", "b200", "gb200"], strategy: ["balanced", "high-throughput"], hicache: ["l2"], spec: ["none"] }, + reason: "This recipe runs DCP, and a storage backend (L3) under DCP is rejected at startup. Switch HiCache to L3 in the Deploy panel (that drops DCP), or stay on L1+L2." }] }, + { id: "hf3fs", label: "HF3FS", + disable: [{ when: { hw: ["b300", "gb300", "b200", "gb200"], strategy: ["balanced", "high-throughput"], hicache: ["l2"], spec: ["none"] }, + reason: "This recipe runs DCP, and a storage backend (L3) under DCP is rejected at startup. Switch HiCache to L3 in the Deploy panel (that drops DCP), or stay on L1+L2." }] }, + { id: "nixl", label: "NiXL", + disable: [{ when: { hw: ["b300", "gb300", "b200", "gb200"], strategy: ["balanced", "high-throughput"], hicache: ["l2"], spec: ["none"] }, + reason: "This recipe runs DCP, and a storage backend (L3) under DCP is rejected at startup. Switch HiCache to L3 in the Deploy panel (that drops DCP), or stay on L1+L2." }] }, + ], + writePolicies: [ + { id: "auto", label: "Auto" }, + { id: "write_through", label: "Write-through" }, + { id: "write_back", label: "Write-back" }, + { id: "write_through_selective", label: "Write-through (selective)" }, + ], + }, + + // ----- Axis: Flag Selects (K3 hybrid dual-pool knobs) ----- + // KDA state pool vs full-KV pool levers (measured dual-pool analysis). The + // flagless option is the accuracy-safe default; the others are capacity/long-ctx + // levers whose accuracy A/B is workload-gated — Playground opt-ins, not cells. + flagSelects: [ + { + // Deployment picks this from the strategy; the row is the override, and the + // engine derives the current pick from the base command so it shows whatever + // Deploy resolved until you change it. + // + // The unit is tokens PROPOSED per step, which each algorithm spells its own + // way — strip all three families so switching never leaves a stale flag: + // DSPARK --speculative-dspark-block-size N (gamma, == proposed) + // DFLASH --speculative-dflash-block-size N+1 (verify window) + // EAGLE --speculative-num-steps N (chain; topk>1 is a tree) + // Only DSPARK is selectable today, so only its form is emitted. + id: "proposedDraftTokens", title: "Proposed Draft Tokens", + showWhen: (b) => b.spec === "dspark", + control: "slider", + stripPrefixes: [ + "--speculative-dspark-block-size", + "--speculative-dflash-block-size", + "--speculative-num-steps", + ], + options: [ + { id: "1", label: "1", flags: ["--speculative-dspark-block-size 1"] }, + { id: "2", label: "2", flags: ["--speculative-dspark-block-size 2"] }, + { id: "3", label: "3", flags: ["--speculative-dspark-block-size 3"] }, + { id: "4", label: "4", flags: ["--speculative-dspark-block-size 4"] }, + { id: "5", label: "5", flags: ["--speculative-dspark-block-size 5"] }, + { id: "6", label: "6", flags: ["--speculative-dspark-block-size 6"] }, + { id: "7", label: "7", flags: ["--speculative-dspark-block-size 7"] }, + ], + }, + { + // ReplaySSM moves the per-draft intermediate SSM states onto a fixed ring, + // freeing state slots for more concurrency at a lower --mamba-full-memory-ratio. + // Spec-only, so gate the row on DSPARK; every DSPARK recipe (except the PD + // prefill role) turns it on in the base, so this row derives to On and + // exists mainly as the opt-out. + // Needs the Triton linear-attn decode backend (the K3 default). + id: "replaySsm", title: "ReplaySSM (spec)", + showWhen: (b) => b.spec === "dspark", + stripPrefixes: ["--enable-linear-replayssm-spec"], + options: [ + { id: "off", label: "Off" }, + { + id: "on", label: "On", + // The ring is spec-verify-only scratch and a prefill server never + // runs verify, so the engine rejects the flag outright there. + disable: { pdMode: ["prefill"] }, + disableReason: + "A PD prefill server never runs speculative verify, so --enable-linear-replayssm-spec is rejected at startup.", + flags: ["--enable-linear-replayssm-spec"], + }, + ], + }, + { + // Compact prunes the verify layout to the SPS budget. SILENT-INERT + // without --speculative-dspark-sps-table-path (every step still + // verifies full width); fails fast with ReplaySSM or DCP > 1. + id: "raggedVerify", title: "Ragged Verify Mode (spec)", + showWhen: (b) => b.spec === "dspark", + stripEnv: ["SGLANG_RAGGED_VERIFY_MODE"], + options: [ + { id: "static", label: "Auto (static)" }, + { id: "compact", label: "Compact (requires SPS table)", env: ["SGLANG_RAGGED_VERIFY_MODE=compact"] }, + ], + }, + { + id: "kvCacheDtype", title: "KV Cache Precision", + stripPrefixes: ["--kv-cache-dtype"], + options: [ + { id: "auto", label: "Auto (BF16)" }, + { id: "fp8", label: "FP8 (E4M3) — halves KV memory", flags: ["--kv-cache-dtype fp8_e4m3"] }, + ], + }, + { + id: "mambaSsmDtype", title: "KDA State Precision", + stripPrefixes: ["--mamba-ssm-dtype"], + options: [ + { id: "auto", label: "Auto (FP32)" }, + { id: "bf16", label: "BFloat16 — halves state memory", flags: ["--mamba-ssm-dtype bfloat16"] }, + // The dtype --enable-mamba-cache-stochastic-rounding requires; no + // serving round has tried it, unlike bf16. + { id: "fp16", label: "Float16 — stochastic-rounding capable", flags: ["--mamba-ssm-dtype float16"] }, + ], + }, + { + // Whole-model prefix cache (the radix tree spans MLA KV + KDA state). + // Off suits prefix-free traffic (offline batch, evals): 1 state slot + // per request instead of 4-5. + id: "prefixCache", title: "Prefix Cache", + stripPrefixes: ["--disable-radix-cache"], + options: [ + { id: "on", label: "On" }, + { id: "off", label: "Off", flags: ["--disable-radix-cache"] }, + ], + }, + { + // How KDA state buffers for radix reuse; no strategy exists with the + // prefix cache off, so the row hides (and stops emitting) there. + // Slot cost per request: extra_buffer 5, extra_buffer_lazy 4. + id: "mambaRadix", title: "KDA Radix Cache Strategy", + showWhen: (b, v, d) => (((v && v.prefixCache) ?? (d && d.prefixCache)) !== "off"), + stripPrefixes: ["--mamba-radix-cache-strategy"], + options: [ + { id: "auto", label: "Auto (extra_buffer)" }, + { id: "lazy", label: "extra_buffer_lazy", flags: ["--mamba-radix-cache-strategy extra_buffer_lazy"] }, + { id: "nobuf", label: "no_buffer", flags: ["--mamba-radix-cache-strategy no_buffer"] }, + ], + }, + { + // Experimental env toggle: SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK skips the + // decode-time mamba lock, freeing one resident state slot per request + // (extra_buffer 5→4, extra_buffer_lazy 4→3; no_buffer stays 3). Off by + // default. Env var, not a flag, so it emits via env/stripEnv. + id: "mambaSlotSaving", title: "KDA Slot Saving (experimental)", + stripEnv: ["SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK"], + options: [ + { id: "off", label: "Off" }, + { id: "on", label: "On", env: ["SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1"] }, + ], + }, + { + // Only meaningful with EP a2a on (MoE card or a large-scale preset). + id: "eplb", title: "Expert Rebalancing (EPLB)", + stripPrefixes: ["--enable-eplb"], + options: [ + { id: "off", label: "Off" }, + { id: "on", label: "On (requires EP a2a)", flags: ["--enable-eplb"] }, + ], + }, + { + // Prefill has no graph by default; BCG captures it as a breakable graph. + // Validated on the no-a2a MXFP4 runner; untested against SBO (EP a2a) + // and DP attention. + id: "prefillGraph", title: "Prefill CUDA Graph", + stripPrefixes: ["--cuda-graph-backend-prefill"], + options: [ + { id: "auto", label: "Auto (off)" }, + { id: "bcg", label: "Breakable (BCG)", flags: ["--cuda-graph-backend-prefill breakable"] }, + ], + }, + { + // A decode server runs a chunk cache by default (1 state slot/req); + // radix restores prefix reuse at the unified per-request slot cost. + id: "pdDecodeRadix", title: "PD Decode Radix Cache", + showWhen: (b) => b.pdMode === "decode", + stripPrefixes: ["--disaggregation-decode-enable-radix-cache"], + options: [ + { id: "off", label: "Off (chunk cache)" }, + { id: "on", label: "On", flags: ["--disaggregation-decode-enable-radix-cache"] }, + ], + }, + { + // The two rows below compose: Cluster Size picks N, Large-Scale Preset + // resolves it into the full parallelism shape (tp/ep/dp/dcp; attn-tp = + // tp/dp); pool sizing rides the calculator-driven ratio. + id: "lsGpus", title: "Cluster Size (large-scale)", + showWhen: (b) => b.pdMode === undefined || b.pdMode === "unified", + // Default follows the base cell's own GPU count (tp8 lanes -> 8, + // tp16 lanes -> 16), so a preset starts from "same hardware, new shape". + default: (b) => + ({ b300: "8", gb300: "8", b200: "16", gb200: "16", + h200: "16", h100: "32", mi350x: "8", mi355x: "8" })[(b || {}).hw] || "32", + stripPrefixes: [], + options: [ + { id: "8", label: "8 GPUs" }, + { id: "16", label: "16 GPUs" }, + { id: "32", label: "32 GPUs" }, + { id: "64", label: "64 GPUs" }, + ], + }, + { + id: "lsPreset", title: "Large-Scale Preset", + showWhen: (b) => b.pdMode === undefined || b.pdMode === "unified", + stripPrefixes: [ + "--tp-size", "--tp", "--tensor-parallel-size", + "--ep-size", "--ep", "--expert-parallel-size", + "--enable-dp-attention", "--dp-size", "--enable-dp-lm-head", + "--dcp-size", "--dcp-comm-backend", + // The B200 Long-Context cell carries --pp-size 2; left standing it + // multiplies against the preset's --tp-size for a world size the + // preset's own --nnodes cannot satisfy. + "--pp-size", "--pipeline-parallel-size", + "--moe-a2a-backend", "--moe-runner-backend", + "--kv-cache-dtype", "--mamba-ssm-dtype", "--mamba-radix-cache-strategy", + "--mem-fraction-static", "--disable-radix-cache", "--enable-symm-mem", + ], + stripEnv: ["SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK"], + options: [ + { id: "off", label: "Off", flags: () => null }, + { + id: "serving", label: "Peak Throughput", + disable: { hw: ["h100", "h200", "mi350x", "mi355x"] }, + disableReason: "The large-scale presets ride the MegaMoE a2a lane (SM100/SM103) — Blackwell only.", + env: ["SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=20480"], + flags: (v, b) => { + const n = Number(v.lsGpus) || 32; + const dp = n / 8; + const gpusPerNode = ["gb200", "gb300"].includes(b.hw) ? 4 : 8; + const nnodes = n / gpusPerNode; + return [ + `--tp-size ${n}`, `--ep-size ${n}`, + ...(dp > 1 ? ["--enable-dp-attention", `--dp-size ${dp}`, "--enable-dp-lm-head"] : []), + ...(nnodes > 1 ? [`--nnodes ${nnodes}`, "--node-rank {{NODE_RANK}}", "--dist-init-addr {{NODE0_IP}}:20000"] : []), + "--moe-a2a-backend megamoe", "--moe-runner-backend deep_gemm", + "--kv-cache-dtype fp8_e4m3", "--mamba-ssm-dtype bfloat16", + "--mamba-radix-cache-strategy extra_buffer_lazy", + "--mem-fraction-static 0.92", + ]; + }, + }, + { + id: "capacity", label: "Peak Capacity (+DCP8)", + // The only playground option that re-adds DCP, so it is also the only + // one that can resurrect the DCP + L3 combination the Deploy panel + // just stripped out. + disable: [ + { + when: { hw: ["h100", "h200", "mi350x", "mi355x"] }, + reason: "The large-scale presets ride the MegaMoE a2a lane (SM100/SM103) — Blackwell only.", + }, + { + when: { hicache: ["l3"] }, + reason: "L3 storage keys are not dcp_rank-aware, so DCP and L3 cannot run together. Use Peak Throughput, or switch HiCache to L1+L2.", + }, + { + when: { hicache: ["l2"], spec: ["dspark"] }, + reason: "HiCache under DCP rejects speculative decoding, so this preset cannot add DCP here. Use Peak Throughput, or run the cell NOSPEC.", + }, + ], + env: ["SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=20480"], + flags: (v, b) => { + const n = Number(v.lsGpus) || 32; + const dp = n / 8; + const gpusPerNode = ["gb200", "gb300"].includes(b.hw) ? 4 : 8; + const nnodes = n / gpusPerNode; + return [ + `--tp-size ${n}`, `--ep-size ${n}`, + ...(dp > 1 ? ["--enable-dp-attention", `--dp-size ${dp}`, "--enable-dp-lm-head"] : []), + "--dcp-size 8", + ...(nnodes > 1 ? [`--nnodes ${nnodes}`, "--node-rank {{NODE_RANK}}", "--dist-init-addr {{NODE0_IP}}:20000"] : []), + "--moe-a2a-backend megamoe", "--moe-runner-backend deep_gemm", + "--kv-cache-dtype fp8_e4m3", "--mamba-ssm-dtype bfloat16", + "--mamba-radix-cache-strategy extra_buffer_lazy", + "--mem-fraction-static 0.92", + ]; + }, + }, + ], + }, + ], + }, + + cells: [ + { + match: { hw: "b300", pdMode: "unified", strategy: "low-latency" }, + nnodes: 1, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 8", + "--disable-custom-all-reduce", + "--enable-symm-mem", + "--mem-fraction-static 0.85", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", pdMode: "unified", strategy: "balanced" }, + nnodes: 1, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 8", + "--dcp-size 8", + "--disable-custom-all-reduce", + "--mem-fraction-static 0.85", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", pdMode: "unified", strategy: "high-throughput" }, + nnodes: 1, + verified: false, + redirect: true, + warn: "High-Throughput is the large-scale lane: pick a Cluster Size and a Large-Scale Preset in the [Playground](#playground) to compose the DP x EP command on top of this hardware's Balanced recipe.", + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 8", + "--dcp-size 8", + "--disable-custom-all-reduce", + "--mem-fraction-static 0.85", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + // Plain TP16 avoids pipeline bubbles at the shallow 16-request point. + match: { hw: "b200", pdMode: "unified", strategy: "low-latency" }, + nnodes: 2, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 16", + "--mem-fraction-static 0.85", + "--disable-flashinfer-autotune", + "--watchdog-timeout 3600", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--model-loader-extra-config '{\"enable_multithread_load\": true}'", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + // TP16 + DCP16 on two B200 nodes. + match: { hw: "b200", pdMode: "unified", strategy: "balanced" }, + nnodes: 2, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 16", + "--dcp-size 16", + "--mem-fraction-static 0.85", + "--disable-flashinfer-autotune", + "--watchdog-timeout 3600", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--model-loader-extra-config '{\"enable_multithread_load\": true}'", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + // Balanced baseline; High-Throughput routes to the large-scale presets. + match: { hw: "b200", pdMode: "unified", strategy: "high-throughput" }, + nnodes: 2, + verified: false, + redirect: true, + warn: "High-Throughput is the large-scale lane: pick a Cluster Size and a Large-Scale Preset in the [Playground](#playground) to compose the DP x EP command on top of this hardware's Balanced recipe.", + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 16", + "--dcp-size 16", + "--mem-fraction-static 0.85", + "--disable-flashinfer-autotune", + "--watchdog-timeout 3600", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--model-loader-extra-config '{\"enable_multithread_load\": true}'", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + // Reference long-context launch: PP2 halves the layer-local KV/state + // footprint per GPU while TP8 spans each 8-GPU pipeline stage. + match: { hw: "b200", pdMode: "unified", strategy: "long-context" }, + nnodes: 2, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 8", + "--pp-size 2", + "--mem-fraction-static 0.85", + "--context-length 131072", + "--chunked-prefill-size 8192", + "--mamba-radix-cache-strategy extra_buffer", + "--disable-flashinfer-autotune", + "--watchdog-timeout 3600", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--model-loader-extra-config '{\"enable_multithread_load\": true}'", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + // MI350X and MI355X use the same single-node TP8 ROCm/AITER profile. + match: { hw: "mi350x", pdMode: "unified", strategy: "balanced" }, + nnodes: 1, + verified: false, + env: [ + "SGLANG_USE_AITER=1", + "SGLANG_AITER_K3_OPT=1", + "AITER_FLYDSL_FORCE=1", + "AITER_SITUV2_A8W4=1", + ], + flags: [ + "--model-path {{MODEL_NAME}}", + "--trust-remote-code", + "--tp-size 8", + "--attention-backend triton", + "--dtype bfloat16", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 256", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + // Same ROCm/AITER profile as MI350X. + match: { hw: "mi355x", pdMode: "unified", strategy: "balanced" }, + nnodes: 1, + verified: false, + env: [ + "SGLANG_USE_AITER=1", + "SGLANG_AITER_K3_OPT=1", + "AITER_FLYDSL_FORCE=1", + "AITER_SITUV2_A8W4=1", + ], + flags: [ + "--model-path {{MODEL_NAME}}", + "--trust-remote-code", + "--tp-size 8", + "--attention-backend triton", + "--dtype bfloat16", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 256", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + // Latency operating point. Keep the fixed H100 state budget but use the + // default extra_buffer strategy; no explicit request-concurrency cap. + match: { hw: "h100", pdMode: "unified", strategy: "low-latency" }, + nnodes: 4, + verified: false, + env: [ + "NCCL_CUMEM_ENABLE=1", + "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True", + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + "SGLANG_K3_ATTN_RES_MODE=jit", + "SGLANG_MOE_FUSED_GATE_RADIX=1", + "SGLANG_HOST_IP={{LOCAL_IP}}", + "NCCL_SOCKET_IFNAME={{NETWORK_IFACE}}", + "GLOO_SOCKET_IFNAME={{NETWORK_IFACE}}", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 32", + "--ep-size 32", + "--moe-runner-backend marlin", + "--decode-attention-backend flashmla", + "--mem-fraction-static 0.85", + "--dist-timeout 3600", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + // Accuracy-preserving default. The fixed state budget is safer than + // guessing a KDA/KV ratio on 80 GB GPUs. + match: { hw: "h100", pdMode: "unified", strategy: "balanced" }, + nnodes: 4, + verified: false, + env: [ + "NCCL_CUMEM_ENABLE=1", + "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True", + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + "SGLANG_K3_ATTN_RES_MODE=jit", + "SGLANG_MOE_FUSED_GATE_RADIX=1", + "SGLANG_HOST_IP={{LOCAL_IP}}", + "NCCL_SOCKET_IFNAME={{NETWORK_IFACE}}", + "GLOO_SOCKET_IFNAME={{NETWORK_IFACE}}", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 32", + "--ep-size 32", + "--moe-runner-backend marlin", + "--decode-attention-backend flashmla", + "--mem-fraction-static 0.85", + "--dist-timeout 3600", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + // Throughput operating point from the 4×8 H100 reference launch. + // extra_buffer_lazy lowers the per-request state cost from 5 to 4 slots. + match: { hw: "h100", pdMode: "unified", strategy: "high-throughput" }, + nnodes: 4, + verified: false, + env: [ + "NCCL_CUMEM_ENABLE=1", + "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True", + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + "SGLANG_K3_ATTN_RES_MODE=jit", + "SGLANG_MOE_FUSED_GATE_RADIX=1", + "SGLANG_HOST_IP={{LOCAL_IP}}", + "NCCL_SOCKET_IFNAME={{NETWORK_IFACE}}", + "GLOO_SOCKET_IFNAME={{NETWORK_IFACE}}", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 32", + "--ep-size 32", + "--moe-runner-backend marlin", + "--decode-attention-backend flashmla", + "--mem-fraction-static 0.85", + "--mamba-radix-cache-strategy extra_buffer_lazy", + "--dist-timeout 3600", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", pdMode: "unified", strategy: "low-latency" }, + nnodes: 2, + verified: false, + env: [ + "NCCL_MNNVL_ENABLE=1", + "NCCL_CUMEM_ENABLE=1", + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 16", + "--ep-size 16", + "--moe-runner-backend marlin", + "--decode-attention-backend flashmla", + "--enable-symm-mem", + "--mem-fraction-static 0.85", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + // Accuracy-preserving default (eval-command: mem-frac 0.85, graph-bs 64). + match: { hw: "h200", pdMode: "unified", strategy: "balanced" }, + nnodes: 2, + verified: false, + env: [ + "NCCL_MNNVL_ENABLE=1", + "NCCL_CUMEM_ENABLE=1", + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 16", + "--ep-size 16", + "--moe-runner-backend marlin", + "--decode-attention-backend flashmla", + "--enable-symm-mem", + "--mem-fraction-static 0.85", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + // Throughput-tuned (perf-command: mem-frac 0.90, graph-bs 256, extra_buffer_lazy → max_running 98). + match: { hw: "h200", pdMode: "unified", strategy: "high-throughput" }, + nnodes: 2, + verified: false, + env: [ + "NCCL_MNNVL_ENABLE=1", + "NCCL_CUMEM_ENABLE=1", + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 16", + "--ep-size 16", + "--moe-runner-backend marlin", + "--decode-attention-backend flashmla", + "--enable-symm-mem", + "--mem-fraction-static 0.90", + "--mamba-radix-cache-strategy extra_buffer_lazy", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", pdMode: "unified", strategy: "low-latency" }, + nnodes: 2, + verified: false, + env: [ + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 8", + "--enable-symm-mem", + "--mem-fraction-static 0.85", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + // Balanced: the DCP8 lane at stock mem-frac 0.85; MoE runner and + // attention backends resolve automatically on SM100/SM103. + match: { hw: "gb300", pdMode: "unified", strategy: "balanced" }, + nnodes: 2, + verified: false, + env: [ + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 8", + "--dcp-size 8", + "--mem-fraction-static 0.85", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", pdMode: "unified", strategy: "high-throughput" }, + nnodes: 2, + verified: false, + redirect: true, + warn: "High-Throughput is the large-scale lane: pick a Cluster Size and a Large-Scale Preset in the [Playground](#playground) to compose the DP x EP command on top of this hardware's Balanced recipe.", + env: [ + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 8", + "--dcp-size 8", + "--mem-fraction-static 0.85", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + // Same low-latency operating point as GB300, expanded from 2×4 to 4×4. + match: { hw: "gb200", pdMode: "unified", strategy: "low-latency" }, + nnodes: 4, + verified: false, + env: [ + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 16", + "--enable-symm-mem", + "--mem-fraction-static 0.85", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + // Same balanced operating point as GB300; TP/DCP span all 16 ranks. + match: { hw: "gb200", pdMode: "unified", strategy: "balanced" }, + nnodes: 4, + verified: false, + env: [ + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 16", + "--dcp-size 16", + "--mem-fraction-static 0.85", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + // Same high-throughput operating point as GB300; TP/DCP span all 16 ranks. + match: { hw: "gb200", pdMode: "unified", strategy: "high-throughput" }, + nnodes: 4, + verified: false, + redirect: true, + warn: "High-Throughput is the large-scale lane: pick a Cluster Size and a Large-Scale Preset in the [Playground](#playground) to compose the DP x EP command on top of this hardware's Balanced recipe.", + env: [ + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 16", + "--dcp-size 16", + "--mem-fraction-static 0.85", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ----- Prefill role: chunked, on the TP8 platforms. The prefill role keeps + // radix caching, so the Unified 5-slots-per-request state cost still holds + // (pool split rides the calculator-driven ratio). Default is TP8; + // Long-Context is one pipeline stage per GPU, which turns the parallelism + // comm from something you wait for into something the next microbatch hides. + // Both roles must agree on --page-size and --kv-cache-dtype (the transfer + // sanity-checks them at connect), so neither is pinned here. ----- + { + match: { hw: "b300", pdMode: "prefill", strategy: "default" }, + nnodes: 1, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 8", + "--disable-custom-all-reduce", + "--enable-symm-mem", + "--mem-fraction-static 0.85", + "--chunked-prefill-size 16384", + "--max-prefill-tokens 16384", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--disaggregation-mode prefill", + "--disaggregation-transfer-backend nixl", + // Must match the positional bootstrap port the router passes after + // --prefill, or only the decode worker registers. + "--disaggregation-bootstrap-port 8998", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + // PP8xTP1: no TP collective left to accelerate, so --enable-symm-mem is + // dropped rather than carried over from the TP8 cell. + match: { hw: "b300", pdMode: "prefill", strategy: "long-context" }, + nnodes: 1, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 1", + "--pp-size 8", + "--disable-custom-all-reduce", + "--mem-fraction-static 0.90", + "--chunked-prefill-size 16384", + "--max-prefill-tokens 16384", + "--disable-flashinfer-autotune", + "--weight-loader-prefetch-checkpoints", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--disaggregation-mode prefill", + "--disaggregation-transfer-backend nixl", + "--disaggregation-bootstrap-port 8998", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", pdMode: "prefill", strategy: "default" }, + nnodes: 2, + verified: false, + env: [ + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 8", + "--enable-symm-mem", + "--mem-fraction-static 0.85", + "--chunked-prefill-size 16384", + "--max-prefill-tokens 16384", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--disaggregation-mode prefill", + "--disaggregation-transfer-backend nixl", + "--disaggregation-bootstrap-port 8998", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", pdMode: "prefill", strategy: "long-context" }, + nnodes: 2, + verified: false, + env: [ + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 1", + "--pp-size 8", + "--mem-fraction-static 0.90", + "--chunked-prefill-size 16384", + "--max-prefill-tokens 16384", + "--disable-flashinfer-autotune", + "--weight-loader-prefetch-checkpoints", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--disaggregation-mode prefill", + "--disaggregation-transfer-backend nixl", + "--disaggregation-bootstrap-port 8998", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ----- Decode role: the unified cell for the same hw and strategy, plus + // the PD role and transport flags, and re-sized KDA state. + // + // Decode runs the KV cache as a chunk cache, so the unified 5-slots-per- + // request reservation (1 state + ping-pong copies for radix reuse) drops to + // a single slot, and --mamba-radix-cache-strategy stops having any effect. + // In-transfer requests holding a slot before decode starts are the only + // extra, so the pool is sized max-running-requests + extra slots. Pinning + // both keeps the identity explicit instead of leaning on the auto-default, + // which reserves nothing at all once the batch exceeds 32. ----- + { + match: { hw: "b300", pdMode: "decode", strategy: "balanced" }, + nnodes: 1, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 8", + "--dcp-size 8", + "--disable-custom-all-reduce", + "--mem-fraction-static 0.85", + "--disaggregation-decode-extra-slots 16", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--disaggregation-mode decode", + "--disaggregation-transfer-backend nixl", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", pdMode: "decode", strategy: "low-latency" }, + nnodes: 1, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 8", + "--disable-custom-all-reduce", + "--enable-symm-mem", + "--mem-fraction-static 0.85", + "--disaggregation-decode-extra-slots 16", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--disaggregation-mode decode", + "--disaggregation-transfer-backend nixl", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", pdMode: "decode", strategy: "high-throughput" }, + nnodes: 1, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 8", + "--dcp-size 8", + "--disable-custom-all-reduce", + "--mem-fraction-static 0.92", + "--disaggregation-decode-extra-slots 16", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--disaggregation-mode decode", + "--disaggregation-transfer-backend nixl", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", pdMode: "decode", strategy: "low-latency" }, + nnodes: 2, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 16", + "--mem-fraction-static 0.85", + "--disaggregation-decode-extra-slots 16", + "--disable-flashinfer-autotune", + "--watchdog-timeout 3600", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--model-loader-extra-config '{\"enable_multithread_load\": true}'", + "--disaggregation-mode decode", + "--disaggregation-transfer-backend nixl", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", pdMode: "decode", strategy: "balanced" }, + nnodes: 2, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 16", + "--dcp-size 16", + "--mem-fraction-static 0.85", + "--disaggregation-decode-extra-slots 16", + "--disable-flashinfer-autotune", + "--watchdog-timeout 3600", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--model-loader-extra-config '{\"enable_multithread_load\": true}'", + "--disaggregation-mode decode", + "--disaggregation-transfer-backend nixl", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", pdMode: "decode", strategy: "high-throughput" }, + nnodes: 2, + verified: false, + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 16", + "--dcp-size 16", + "--mem-fraction-static 0.92", + "--disaggregation-decode-extra-slots 16", + "--disable-flashinfer-autotune", + "--watchdog-timeout 3600", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--model-loader-extra-config '{\"enable_multithread_load\": true}'", + "--disaggregation-mode decode", + "--disaggregation-transfer-backend nixl", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi350x", pdMode: "decode", strategy: "balanced" }, + nnodes: 1, + verified: false, + env: [ + "SGLANG_USE_AITER=1", + "SGLANG_AITER_K3_OPT=1", + "AITER_FLYDSL_FORCE=1", + "AITER_SITUV2_A8W4=1", + ], + flags: [ + "--model-path {{MODEL_NAME}}", + "--trust-remote-code", + "--tp-size 8", + "--attention-backend triton", + "--dtype bfloat16", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 256", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--disaggregation-mode decode", + "--disaggregation-transfer-backend nixl", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi355x", pdMode: "decode", strategy: "balanced" }, + nnodes: 1, + verified: false, + env: [ + "SGLANG_USE_AITER=1", + "SGLANG_AITER_K3_OPT=1", + "AITER_FLYDSL_FORCE=1", + "AITER_SITUV2_A8W4=1", + ], + flags: [ + "--model-path {{MODEL_NAME}}", + "--trust-remote-code", + "--tp-size 8", + "--attention-backend triton", + "--dtype bfloat16", + "--mem-fraction-static 0.85", + "--cuda-graph-max-bs 256", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--disaggregation-mode decode", + "--disaggregation-transfer-backend nixl", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h100", pdMode: "decode", strategy: "low-latency" }, + nnodes: 4, + verified: false, + env: [ + "NCCL_CUMEM_ENABLE=1", + "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True", + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + "SGLANG_K3_ATTN_RES_MODE=jit", + "SGLANG_MOE_FUSED_GATE_RADIX=1", + "SGLANG_HOST_IP={{LOCAL_IP}}", + "NCCL_SOCKET_IFNAME={{NETWORK_IFACE}}", + "GLOO_SOCKET_IFNAME={{NETWORK_IFACE}}", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 32", + "--ep-size 32", + "--moe-runner-backend marlin", + "--decode-attention-backend flashmla", + "--mem-fraction-static 0.85", + "--dist-timeout 3600", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--disaggregation-mode decode", + "--disaggregation-transfer-backend nixl", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h100", pdMode: "decode", strategy: "balanced" }, + nnodes: 4, + verified: false, + env: [ + "NCCL_CUMEM_ENABLE=1", + "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True", + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + "SGLANG_K3_ATTN_RES_MODE=jit", + "SGLANG_MOE_FUSED_GATE_RADIX=1", + "SGLANG_HOST_IP={{LOCAL_IP}}", + "NCCL_SOCKET_IFNAME={{NETWORK_IFACE}}", + "GLOO_SOCKET_IFNAME={{NETWORK_IFACE}}", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 32", + "--ep-size 32", + "--moe-runner-backend marlin", + "--decode-attention-backend flashmla", + "--mem-fraction-static 0.85", + "--dist-timeout 3600", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--disaggregation-mode decode", + "--disaggregation-transfer-backend nixl", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h100", pdMode: "decode", strategy: "high-throughput" }, + nnodes: 4, + verified: false, + env: [ + "NCCL_CUMEM_ENABLE=1", + "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True", + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + "SGLANG_K3_ATTN_RES_MODE=jit", + "SGLANG_MOE_FUSED_GATE_RADIX=1", + "SGLANG_HOST_IP={{LOCAL_IP}}", + "NCCL_SOCKET_IFNAME={{NETWORK_IFACE}}", + "GLOO_SOCKET_IFNAME={{NETWORK_IFACE}}", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 32", + "--ep-size 32", + "--moe-runner-backend marlin", + "--decode-attention-backend flashmla", + "--mem-fraction-static 0.85", + "--dist-timeout 3600", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--disaggregation-mode decode", + "--disaggregation-transfer-backend nixl", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", pdMode: "decode", strategy: "low-latency" }, + nnodes: 2, + verified: false, + env: [ + "NCCL_MNNVL_ENABLE=1", + "NCCL_CUMEM_ENABLE=1", + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 16", + "--ep-size 16", + "--moe-runner-backend marlin", + "--decode-attention-backend flashmla", + "--enable-symm-mem", + "--mem-fraction-static 0.85", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--disaggregation-mode decode", + "--disaggregation-transfer-backend nixl", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", pdMode: "decode", strategy: "balanced" }, + nnodes: 2, + verified: false, + env: [ + "NCCL_MNNVL_ENABLE=1", + "NCCL_CUMEM_ENABLE=1", + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 16", + "--ep-size 16", + "--moe-runner-backend marlin", + "--decode-attention-backend flashmla", + "--enable-symm-mem", + "--mem-fraction-static 0.85", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--disaggregation-mode decode", + "--disaggregation-transfer-backend nixl", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", pdMode: "decode", strategy: "high-throughput" }, + nnodes: 2, + verified: false, + env: [ + "NCCL_MNNVL_ENABLE=1", + "NCCL_CUMEM_ENABLE=1", + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 16", + "--ep-size 16", + "--moe-runner-backend marlin", + "--decode-attention-backend flashmla", + "--enable-symm-mem", + "--mem-fraction-static 0.90", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--disaggregation-mode decode", + "--disaggregation-transfer-backend nixl", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", pdMode: "decode", strategy: "low-latency" }, + nnodes: 2, + verified: false, + env: [ + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 8", + "--enable-symm-mem", + "--mem-fraction-static 0.85", + "--disaggregation-decode-extra-slots 16", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--disaggregation-mode decode", + "--disaggregation-transfer-backend nixl", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", pdMode: "decode", strategy: "balanced" }, + nnodes: 2, + verified: false, + env: [ + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 8", + "--dcp-size 8", + "--mem-fraction-static 0.85", + "--disaggregation-decode-extra-slots 16", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--disaggregation-mode decode", + "--disaggregation-transfer-backend nixl", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", pdMode: "decode", strategy: "high-throughput" }, + nnodes: 2, + verified: false, + env: [ + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 8", + "--dcp-size 8", + "--mem-fraction-static 0.92", + "--disaggregation-decode-extra-slots 16", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--disaggregation-mode decode", + "--disaggregation-transfer-backend nixl", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb200", pdMode: "decode", strategy: "low-latency" }, + nnodes: 4, + verified: false, + env: [ + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 16", + "--enable-symm-mem", + "--mem-fraction-static 0.85", + "--disaggregation-decode-extra-slots 16", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--disaggregation-mode decode", + "--disaggregation-transfer-backend nixl", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb200", pdMode: "decode", strategy: "balanced" }, + nnodes: 4, + verified: false, + env: [ + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 16", + "--dcp-size 16", + "--mem-fraction-static 0.85", + "--disaggregation-decode-extra-slots 16", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--disaggregation-mode decode", + "--disaggregation-transfer-backend nixl", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb200", pdMode: "decode", strategy: "high-throughput" }, + nnodes: 4, + verified: false, + env: [ + "SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 16", + "--dcp-size 16", + "--mem-fraction-static 0.92", + "--disaggregation-decode-extra-slots 16", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--disaggregation-mode decode", + "--disaggregation-transfer-backend nixl", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + ], + + // Cross-node fabric env (substitute the NIC used by every rank). + multiNodeHints: { + b200: [ + "Low-Latency, Balanced, and High-Throughput use TP16 across both nodes; Long-Context uses TP8 within each PP2 stage.", + "Multi-node K3 needs the cross-node NIC pinned on BOTH ranks:", + " GLOO_SOCKET_IFNAME= # bootstrap interface", + " NCCL_SOCKET_IFNAME= # force NCCL off kube-ipvs0", + " SGLANG_HOST_IP=", + " NCCL_IB_HCA= # RDMA fabrics only", + ], + h100: [ + "Set This node IP separately on each node; use the same cross-node NIC name on all four nodes.", + ], + h200: [ + "Multi-node K3 needs the cross-node NIC pinned on BOTH ranks:", + " GLOO_SOCKET_IFNAME= # e.g. bond0", + " NCCL_SOCKET_IFNAME= # force NCCL off kube-ipvs0", + " SGLANG_HOST_IP=", + ], + }, +};