diff --git a/docs_new/cookbook/autoregressive/Moonshotai/Kimi-K3.mdx b/docs_new/cookbook/autoregressive/Moonshotai/Kimi-K3.mdx index 518fc6d0f..e16398d44 100644 --- a/docs_new/cookbook/autoregressive/Moonshotai/Kimi-K3.mdx +++ b/docs_new/cookbook/autoregressive/Moonshotai/Kimi-K3.mdx @@ -36,16 +36,17 @@ If you do not want to use a Docker image, reproduce the dependency installation Pick your hardware, then the deployment shape and operating point. Node count follows the hardware recipe (B200 2×8, GB200 4×4, H100 4×8, B300 1×8, H200 2×8 — 4×8 on Unified High-Throughput, GB300 2×4, MI350X/MI355X 1×8), so it is not a separate choice. -**PD Mode** — `Unified` serves prefill and decode together. `Prefill` / `Decode` split them into dedicated pools (see [PD disaggregation](#3-4-pd-disaggregation)); `Prefill` ships two strategies on the TP8 platforms, both chunked at 16k: `Default` (TP8) and `Long-Context` (`--pp-size 8 --tp-size 1`, see [Deep PP](#deep-pp-for-long-context-prefill)). +**PD Mode** — `Unified` serves prefill and decode together. `Prefill` / `Decode` split them into dedicated pools (see [PD disaggregation](#3-4-pd-disaggregation)); `Prefill` ships two strategies, both chunked at 16k. On the 8-GPU platforms (B300 1×8, GB300 2×4), `Default` is TP8 and `Long-Context` is `--pp-size 8 --tp-size 1`. On the 16-GPU platforms (B200 2×8, GB200 4×4), both are `--pp-size 16 --tp-size 1` and differ only in `--mem-fraction-static` (0.85 vs 0.90) — deep PP is the throughput shape there, not just the long-context one (see [Deep PP](#deep-pp-for-prefill)). **Strategy** — the operating point within that shape: -- **Low-Latency** — plain TP, no DCP. For chat. -- **Balanced** — the accuracy-preserving default: TP16/DCP16 on B200/GB200, TP8/DCP8 on B300/GB300, TP8 ROCm/AITER on MI35x. -- **High-Throughput** — the large-scale lane: pick a **Cluster Size** and **Large-Scale Preset** in the Playground ([details](#large-scale-presets)). On H100: Balanced plus `extra_buffer_lazy`; on H200 the cell itself widens to 4×8 TP32/EP32 at `--mem-fraction-static 0.90`. -- **Long-Context** — B200 only: TP8/PP2 splits KV and KDA state across two pipeline stages. +- **Low-Latency** — no DCP, so the MLA KV stays TP-replicated. For chat. B200 splits its two nodes into PP2 × TP8; every other platform is flat TP. +- **Balanced** — the accuracy-preserving default: PP2 × DCPEP8 on B200 (the two pipeline stages and DCP8 split KV and KDA state), TP16/DCP16 on GB200, TP8/DCP8 on B300/GB300, TP8 ROCm/AITER on MI35x. +- **High-Throughput** — the large-scale lane: pick a **Cluster Size** and **Large-Scale Preset** in the Playground ([details](#large-scale-presets)). The cell itself is Balanced, except on H100 (plus `extra_buffer_lazy`) and H200 (widens to 4×8 TP32/EP32 at `--mem-fraction-static 0.90`). -**Spec Decode** — independent of the strategy: DSPARK layers onto any `pp_size == 1` recipe, proposing 7 draft tokens per step (tune in the Playground); DFLASH has no published draft checkpoint. Its win is largest on short interactive traffic and fades as the prompt grows. +`Long-Context` appears only under the `Prefill` PD mode; for long-context unified serving on B200, start from High-Throughput and raise `--context-length`. + +**Spec Decode** — layers onto the strategy without changing it, on every platform except B200. DSPARK proposes 7 draft tokens per step (tune in the Playground) and requires `pp_size == 1`, so on B200 it also drops the pipeline and re-lays the same 16 GPUs flat: PP2 × TP8 → TP16, PP2 × DCPEP8 → DCPEP16. DFLASH has no published draft checkpoint. The win is largest on short interactive traffic and fades as the prompt grows. `--mamba-full-memory-ratio` is the one sizing flag, computed live: set your average request length in the [Mamba ratio calculator](#mamba-ratio-calculator); everything else follows the panels, and the result is pinned into the command. @@ -126,7 +127,7 @@ Capacity levers, all in the Playground. Each trades precision or cache behavior Speculation: DSPARK holds block size + 1 (= 8) intermediate states per request — the calculator folds this in — and an unset `--max-running-requests` resets to 48 under spec (the command panel reminds you; set it explicitly to raise). -**MoE runner.** Leave `--moe-runner-backend` unset on Blackwell: FlashInfer MXFP4 (W4A8, prebuilt trtllm-gen SiTU kernels) when the cubin pool is installed, Marlin (W4A16) otherwise; H100/H200 pin Marlin. The published Docker images already provision the **SiTU cubin pool**; to install it independently, run the same flow as the Dockerfile: +**MoE runner.** Leave `--moe-runner-backend` unset on Blackwell and it resolves to FlashInfer MXFP4 (W4A8, prebuilt trtllm-gen SiTU kernels) when the cubin pool is installed, Marlin (W4A16) otherwise; H100/H200 pin Marlin. The B200 Balanced and High-Throughput cells pin `flashinfer_mxfp4` explicitly because that is the shape they were brought up on — on an install without the pool, drop the flag to fall back to Marlin. The published Docker images already provision the **SiTU cubin pool**; to install it independently, run the same flow as the Dockerfile: ```bash wget https://github.com/sgl-project/whl/releases/download/trtllm_gen_moe_cubin_20260617/trtllm_gen_moe_cubin_pool_20260617_v0613rc1.zip @@ -137,7 +138,7 @@ export SGLANG_TRTLLM_GEN_MOE_CUBIN_POOL=/opt/trtllm_gen_moe_cubin_pool/trtllm_ge Remaining kernel sources JIT once from the public `flashinfer` wheel (a few minutes, cached). -**Attention backend.** Leave all three attention knobs unset on Blackwell: K3 resolves prefill, decode, and — under DSPARK — verification as a set (`trtllm_mla` across the board; `cutedsl_mla` takes decode and verification under DCP). On the non-DCP recipes, setting any one of the three cancels the auto-resolution for the others. H100/H200 pin `flashmla` for decode. +**Attention backend.** Leave all three attention knobs unset on Blackwell: K3 resolves prefill, decode, and — under DSPARK — verification as a set (`trtllm_mla` across the board; `cutedsl_mla` takes decode and verification under DCP). On the non-DCP recipes, setting any one of the three cancels the auto-resolution for the others. The B200 Balanced and High-Throughput cells pin `--decode-attention-backend cutedsl_mla`, which is what auto-resolution picks for those DCP recipes anyway — it is written out because it is the shape they were brought up on, not because it changes the resolution. H100/H200 pin `flashmla` for decode. **Context length.** `--context-length` bounds the longest accepted request plus some context-scaled buffers; it does not size the KV pool. For long context the lever that adds capacity is `fp8_e4m3` KV. @@ -149,20 +150,20 @@ Remaining kernel sources JIT once from the public `flashinfer` wheel (a few minu |---|---|---| | B300 1×8 | TP8 (+DCP8) | accuracy-first defaults on Low-Latency and Balanced | | GB300 2×4 | TP8/DCP8 | MNNVL transport and cuMem auto-detected | -| B200 2×8 | TP16 (+DCP16); Long-Context TP8/PP2, 128K ctx, 8192 chunked prefill | DSPARK off on Long-Context (`pp_size == 1` required) | +| B200 2×8 | PP2 × TP8 on Low-Latency, PP2 × DCPEP8 on Balanced and High-Throughput. DSPARK re-lays the same 16 GPUs as TP16 / TP16+DCP16+EP16. PD prefill is TP1 × PP16 | Unified serves all three operating points; `Long-Context` is a `Prefill`-only strategy | | GB200 4×4 | TP16/DCP16 | MNNVL auto-detected | | H200 2×8 (4×8 on Unified High-Throughput) | TP16/EP16 + symm-mem, Marlin + FlashMLA; High-Throughput widens to TP32/EP32 over 4 nodes at mem-frac 0.90 with `extra_buffer_lazy` | same block on every node; export the cross-node NIC (`GLOO_SOCKET_IFNAME` / `NCCL_SOCKET_IFNAME`, `SGLANG_HOST_IP`); keep `NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1` | | H100 4×8 | TP32/EP32, Marlin + FlashMLA | SM90a build of the K3 image; pin NCCL/Gloo to the same NIC on all nodes; least post-weight headroom (80 GB) | | MI350X/MI355X 1×8 | TP8 ROCm/AITER | AITER A8W4 FlyDSL MoE, Triton attention, graph bs up to 256; DSPARK supported | -**DCP notes** (Blackwell Balanced / High-Throughput): +**DCP notes** — the DCP cells are Balanced and High-Throughput on every Blackwell platform, in both the `Unified` and `Decode` roles: - DCP is the only axis that shards the TP-replicated MLA KV; Low-Latency skips it. - Leave `--dcp-comm-backend` unset (fabric-resolved: `fi_a2a` on GB200/GB300, `a2a` on B200/B300). - No `--enable-symm-mem` under DCP (force-disabled for decode-graph correctness). - Explicit `tokenspeed_mla` force-rewrites `--kv-cache-dtype` to fp8; the default `cutedsl_mla` serves either dtype. - Calculator ratios run well above 1 here (`r > 1` is legal): `bfloat16` state buys admission, `fp8` KV buys context. -- Don't add EP — its a2a buffers reclaim the KV that DCP buys. Compose only to measure. +- Don't use EP with an a2a backend: a2a buffers reclaim the KV that DCP buys. Compose only to measure. a2a backend is set when `SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1` or `--moe-a2a-backend` is set. No cell has a serving round in this exact shape — treat them as starting points to verify. @@ -244,7 +245,7 @@ Pending update... K3's hybrid HiCache tiers the paged MLA KV **and** the KDA/mamba state across L1 (GPU) / L2 (host) / L3 (Mooncake) — enable it from the **HiCache** card in the [Playground above](#playground) for long multi-turn workloads. -- On the DCP recipes (Blackwell Balanced / High-Throughput), the host tiers are not fully DCP-aware yet: **L3 always, and L1+L2 with Spec Decode on, drop the DCP flags and run plain TP** (the command hints call it out — per-request KV capacity shrinks accordingly). L1+L2 NOSPEC keeps DCP. +- On the DCP recipes (Blackwell Balanced / High-Throughput, in both the `Unified` and `Decode` roles), the host tiers are not fully DCP-aware yet: **L3 always, and L1+L2 with Spec Decode on, drop the DCP flags** (the command hints call it out — per-request KV capacity shrinks accordingly). L1+L2 with Spec Decode off keeps DCP. Only DCP goes: the MLA KV reverts to TP-replicated, but the cell's other parallelism stays, so B300/GB300/GB200 land on plain TP while B200 Unified keeps its `--pp-size 2` / `--ep-size`. - Low-Latency and the Hopper recipes take all tiers unchanged. @@ -257,13 +258,13 @@ PD splits prefill and decode into separate server groups; because K3 is hybrid, - **Ports**: prefill `30000`, decode `30100` (derived ZMQ/dist ranges must not collide on a shared host). The positional `8998` after `--prefill` must match `--disaggregation-bootstrap-port`, or only the decode worker registers. - **Decode state pool**: chunk cache — one slot per request; `--mamba-radix-cache-strategy` is inert. Keep `--disaggregation-decode-extra-slots` pinned: unpinned it defaults to twice the batch below 32 requests and **zero** above. -#### Deep PP for long-context prefill +#### Deep PP for prefill -The `Long-Context` prefill strategy is `--pp-size 8 --tp-size 1`: pipeline P2P overlaps the next microbatch's compute, unlike TP/EP collectives, and each stage owns whole layers (a clean slice of KV and state). +Deep PP is `--tp-size 1` with one pipeline stage per GPU — `--pp-size 8` on B300/GB300, `--pp-size 16` on B200/GB200. Pipeline P2P overlaps the next microbatch's compute, unlike TP/EP collectives, and each stage owns whole layers (a clean slice of KV and state). `--tp-size 1` is also what buys context: above TP1 the MLA KV is replicated across the TP ranks, so TP2 × PP8 holds roughly half the tokens of TP1 × PP16 for the same memory. -- Use all eight stages; a shallow split still pays the in-stage all-reduce and can lose to TP8. -- Pays only with several requests in flight (hence `Default` = TP8). -- DSPARK off (`pp_size == 1` required). +- Use one stage per GPU; a shallow split still pays the in-stage all-reduce and can lose to flat TP. +- Pays only with several requests in flight. On the 8-GPU platforms that is why `Default` stays TP8; on the 16-GPU platforms deep PP wins at the Default operating point too, so both strategies use it — measured on GB200 at ISL 8192 / concurrency 32, PP16 × TP1 reached 4550 prefill tok/s/GPU vs 3596 (PP8 × TP2), 2407 (TEP16), and 1652 (TP16). Below concurrency ~8 the pipeline cannot fill and TEP16 leads instead (1947 vs 1227) — use `--tp-size 16 --ep-size 16` there. +- DSPARK off (`pp_size == 1` required) — on B200/GB200 that applies to `Default` as well. - Fan one prefill role out to several decode roles; budget for in-transfer KV on the decode side. diff --git a/docs_new/src/snippets/_deployment.jsx b/docs_new/src/snippets/_deployment.jsx index 37423e511..af356f1ba 100644 --- a/docs_new/src/snippets/_deployment.jsx +++ b/docs_new/src/snippets/_deployment.jsx @@ -502,10 +502,34 @@ export const Deployment = ({ config, benchmarks }) => { // An overlay option may also REMOVE cell flags, declared as `stripPrefixes` // (a static list, or a function of the selection). L3 uses it to drop the // whole DCP operating point, which the server rejects with an L3 backend. - const overlayStrip = (cellFlags, sel) => { + // + // Overlay flags append, except a flag whose family the same overlay stripped: + // that one is spliced back where the stripped flag was, so a rewritten + // parallelism block (DSPARK folding a pipeline flat) stays put instead of + // landing past the --host/--port tail with the multi-node trio behind it. + const overlayCompose = (cellFlags, sel) => { const strip = overlayPart(sel, "stripPrefixes"); - if (!strip.length) return [...(cellFlags || [])]; - return (cellFlags || []).filter((f) => !strip.includes(f.split(/[\s=]/)[0])); + const add = overlayPart(sel, "flags"); + if (!strip.length) return [...(cellFlags || []), ...add]; + const used = new Set(); + // Consumed once, so a family the cell carries twice is not emitted twice. + const replacementsFor = (tok) => { + const out = []; + add.forEach((f, i) => { + if (used.has(i) || f.split(/[\s=]/)[0] !== tok) return; + used.add(i); + out.push(f); + }); + return out; + }; + const out = []; + for (const f of (cellFlags || [])) { + const tok = f.split(/[\s=]/)[0]; + if (!strip.includes(tok)) out.push(f); + else out.push(...replacementsFor(tok)); + } + add.forEach((f, i) => { if (!used.has(i)) out.push(f); }); + return out; }; // ==== end MIRROR ==== const findCell = (cells, sel) => @@ -653,7 +677,6 @@ export const Deployment = ({ config, benchmarks }) => { // `flags` / `env` / `hints` may each be a function of the whole selection, so an // "Auto" option can resolve against another row (draft tokens per strategy). - const overlayFlags = (sel) => overlayPart(sel, "flags"); const overlayEnv = (sel) => overlayPart(sel, "env"); const overlayHints = (sel) => overlayPart(sel, "hints"); @@ -664,7 +687,7 @@ export const Deployment = ({ config, benchmarks }) => { const nnodes = cellNnodes(cell, sel); const multinode = nnodes > 1; const cellEnv = [...(cell.env || []), ...overlayEnv(sel)]; - const flags = [...overlayStrip(cell.flags, sel), ...overlayFlags(sel)]; + const flags = overlayCompose(cell.flags, sel); if (multinode) { // Insert the multi-node trio after the last parallelism flag, // falling back to right after --model-path. @@ -1224,7 +1247,7 @@ export const Deployment = ({ config, benchmarks }) => { // the Spec Decode overlay as well as the cell. SGLang resets // --max-running-requests to 48 when spec is on and it's unset; verified for both // EAGLE/MTP and DSPARK (server_args reports max_running_requests=48 either way). - const effFlags = cell ? [...overlayStrip(cell.flags, sel), ...overlayFlags(sel)] : []; + const effFlags = cell ? overlayCompose(cell.flags, sel) : []; const specAlgoFlag = effFlags.find( (f) => f.split(/[\s=]/)[0] === "--speculative-algorithm"); const specMrrFlag = effFlags.find( diff --git a/docs_new/src/snippets/_playground.jsx b/docs_new/src/snippets/_playground.jsx index 9cd398e9e..5bf7f60e2 100644 --- a/docs_new/src/snippets/_playground.jsx +++ b/docs_new/src/snippets/_playground.jsx @@ -88,15 +88,39 @@ export const Playground = ({ config }) => { // An overlay option may also REMOVE cell flags, declared as `stripPrefixes` // (a static list, or a function of the selection). L3 uses it to drop the // whole DCP operating point, which the server rejects with an L3 backend. - const overlayStrip = (cellFlags, sel) => { + // + // Overlay flags append, except a flag whose family the same overlay stripped: + // that one is spliced back where the stripped flag was, so a rewritten + // parallelism block (DSPARK folding a pipeline flat) stays put instead of + // landing past the --host/--port tail with the multi-node trio behind it. + const overlayCompose = (cellFlags, sel) => { const strip = overlayPart(sel, "stripPrefixes"); - if (!strip.length) return [...(cellFlags || [])]; - return (cellFlags || []).filter((f) => !strip.includes(f.split(/[\s=]/)[0])); + const add = overlayPart(sel, "flags"); + if (!strip.length) return [...(cellFlags || []), ...add]; + const used = new Set(); + // Consumed once, so a family the cell carries twice is not emitted twice. + const replacementsFor = (tok) => { + const out = []; + add.forEach((f, i) => { + if (used.has(i) || f.split(/[\s=]/)[0] !== tok) return; + used.add(i); + out.push(f); + }); + return out; + }; + const out = []; + for (const f of (cellFlags || [])) { + const tok = f.split(/[\s=]/)[0]; + if (!strip.includes(tok)) out.push(f); + else out.push(...replacementsFor(tok)); + } + add.forEach((f, i) => { if (!used.has(i)) out.push(f); }); + return out; }; // ==== end MIRROR ==== const withOverlay = (cell, sel) => (cell && { ...cell, - flags: [...overlayStrip(cell.flags, sel), ...overlayPart(sel, "flags")], + flags: overlayCompose(cell.flags, sel), env: [...(cell.env || []), ...overlayPart(sel, "env")], }) || cell; // Shared with `_deployment.jsx` (HOST/PORT/etc. unified across the page). diff --git a/docs_new/src/snippets/configs/moonshotai/kimi-k3-benchmarks.jsx b/docs_new/src/snippets/configs/moonshotai/kimi-k3-benchmarks.jsx index b975a276a..525e0aeba 100644 --- a/docs_new/src/snippets/configs/moonshotai/kimi-k3-benchmarks.jsx +++ b/docs_new/src/snippets/configs/moonshotai/kimi-k3-benchmarks.jsx @@ -5,7 +5,6 @@ export const benchmarks = [ { match: { hw: "b200", pdMode: "unified", strategy: "low-latency" } }, { match: { hw: "b200", pdMode: "unified", strategy: "balanced" } }, { match: { hw: "b200", pdMode: "unified", strategy: "high-throughput" } }, - { match: { hw: "b200", pdMode: "unified", strategy: "long-context" } }, { match: { hw: "mi350x", pdMode: "unified", strategy: "balanced" } }, { match: { hw: "mi355x", pdMode: "unified", strategy: "balanced" } }, { match: { hw: "h100", pdMode: "unified", strategy: "low-latency" } }, diff --git a/docs_new/src/snippets/configs/moonshotai/kimi-k3.jsx b/docs_new/src/snippets/configs/moonshotai/kimi-k3.jsx index f8cd68346..2dad47171 100644 --- a/docs_new/src/snippets/configs/moonshotai/kimi-k3.jsx +++ b/docs_new/src/snippets/configs/moonshotai/kimi-k3.jsx @@ -10,12 +10,123 @@ export const config = { modelName: "Kimi-K3", - // B300 (1×8 TP8), GB300 (2×4 TP8 MNNVL), B200 (2×8 TP16, or TP8/PP2 for - // Long-Context), GB200 (4×4 TP16 MNNVL), H200 (2×8 TP16/EP16, or 4×8 TP32/EP32 - // for High-Throughput), H100 (4×8 TP32/EP32), and MI350X/MI355X (1×8 TP8) have - // serving recipes. + // B300 (1×8 TP8), GB300 (2×4 TP8 MNNVL), B200 (2×8: NOSPEC is PP2 × TP8 on + // Low-Latency, PP2 × DCPEP8 on Balanced and High-Throughput, while DSPARK + // re-lays the same 16 as flat TP16 / DCPEP16), GB200 (4×4 TP16 MNNVL), + // H200 (2×8 TP16/EP16, or 4×8 TP32/EP32 for High-Throughput), H100 + // (4×8 TP32/EP32), and MI350X/MI355X (1×8 TP8) have serving recipes. supportedHardware: ["b300", "gb300", "b200", "gb200", "h200", "h100", "mi350x", "mi355x"], + // ---- Cell introspection (config-internal; the engines ignore these keys) ---- + // + // The overlay callbacks below are handed only the SELECTION, never the resolved + // cell, so anything they need to know about the recipe has to be looked up. + // Doing that lookup here — instead of restating it as per-hardware tables — + // keeps one source of truth: the cell's own flags. A recipe that gains DCP, or + // drops a pipeline stage, or a whole new platform, then needs no edit to the + // overlay logic. The tables it replaces had to be revised by hand for every + // such change, and a missed revision is invisible (the panel just quietly + // emits a shape the server rejects). + // + // Mintlify strips module-level statements from snippets, so these live as + // properties on the exported object and are called as `config.cellFor(s)`. + cellFor(s) { + return (config.cells || []).find( + (c) => c.match.hw === s.hw && + c.match.pdMode === s.pdMode && + c.match.strategy === s.strategy); + }, + // Flags are authored as "--name value" / "--name=value" strings, matching how + // _deployment.jsx splits them for stripPrefixes. + flagOf(cell, name) { + return ((cell || {}).flags || []).find((f) => f.split(/[\s=]/)[0] === name); + }, + // Absent parallelism flags mean 1 (SGLang's own default), so callers can do + // arithmetic without null checks. + sizeOf(cell, name) { + const f = config.flagOf(cell, name); + return f ? Number(f.split(/[\s=]/)[1]) || 1 : 1; + }, + // Does this recipe shard the TP-replicated MLA KV? Replaces the per-platform + // "which cells carry DCP" tables the HiCache tiers used to hardcode. + hasDcp(s) { + return !!config.flagOf(config.cellFor(s), "--dcp-size"); + }, + // A pp_size == 1 speculative algorithm cannot run a pipelined recipe. Cells + // that would rather be re-laid flat than blocked opt in with + // `specCollapsePp: true`; the rest are simply unavailable under speculation. + isPipelined(s) { + return config.sizeOf(config.cellFor(s), "--pp-size") > 1; + }, + specCollapses(s) { + return config.isPipelined(s) && !!(config.cellFor(s) || {}).specCollapsePp; + }, + // The playground's chip `disable` is declarative only (no predicates), so the + // DCP-carrying recipes are enumerated — but generated from the cells here + // rather than typed out per platform, so it is the same single source of truth + // as hasDcp() above. One rule per (hardware, PD role): which strategies carry + // DCP varies along both axes — B200 reaches DCP only at High-Throughput when + // unified, but carries it on Balanced too in the decode role — and keying on + // hw+strategy alone (as the hand-written rules did) cannot express that without + // over-matching one role or under-matching the other. + get dcpStorageDisableRules() { + const groups = new Map(); + for (const c of (config.cells || [])) { + if (!config.flagOf(c, "--dcp-size")) continue; + const k = `${c.match.hw}|${c.match.pdMode}`; + if (!groups.has(k)) groups.set(k, new Set()); + groups.get(k).add(c.match.strategy); + } + return [...groups].map(([k, strategies]) => { + const [hw, pdMode] = k.split("|"); + return { + when: { hw: [hw], pdMode: [pdMode], strategy: [...strategies], + hicache: ["l2"], spec: ["none"] }, + reason: "This recipe runs DCP, and a storage backend (L3) under DCP is rejected at startup. Switch HiCache to L3 in the Deploy panel (that drops DCP), or stay on L1+L2.", + }; + }); + }, + // Companion to the hand-written TP/DP-Attention rules below, which enumerate + // the platforms whose recipes are too small for a 16-rank knob. Those cover the + // flat recipes; this covers the other reason a knob cannot widen — the recipe + // spends its ranks on pipeline stages instead. Only cells that never collapse + // the pipeline are listed (a specCollapsePp cell becomes flat under DSPARK and + // is handled by its own spec-keyed rule), so no `spec` key is needed here. + get pipelinedKnobDisableRules() { + const groups = new Map(); + for (const c of (config.cells || [])) { + if (config.sizeOf(c, "--pp-size") <= 1 || c.specCollapsePp) continue; + const k = `${c.match.hw}|${c.match.pdMode}`; + if (!groups.has(k)) groups.set(k, new Set()); + groups.get(k).add(c.match.strategy); + } + return [...groups].map(([k, strategies]) => { + const [hw, pdMode] = k.split("|"); + return { + when: { hw: [hw], pdMode: [pdMode], strategy: [...strategies] }, + reason: "This recipe spends its ranks on pipeline stages (--pp-size > 1 with a small --tp-size), so widening the attention parallelism would need more GPUs than the deployment has. Pick a flat recipe to change TP or DP-Attention.", + }; + }); + }, + // Fold the pipeline back into the other axes at constant world size: + // TP8 × PP2 -> TP16, and DCP8/EP8 scale with it -> DCP16/EP16. Derived from + // the cell rather than written out, so it stays right if a cell is re-shaped. + specCollapsedFlags(s) { + const cell = config.cellFor(s); + const pp = config.sizeOf(cell, "--pp-size"); + const out = [`--tp-size ${config.sizeOf(cell, "--tp-size") * pp}`]; + // HiCache under DCP rejects speculative decoding, so when a host tier is on + // the DCP half is dropped instead of scaled (the HiCache options' own + // stripPrefixes reach cell flags only, never these overlay flags). + if (config.flagOf(cell, "--dcp-size") && s.hicache !== "l2" && s.hicache !== "l3") { + out.push(`--dcp-size ${config.sizeOf(cell, "--dcp-size") * pp}`); + } + if (config.flagOf(cell, "--ep-size")) { + out.push(`--ep-size ${config.sizeOf(cell, "--ep-size") * pp}`); + } + return out; + }, + // Single checkpoint and a single shipped quantization (MXFP4), so neither is a // reader-facing axis. Node count is fixed by the hardware recipe (B200 2x8, // H100 4x8, B300 1x8, H200 2x8 — 4x8 on Unified High-Throughput, GB200 4x4, @@ -41,13 +152,7 @@ export const config = { { id: "balanced", label: "Balanced", showWhen: (s) => s.pdMode !== "prefill" }, { id: "high-throughput", label: "High-Throughput", showWhen: (s) => s.pdMode !== "prefill" }, { id: "default", label: "Default", showWhen: (s) => s.pdMode === "prefill" }, - { - id: "long-context", - label: "Long-Context", - showWhen: (s) => - s.pdMode === "prefill" || - (s.pdMode === "unified" && s.hw === "b200"), - }, + { id: "long-context", label: "Long-Context", showWhen: (s) => s.pdMode === "prefill" }, ], }, ], @@ -64,15 +169,32 @@ export const config = { { id: "dspark", label: "DSPARK", - disabled: (s) => s.strategy === "long-context", + // DSPARK requires pp_size == 1, so a pipelined recipe is either re-laid + // flat (cells opting in with `specCollapsePp`) or unavailable. Both + // branches read --pp-size off the cell, so no platform is named here: + // whichever recipes happen to be pipelined are the ones affected. + // Combinations with no published cell are left alone here — they render + // an empty command panel either way, and gating on cell existence would + // change this button on every such combination across every platform, + // which is a separate decision from how speculation reads a recipe. + disabled: (s) => config.isPipelined(s) && !config.specCollapses(s), disableReason: - "The Long-Context recipes use pipeline parallelism (--pp-size 2 on B200 Unified, --pp-size 8 on Prefill), while DSPARK currently requires pp_size == 1.", + "This recipe is pipelined (--pp-size > 1) and DSPARK requires pp_size == 1. Pick a recipe that runs a single pipeline stage, or run this one NOSPEC.", + // Where a cell does opt in, DSPARK rewrites its parallelism instead of + // layering on top: the pipeline is stripped and folded into the other + // axes at constant world size (see specCollapsedFlags). Cells that are + // already flat are untouched. + stripPrefixes: (s) => + config.specCollapses(s) + ? ["--tp-size", "--pp-size", "--dcp-size", "--ep-size"] + : [], // Every DSPARK recipe layers ReplaySSM on: it moves the per-draft // intermediate SSM states onto a fixed ring, lifting the concurrency // the state pool admits (needs the Triton decode kernel, the K3 // default). Only the PD prefill role opts out — it never runs verify // and rejects the flag at startup. flags: (s) => [ + ...(config.specCollapses(s) ? config.specCollapsedFlags(s) : []), "--speculative-algorithm DSPARK", "--speculative-draft-model-path RadixArk/Kimi-K3-DSpark", "--speculative-dspark-block-size 7", @@ -118,19 +240,19 @@ export const config = { // point is dropped instead of blocking the option, same as L3 does. The // ratio calculator reads --dcp-size off this command, so dropping it also // re-solves --mamba-full-memory-ratio for the plain-TP shape. + // + // Which cells carry DCP is read off the cells themselves, so adding or + // reshaping a DCP recipe needs no edit here. stripPrefixes: (s) => - ["b300", "gb300", "b200", "gb200"].includes(s.hw) && - ["balanced", "high-throughput"].includes(s.strategy) && - s.spec !== "none" + s.spec !== "none" && config.hasDcp(s) ? ["--dcp-size", "--dcp-comm-backend"] : [], hints: (s) => - ["b300", "gb300", "b200", "gb200"].includes(s.hw) && - ["balanced", "high-throughput"].includes(s.strategy) && - s.spec !== "none" + s.spec !== "none" && config.hasDcp(s) ? [ "HiCache under DCP rejects speculative decoding, so this recipe drops DCP", - "and runs plain TP. Per-request context is far shorter than the DCP", + "and serves the MLA KV TP-replicated (any PP/EP in the cell stays).", + "Per-request context is far shorter than the DCP", "version — DCP is what buys KV capacity. Run the cell NOSPEC to keep DCP.", ] : [], @@ -145,23 +267,24 @@ export const config = { env: ["SGLANG_HICACHE_MOONCAKE_CONFIG_PATH={{MOONCAKE_CONFIG}}"], // L3 under DCP is rejected at startup (the rank-0 replicated-MLA backup // and the storage keys are not dcp_rank-aware), so on the DCP recipes L3 - // drops the DCP operating point and runs plain TP instead. The ratio + // drops the DCP operating point instead (only DCP — any PP/EP in the + // cell stays, so on B200 the result is not plain TP). The ratio // calculator reads --dcp-size off this command, so dropping it also // re-solves --mamba-full-memory-ratio for the plain-TP shape. + // + // Same cell-derived DCP test as the L1+L2 option above, except this one + // fires NOSPEC too rather than only under speculation. stripPrefixes: (s) => - ["b300", "gb300", "b200", "gb200"].includes(s.hw) && - ["balanced", "high-throughput"].includes(s.strategy) - ? ["--dcp-size", "--dcp-comm-backend"] - : [], + config.hasDcp(s) ? ["--dcp-size", "--dcp-comm-backend"] : [], hints: (s) => [ "L3 also needs a mooncake_master process on rank 0 and the config file", "above present on every rank — the launch command alone is not enough.", - ...(["b300", "gb300", "b200", "gb200"].includes(s.hw) && - ["balanced", "high-throughput"].includes(s.strategy) + ...(config.hasDcp(s) ? [ "L3 storage keys are not dcp_rank-aware yet, so this recipe drops DCP", - "and runs plain TP. Concurrency lands on a similar target, but", + "and serves the MLA KV TP-replicated (any PP/EP in the cell stays).", + "Concurrency lands on a similar target, but", "per-request context is far shorter than the DCP version — DCP is", "what buys KV capacity.", ] @@ -239,35 +362,49 @@ export const config = { null, 8, { value: 16, - disable: [ + get disable() { return [ { when: { hw: ["b300", "gb300"] }, reason: "TP=16 needs 16 ranks; the B300 and GB300 recipes have 8 ranks.", }, { - when: { hw: ["b200"], strategy: ["long-context"] }, - reason: "The B200 Long-Context recipe already uses all 16 GPUs as TP8 × PP2; changing TP to 16 would require 32 ranks.", + when: { hw: ["b200"], pdMode: ["unified"], spec: ["none"] }, + reason: "With Spec Decode off, the B200 Unified recipes already use all 16 GPUs as TP8 × PP2, so changing TP to 16 would require 32 ranks. Switch Spec Decode to DSPARK — it drops the pipeline and re-lays the same GPUs as flat TP16.", }, - ], + ...config.pipelinedKnobDisableRules, + ]; }, }, ]}, { id: "dpAttn", label: "DP-Attention", values: [ null, false, 2, 4, - { value: 8, disable: { hw: ["b300", "gb300"] }, - disableReason: "On an 8-rank deployment (B300 1×8, GB300 2×4) dp=8 leaves attn_tp=1, so each rank holds the full unsharded MLA KV and OOMs — prefer dp=2/attn_tp=4." }, + { + value: 8, + get disable() { return [ + { + when: { hw: ["b300", "gb300"] }, + reason: "On an 8-rank deployment (B300 1×8, GB300 2×4) dp=8 leaves attn_tp=1, so each rank holds the full unsharded MLA KV and OOMs — prefer dp=2/attn_tp=4.", + }, + { + when: { hw: ["b200"], pdMode: ["unified"], spec: ["none"] }, + reason: "With Spec Decode off, the B200 Unified recipes run TP8 within each PP2 stage, so dp=8 leaves attn_tp=1 and each rank holds the full unsharded MLA KV — prefer dp=2/attn_tp=4, or switch Spec Decode to DSPARK for the flat TP16 shape.", + }, + ...config.pipelinedKnobDisableRules, + ]; }, + }, { value: 16, - disable: [ + get disable() { return [ { when: { hw: ["b300", "gb300"] }, reason: "DP-Attention=16 needs 16 TP ranks; the B300 and GB300 recipes have 8.", }, { - when: { hw: ["b200"], strategy: ["long-context"] }, - reason: "The B200 Long-Context recipe uses TP8 within each PP stage, so DP-Attention cannot exceed 8.", + when: { hw: ["b200"], pdMode: ["unified"], spec: ["none"] }, + reason: "With Spec Decode off, the B200 Unified recipes use TP8 within each PP2 stage, so DP-Attention cannot exceed 8. Switch Spec Decode to DSPARK for the flat TP16 shape.", }, - ], + ...config.pipelinedKnobDisableRules, + ]; }, }, ], labels: { "auto": "Auto", "false": "Off" } }, @@ -311,16 +448,17 @@ export const config = { null, 1, 2, 4, 8, { value: 16, - disable: [ + get disable() { return [ { when: { hw: ["b300", "gb300"] }, reason: "EP=16 needs 16 TP ranks; the B300 and GB300 recipes have 8.", }, { - when: { hw: ["b200"], strategy: ["long-context"] }, - reason: "The B200 Long-Context recipe uses TP8 within each PP stage, so EP cannot exceed 8.", + when: { hw: ["b200"], pdMode: ["unified"], spec: ["none"] }, + reason: "With Spec Decode off, the B200 Unified recipes use TP8 within each PP2 stage, so EP cannot exceed 8. Switch Spec Decode to DSPARK for the flat TP16 shape.", }, - ], + ...config.pipelinedKnobDisableRules, + ]; }, }, ]}, }, @@ -373,17 +511,13 @@ export const config = { backends: [ { id: null, label: "Auto" }, { id: "file", label: "File", - disable: [{ when: { hw: ["b300", "gb300", "b200", "gb200"], strategy: ["balanced", "high-throughput"], hicache: ["l2"], spec: ["none"] }, - reason: "This recipe runs DCP, and a storage backend (L3) under DCP is rejected at startup. Switch HiCache to L3 in the Deploy panel (that drops DCP), or stay on L1+L2." }] }, + get disable() { return config.dcpStorageDisableRules; } }, { id: "mooncake", label: "Mooncake", - disable: [{ when: { hw: ["b300", "gb300", "b200", "gb200"], strategy: ["balanced", "high-throughput"], hicache: ["l2"], spec: ["none"] }, - reason: "This recipe runs DCP, and a storage backend (L3) under DCP is rejected at startup. Switch HiCache to L3 in the Deploy panel (that drops DCP), or stay on L1+L2." }] }, + get disable() { return config.dcpStorageDisableRules; } }, { id: "hf3fs", label: "HF3FS", - disable: [{ when: { hw: ["b300", "gb300", "b200", "gb200"], strategy: ["balanced", "high-throughput"], hicache: ["l2"], spec: ["none"] }, - reason: "This recipe runs DCP, and a storage backend (L3) under DCP is rejected at startup. Switch HiCache to L3 in the Deploy panel (that drops DCP), or stay on L1+L2." }] }, + get disable() { return config.dcpStorageDisableRules; } }, { id: "nixl", label: "NiXL", - disable: [{ when: { hw: ["b300", "gb300", "b200", "gb200"], strategy: ["balanced", "high-throughput"], hicache: ["l2"], spec: ["none"] }, - reason: "This recipe runs DCP, and a storage backend (L3) under DCP is rejected at startup. Switch HiCache to L3 in the Deploy panel (that drops DCP), or stay on L1+L2." }] }, + get disable() { return config.dcpStorageDisableRules; } }, ], writePolicies: [ { id: "auto", label: "Auto" }, @@ -575,7 +709,7 @@ export const config = { "--ep-size", "--ep", "--expert-parallel-size", "--enable-dp-attention", "--dp-size", "--enable-dp-lm-head", "--dcp-size", "--dcp-comm-backend", - // The B200 Long-Context cell carries --pp-size 2; left standing it + // Every B200 Unified cell carries --pp-size 2; left standing it // multiplies against the preset's --tp-size for a world size the // preset's own --nnodes cannot satisfy. "--pp-size", "--pipeline-parallel-size", @@ -713,16 +847,24 @@ export const config = { ], }, { - // Plain TP16 avoids pipeline bubbles at the shallow 16-request point. + // NOSPEC shape. PP2 × TP8 keeps every TP collective inside one node and + // halves the layer-local KV/state bill per GPU; the cross-node hop is a + // pipeline P2P the next microbatch hides, not an all-reduce to wait on. + // With DSPARK the Spec Decode overlay rewrites this to flat TP16 — + // speculation requires pp_size == 1. match: { hw: "b200", pdMode: "unified", strategy: "low-latency" }, nnodes: 2, + // Under a pp_size == 1 speculative algorithm, re-lay this recipe flat at + // constant world size instead of making speculation unavailable. + specCollapsePp: true, verified: false, verificationStatus: "in-progress", env: [], flags: [ "--trust-remote-code", "--model-path {{MODEL_NAME}}", - "--tp-size 16", + "--tp-size 8", + "--pp-size 2", "--mem-fraction-static 0.85", "--disable-flashinfer-autotune", "--watchdog-timeout 3600", @@ -734,18 +876,34 @@ export const config = { ], }, { - // TP16 + DCP16 on two B200 nodes. + // PP2 × DCPEP8: DCP8 deduplicates the TP-replicated MLA KV within each + // pipeline stage, EP8 shards the 896 experts across the same 8 ranks. + // EP here is plain expert sharding (a2a backend stays `none`), so unlike + // DeepEP/MegaMoE it allocates no dispatch buffers to reclaim the KV DCP + // just bought. DSPARK rewrites the pair to TP16 + DCP16 + EP16. match: { hw: "b200", pdMode: "unified", strategy: "balanced" }, nnodes: 2, + // Under a pp_size == 1 speculative algorithm, re-lay this recipe flat at + // constant world size instead of making speculation unavailable. + specCollapsePp: true, verified: false, verificationStatus: "in-progress", env: [], flags: [ "--trust-remote-code", "--model-path {{MODEL_NAME}}", - "--tp-size 16", - "--dcp-size 16", + "--tp-size 8", + "--pp-size 2", + "--dcp-size 8", + "--ep-size 8", + // Both pinned to the brought-up shape rather than left to the auto + // resolution the rest of Blackwell uses. The MXFP4 runner needs the SiTU + // cubin pool the published image ships; drop it to get the Marlin + // fallback on an install without one. + "--moe-runner-backend flashinfer_mxfp4", + "--decode-attention-backend cutedsl_mla", "--mem-fraction-static 0.85", + "--chunked-prefill-size 8192", "--disable-flashinfer-autotune", "--watchdog-timeout 3600", "--reasoning-parser kimi_k3", @@ -757,45 +915,28 @@ export const config = { }, { // Balanced baseline; High-Throughput routes to the large-scale presets. + // Still redirects: the 16-GPU cell is the floor of the large-scale lane. match: { hw: "b200", pdMode: "unified", strategy: "high-throughput" }, nnodes: 2, + // Under a pp_size == 1 speculative algorithm, re-lay this recipe flat at + // constant world size instead of making speculation unavailable. + specCollapsePp: true, verified: false, verificationStatus: "in-progress", redirect: true, warn: "High-Throughput is the large-scale lane: pick a Cluster Size and a Large-Scale Preset in the [Playground](#playground) to compose the DP x EP command on top of this hardware's Balanced recipe.", env: [], - flags: [ - "--trust-remote-code", - "--model-path {{MODEL_NAME}}", - "--tp-size 16", - "--dcp-size 16", - "--mem-fraction-static 0.85", - "--disable-flashinfer-autotune", - "--watchdog-timeout 3600", - "--reasoning-parser kimi_k3", - "--tool-call-parser kimi_k3", - "--model-loader-extra-config '{\"enable_multithread_load\": true}'", - "--host {{HOST_IP}}", - "--port {{PORT}}", - ], - }, - { - // Reference long-context launch: PP2 halves the layer-local KV/state - // footprint per GPU while TP8 spans each 8-GPU pipeline stage. - match: { hw: "b200", pdMode: "unified", strategy: "long-context" }, - nnodes: 2, - verified: false, - verificationStatus: "in-progress", - env: [], flags: [ "--trust-remote-code", "--model-path {{MODEL_NAME}}", "--tp-size 8", "--pp-size 2", + "--dcp-size 8", + "--ep-size 8", + "--moe-runner-backend flashinfer_mxfp4", + "--decode-attention-backend cutedsl_mla", "--mem-fraction-static 0.85", - "--context-length 131072", "--chunked-prefill-size 8192", - "--mamba-radix-cache-strategy extra_buffer", "--disable-flashinfer-autotune", "--watchdog-timeout 3600", "--reasoning-parser kimi_k3", @@ -1171,11 +1312,14 @@ export const config = { ], }, - // ----- Prefill role: chunked, on the TP8 platforms. The prefill role keeps - // radix caching, so the Unified 5-slots-per-request state cost still holds - // (pool split rides the calculator-driven ratio). Default is TP8; - // Long-Context is one pipeline stage per GPU, which turns the parallelism - // comm from something you wait for into something the next microbatch hides. + // ----- Prefill role: chunked. The prefill role keeps radix caching, so the + // Unified 5-slots-per-request state cost still holds (pool split rides the + // calculator-driven ratio). Two shapes, split by cell width: the 8-GPU + // platforms (B300 1×8, GB300 2×4) run Default as TP8 and reserve deep PP for + // Long-Context, while the 16-GPU platforms (B200 2×8, GB200 4×4) run PP16 × + // TP1 on both and differ only in --mem-fraction-static. Deep PP is one + // pipeline stage per GPU, which turns the parallelism comm from something you + // wait for into something the next microbatch hides. // Both roles must agree on --page-size and --kv-cache-dtype (the transfer // sanity-checks them at connect), so neither is pinned here. ----- { @@ -1354,6 +1498,81 @@ export const config = { ], }, + { + // B200 2×8 has the same 16 ranks as GB200 4×4, so the prefill role takes + // the same PP16 × TP1 shape — one pipeline stage per GPU. The pipeline is + // what makes this portable off MNNVL: the only cross-node traffic is a + // pipeline P2P handoff at the node boundary (one stage boundary out of 15, + // activations not weights), which the next microbatch hides, where a + // TP16 or TEP16 prefill would put an all-reduce on the same link and wait + // on it. Same caveat as GB200: below concurrency ~8 the pipeline cannot + // fill and `--tp-size 16 --ep-size 16` leads instead. No --enable-symm-mem + // and no DCP — at TP1 there is no TP collective to accelerate and no + // TP-replicated KV to shard. Pairs with all three B200 decode cells: SGLang + // requires `decode pp_size == prefill pp_size or 1`, and every B200 decode + // cell is flat TP16 (pp=1), so none of them carry the GB200 Low-Latency + // cell's PP2 constraint that forces a PP2 × TP8 prefill instead. + match: { hw: "b200", pdMode: "prefill", strategy: "default" }, + nnodes: 2, + verified: false, + verificationStatus: "in-progress", + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 1", + "--pp-size 16", + "--mem-fraction-static 0.85", + "--chunked-prefill-size 16384", + "--max-prefill-tokens 16384", + "--disable-flashinfer-autotune", + "--weight-loader-prefetch-checkpoints", + "--watchdog-timeout 3600", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + // Explicit multithread_load is also what keeps it on: prefetch otherwise + // forces the single-threaded loader to avoid I/O oversubscription. + "--model-loader-extra-config '{\"enable_multithread_load\": true}'", + "--disaggregation-mode prefill", + "--disaggregation-transfer-backend nixl", + "--disaggregation-bootstrap-port 8998", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + // Same PP16 × TP1 shape as the Default cell with mem-fraction raised to + // 0.90 for KV headroom, exactly the GB200 pair's trade. TP1 is what buys + // the context length: with TP > 1 the MLA KV is replicated across the TP + // ranks, so TP2 × PP8 would hold roughly half the tokens for the same + // memory. Not yet benchmarked on long-context workloads. + match: { hw: "b200", pdMode: "prefill", strategy: "long-context" }, + nnodes: 2, + verified: false, + verificationStatus: "in-progress", + env: [], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp-size 1", + "--pp-size 16", + "--mem-fraction-static 0.90", + "--chunked-prefill-size 16384", + "--max-prefill-tokens 16384", + "--disable-flashinfer-autotune", + "--weight-loader-prefetch-checkpoints", + "--watchdog-timeout 3600", + "--reasoning-parser kimi_k3", + "--tool-call-parser kimi_k3", + "--model-loader-extra-config '{\"enable_multithread_load\": true}'", + "--disaggregation-mode prefill", + "--disaggregation-transfer-backend nixl", + "--disaggregation-bootstrap-port 8998", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + // ----- Decode role: the unified cell for the same hw and strategy, plus // the PD role and transport flags, and re-sized KDA state. // @@ -1903,7 +2122,10 @@ export const config = { // Cross-node fabric env (substitute the NIC used by every rank). multiNodeHints: { b200: [ - "Low-Latency, Balanced, and High-Throughput use TP16 across both nodes; Long-Context uses TP8 within each PP2 stage.", + // One hint list per hw, shared by every cell, so it has to name the shape + // per role rather than assume the Unified one. + "Unified with Spec Decode off runs TP8 within a node and PP2 across the two (+DCP8/EP8 on Balanced and High-Throughput); with DSPARK (pp_size == 1) it runs TP16 across both nodes instead.", + "Prefill is TP1 × PP16 — one pipeline stage per GPU, non-speculative only. Decode is flat TP16 (+DCP16 on Balanced and High-Throughput).", "Multi-node K3 needs the cross-node NIC pinned on BOTH ranks:", " GLOO_SOCKET_IFNAME= # bootstrap interface", " NCCL_SOCKET_IFNAME= # force NCCL off kube-ipvs0",