diff --git a/.claude/skills/cookbook-add-model/SKILL.md b/.claude/skills/cookbook-add-model/SKILL.md index 3c3e10571..2cc79cc76 100644 --- a/.claude/skills/cookbook-add-model/SKILL.md +++ b/.claude/skills/cookbook-add-model/SKILL.md @@ -109,6 +109,7 @@ this table (RTX PRO 6000, GH200, future chips) goes in the model's own `config.h | B300 | NVIDIA | 288GB | `lmsysorg/sglang:` (or `-cu130` when required) | | GB200 | NVIDIA | 192GB | `lmsysorg/sglang:` (or `-cu130`) | | GB300 | NVIDIA | 288GB | `lmsysorg/sglang:` (or `-cu130`) | +| DGX Spark | NVIDIA | 128GB (unified) | `lmsysorg/sglang:` — needs a **CUDA 13** build | | MI300X | AMD | 192GB | `lmsysorg/sglang:-rocm720-mi30x` | | MI325X | AMD | 256GB | `lmsysorg/sglang:-rocm720-mi30x` | | MI350X | AMD | 288GB | `lmsysorg/sglang:-rocm720-mi35x` | @@ -119,8 +120,9 @@ this table (RTX PRO 6000, GH200, future chips) goes in the model's own `config.h `sglang_version`; the engine falls back to `lmsysorg/sglang:dev` for any unmapped hw. - **TP sizing** (sanity-check recipes): `weight_GB / gpu_mem`, round up to a power of 2, ~20–30% headroom. BF16 ≈ params×2 GB, FP8 ≈ ×1, FP4 ≈ ×0.5. MoE → **total** weight, not - active params. FP4 is Blackwell-only (B200/B300/GB200/GB300). GB200/GB300 single-node - hosts are typically **4 GPUs** (TP=4 ceiling). + active params. FP4 is Blackwell-only (B200/B300/GB200/GB300/DGX Spark). GB200/GB300 + single-node hosts are typically **4 GPUs** (TP=4 ceiling); a DGX Spark is **1 GPU**, so + its only multi-GPU topology is TP=2 across 2 nodes. - **Platform flags**: Blackwell may need `--attention-backend trtllm_mha`; AMD typically needs `--attention-backend triton` + env `SGLANG_USE_AITER=1` / `SGLANG_ROCM_FUSED_DECODE_MLA=0` (check AITER TP constraints, e.g. `heads_per_gpu % 16 == 0`). diff --git a/.claude/skills/cookbook-add-model/references/authoring-reference.md b/.claude/skills/cookbook-add-model/references/authoring-reference.md index e820aabef..5e7f570e2 100644 --- a/.claude/skills/cookbook-add-model/references/authoring-reference.md +++ b/.claude/skills/cookbook-add-model/references/authoring-reference.md @@ -29,7 +29,7 @@ the full contract): |---|---|---| | `modelName` | string | Display label only. Not used for HF slug — see `modelNames`. | | `supportedHardware` | `string[]` | Which hw ids appear in the catalog. Subset of `HARDWARE_CATALOG` (in `_deployment.jsx`) ∪ `config.hardware`. Listing an id makes its button appear; if no cell uses it, the engine greys it out automatically. | -| `hardware` | `{id,label,vram,vendor}[]` | Optional. GPUs the shared `HARDWARE_CATALOG` doesn't carry (workstation / desktop / future chips, e.g. RTX PRO 6000). The engine merges these into the catalog, so a model-specific GPU is config data — **no engine-catalog edit**. Also add the id to `supportedHardware`. | +| `hardware` | `{id,label,vram,vendor}[]` | Optional. GPUs the shared `HARDWARE_CATALOG` doesn't carry (workstation / desktop / future chips, e.g. RTX PRO 6000). The engine merges these into the catalog, so a model-specific GPU is config data — **no engine-catalog edit**. Also add the id to `supportedHardware`. A catalog entry (shared or per-model) may add `multiNodeDockerFlags: string[]` — `docker run` flags the platform's multi-node fabric needs, emitted into the Docker command for multi-node cells only (e.g. DGX Spark's ConnectX-7 RDMA: `--ulimit memlock=-1:-1 --cap-add IPC_LOCK --device /dev/infiniband`). Platform-invariant, so it lives on the hardware entry, not in each model's config. | | `variants` | `{id, label, subtitle?}[]` | 2nd-dim option list. Use `default` / single-element if the model has no variant axis. | | `quantizations` | `{id, label}[]` | 3rd-dim option list. | | `strategies` | `{id, label}[]` | 4th-dim option list. Canonical ids: `low-latency` / `balanced` / `high-throughput` (never model-specific ids like `mtp`). **The count follows the page's operating points**: one recipe → a single `balanced`; two → `low-latency` + `high-throughput`; three → the full trio (the ideal). Tiers apply per (hw × variant × quant) combination — a single-recipe combination parks under its semantically honest tier (clear slant → that tier, e.g. DSv4's RTX 6000 → `low-latency`; no slant → `balanced`, e.g. Qwen3.5's Xeon); the page's list is the union and the engine greys unused chips per selection. Never invent a recipe just to fill chips. When two recipes differ by MTP / speculative decoding, the assignment is deterministic: spec ON → `low-latency`, spec OFF → `high-throughput` (at saturation the draft+verify overhead outweighs the speedup — same reason DSv4's high-throughput recipes disable MTP). The recurring markers in the other direction: dp-attention ON (MLA-attention models) and EP / DP+EP ON (MoE models) → `high-throughput`. | @@ -43,7 +43,7 @@ the full contract): | Field | Type | Purpose | |---|---|---| -| `multiNodeHints` | `{[hwId]: string[]}` | Lines prepended as `# ...` comments to multi-node commands (env-var hints). Per-hw, and only for hw whose **cluster fabric needs manual NIC config** (e.g. `gb200` NVL72/MNNVL → NVSHMEM/Gloo hints). NOT every multi-N hw needs an entry — standard-IB DeepEP (h200) auto-detects the HCA, and Marlin multi-node (h100) uses no DeepEP/NVSHMEM at all. | +| `multiNodeHints` | `{[hwId]: string[]}` | Lines prepended as `# ...` comments to multi-node commands (env-var hints). Per-hw, and only for hw whose **cluster fabric needs manual NIC config** (e.g. `gb200` NVL72/MNNVL → NVSHMEM/Gloo hints). NOT every multi-N hw needs an entry — standard-IB DeepEP (h200) auto-detects the HCA, and Marlin multi-node (h100) uses no DeepEP/NVSHMEM at all. Hints render above **both** run modes, so keep them mode-agnostic: `docker run` flags belong in the hardware entry's `multiNodeDockerFlags` (above), which the engine puts in the command itself. | | `dockerImages` | `{[key]: string}` | Image for `docker run` framing, keyed by `hw\|quant` (most specific) then `hw`. Use a `hw\|quant` key only when one quant on a shared GPU needs a different image (e.g. an NVFP4 dev build on b300/gb300 while FP8/BF16 stay on the release image); otherwise key by plain `hw`. **Ask the user which sglang build the recipes ran on; don't guess a supporting release.** Falls back to `lmsysorg/sglang:dev` if missing — also the sensible default when unsure. | | `playgroundFeatures` | `{[axisId]: {...}}` | Opts into the Playground widget. See §2.3. | | `benchmarkCommands` | `{speed: string, accuracy: {[accKey]: string \| {[variant]: string}}, numPromptsByConc?: {[c]: number}}` | Powers the benchmark card's **"⚡ Reproduce"** modal. `speed` is ONE `bench_serving` template; the engine fills `{{DATASET}}`/`{{ISL}}`/`{{OSL}}` from each cell's `speed[].workload`, the chip-picked `{{MAX_CONCURRENCY}}`, and `{{NUM_PROMPTS}}` (resolved `workload.num_prompts ?? numPromptsByConc[c] ?? max(c*2, 200)`). `accuracy` maps an accuracy field (e.g. `gsm8k_pct`) to a per-eval template — a string, OR a `{flash, pro, …}` object keyed by variant when the command differs per variant (e.g. GPQA/AIME `--max-tokens`). The modal renders a chip per eval (one command area, like Speed). Both also use `{{MODEL_NAME}}` + `{{CURL_HOST}}`/`{{CURL_PORT}}` like `curl`. `speed` should carry `--flush-cache` (bench_serving's `random` prompts are deterministic — warm reruns hit the radix cache and inflate throughput; measure cache-cold). Optional; the button only appears when this AND `benchmarks` are present. | diff --git a/.claude/skills/cookbook-migrate-model/references/dimension-mapping.md b/.claude/skills/cookbook-migrate-model/references/dimension-mapping.md index 89c9ae590..acf87415d 100644 --- a/.claude/skills/cookbook-migrate-model/references/dimension-mapping.md +++ b/.claude/skills/cookbook-migrate-model/references/dimension-mapping.md @@ -37,7 +37,8 @@ Caveats discovered in the pilot: `--expert-parallel-size N` to `--ep N` (alias, see table above) so the knob can recognize/strip it. - `multiNodeHints` only for hw whose fabric needs manual NIC env (gb200-class); - standard-IB H100 multi-node needs none. + standard-IB H100 multi-node needs none. Hints show on both run modes, so + `docker run` flags go in the hardware entry's `multiNodeDockerFlags` instead. - `dockerImages`: only the tags the legacy page pinned. CPU/Xeon stays unmapped (`:dev` fallback) with a "install from source" tip. - **Accuracy-degrading flags** (`--kv-cache-dtype fp8_e4m3`, W4A4-style diff --git a/.claude/skills/cookbook-review-pr/SKILL.md b/.claude/skills/cookbook-review-pr/SKILL.md index 1bbce320d..0b22824a3 100644 --- a/.claude/skills/cookbook-review-pr/SKILL.md +++ b/.claude/skills/cookbook-review-pr/SKILL.md @@ -45,7 +45,10 @@ than restating. - No `!(x in y)` anywhere (Mintlify AST walker crashes) — use `obj.key === undefined`. - `supportedHardware` ⊆ `HARDWARE_CATALOG` (in `_deployment.jsx`) ∪ `config.hardware`. A model-specific GPU the shared catalog lacks must be declared in `config.hardware` - (`{id,label,vram,vendor}`), **not** added to the engine catalog. + (`{id,label,vram,vendor}`), **not** added to the engine catalog — unless the platform is + genuinely shared, which is an engine change the PR body has to argue (and the + `cookbook-add-model` hardware table moves with it). A new catalog entry is inert for + other pages: each filters the catalog by its own `supportedHardware`. - `placeholders` declares every `{{KEY}}` used in `curl` or any cell. - `modelNames` covers every cell (by `hw|variant|quant` triple or `variant|quant` pair). - `strategies` count matches the page's operating points — 1 recipe → a single `balanced`; @@ -66,6 +69,9 @@ than restating. a different image (e.g. an FP4 dev build) — don't flag those. - `multiNodeHints` present ONLY for hw whose fabric needs manual NIC env (e.g. `gb200` NVL72) — NOT every `multi-N` hw (standard-IB DeepEP / Marlin multi-node don't need it). + Hints render above BOTH run modes, so flag one that reads as docker-only ("add these + docker run flags") — `docker run` flags belong in the hardware entry's + `multiNodeDockerFlags`, which the engine emits into the Docker command itself. - `github.cookbookModel` is set to the model's HF id (`/`). The issue template's `model` field is a free-form input prefilled from this value; if the config omits the `github` block, the engine falls back to `deepseek-ai/deepseek-v4` and the diff --git a/docs_new/cookbook/autoregressive/ThinkingMachines/Inkling-Small.mdx b/docs_new/cookbook/autoregressive/ThinkingMachines/Inkling-Small.mdx index ed89c3e10..63593844c 100644 --- a/docs_new/cookbook/autoregressive/ThinkingMachines/Inkling-Small.mdx +++ b/docs_new/cookbook/autoregressive/ThinkingMachines/Inkling-Small.mdx @@ -31,11 +31,12 @@ Then run the **Python** output of the command panel below. The Inkling-Small images are being published to [`lmsysorg/sglang`](https://hub.docker.com/r/lmsysorg/sglang/tags) — watch the tag list for status. -There are two multi-arch (amd64 / arm64) CUDA builds plus a ROCm build; pick the CUDA build by your CUDA version, not your GPU: +There are two multi-arch (amd64 / arm64) CUDA builds plus a ROCm build; pick the CUDA build by your CUDA version, not your GPU. DGX Spark (GB10) uses a dedicated arm64 CUDA 13 image: ```bash Command docker pull lmsysorg/sglang:dev-inkling-dspark # CUDA 13 docker pull lmsysorg/sglang:dev-cu12-inkling-dspark # CUDA 12 +docker pull lmsysorg/sglang:dev-inkling-small-dgx-spark # DGX Spark (GB10 / SM121) docker pull lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark # AMD MI350X / MI355X ``` @@ -85,6 +86,8 @@ import { Playground } from "/src/snippets/_playground.jsx"; **Multimodal.** The recipes pass `--enable-multimodal` so the server accepts image and audio inputs alongside text — drop it for text-only serving. +**DGX Spark (2× GB10).** The verified cell runs NVFP4 with TP=2 across two Sparks over ConnectX-7 (1 GPU per node). Use the `dev-inkling-small-dgx-spark` image, Triton attention + Marlin FP4/MoE, and `--disable-prefill-cuda-graph`. The Docker command already carries the ConnectX-7 flags `--ulimit memlock=-1:-1 --cap-add IPC_LOCK --device /dev/infiniband`. + **Memory pool ratios.** `--swa-full-tokens-ratio` and `--mamba-full-memory-ratio` (both default `0.1`) size the SWA and Mamba/sconv state pools; tune them to your workload's usage. **MTP needs `--enable-multi-layer-eagle`.** The MTP recipe drives Inkling-Small's multi-layer draft head; without this flag the standard EAGLE worker runs against it and outputs garbage. diff --git a/docs_new/src/snippets/_deployment.jsx b/docs_new/src/snippets/_deployment.jsx index 024c00f29..2687232c7 100644 --- a/docs_new/src/snippets/_deployment.jsx +++ b/docs_new/src/snippets/_deployment.jsx @@ -9,7 +9,9 @@ // hardware optional — per-model GPUs the shared HARDWARE_CATALOG lacks: // {id, label, vram, vendor}[] merged into the catalog at render // (so a model-specific GPU never needs an engine-catalog edit); -// vendor picks the selector group: blackwell | hopper | amd +// vendor picks the selector group: blackwell | hopper | amd. +// `multiNodeDockerFlags: string[]` (either source) adds +// `docker run` flags the platform's fabric needs // variants/quantizations/strategies/nodesOptions LEGACY 4-dim option lists, // used when `matchDims` is absent (nodesOptions id is // `single` or `multi-N` → --nnodes N) @@ -79,6 +81,12 @@ export const Deployment = ({ config, benchmarks }) => { { id: "gb300", label: "GB300", vram: "288GB" }, { id: "b200", label: "B200", vram: "192GB" }, { id: "gb200", label: "GB200", vram: "192GB" }, + // GB10 Grace Blackwell — 128 GB coherent unified system memory (not discrete VRAM). + // Multi-node runs over ConnectX-7 RDMA (pinned memory + IB passthrough). + { id: "dgx-spark", label: "DGX Spark", vram: "128GB", + multiNodeDockerFlags: [ + "--ulimit memlock=-1:-1", "--cap-add IPC_LOCK", "--device /dev/infiniband", + ] }, ], hopper: [ { id: "h200", label: "H200", vram: "141GB" }, @@ -690,6 +698,16 @@ export const Deployment = ({ config, benchmarks }) => { const extra = (config.hardware || []).find((h) => h.id === hwId); return (extra && extra.vendor) || "nvidia"; }; + // `config.hardware` overrides by id, as in buildHardwareGroups. + const fabricFlagsOf = (hwId) => { + const extra = (config.hardware || []).find((h) => h.id === hwId); + if (extra) return extra.multiNodeDockerFlags || []; + for (const list of Object.values(HARDWARE_CATALOG)) { + const hit = list.find((h) => h.id === hwId); + if (hit) return hit.multiNodeDockerFlags || []; + } + return []; + }; const gpuAccessLines = vendorOf(sel.hw) === "amd" ? [ "docker run", @@ -708,6 +726,7 @@ export const Deployment = ({ config, benchmarks }) => { // (--dist-init-addr) and NCCL/GLOO traffic are reachable; single-node // just maps the serve port. multinode ? " --network host" : ` -p ${servePort}:${servePort}`, + ...(multinode ? fabricFlagsOf(sel.hw).map((f) => " " + f) : []), " -v ~/.cache/huggingface:/root/.cache/huggingface", // HF token only for gated checkpoints — configs that declare an HF_TOKEN placeholder. ...(config.placeholders && config.placeholders.HF_TOKEN diff --git a/docs_new/src/snippets/_playground.jsx b/docs_new/src/snippets/_playground.jsx index 666a40756..2b994998a 100644 --- a/docs_new/src/snippets/_playground.jsx +++ b/docs_new/src/snippets/_playground.jsx @@ -1392,10 +1392,19 @@ export const Playground = ({ config }) => { || di[`${sel.hw}|${sel.quant}`] || di[sel.hw] || "lmsysorg/sglang:dev"; const portFlag = f.find((x) => x.split(/[\s=]/)[0] === "--port"); const servePort = portFlag ? portFlag.slice("--port".length).trim() : "{{PORT}}"; + // Mirrors `multiNodeDockerFlags` on the _deployment.jsx HARDWARE_CATALOG + // (Mintlify strips module state, so the engines cannot share it). + const HW_MULTINODE_DOCKER_FLAGS = { + "dgx-spark": [ + "--ulimit memlock=-1:-1", "--cap-add IPC_LOCK", "--device /dev/infiniband", + ], + }; + const fabricFlags = HW_MULTINODE_DOCKER_FLAGS[sel.hw] || []; const dockerLines = [ "docker run --gpus all", " --shm-size 32g", (multinode || pdMode) ? " --network host" : ` -p ${servePort}:${servePort}`, + ...(multinode ? fabricFlags.map((x) => " " + x) : []), " -v ~/.cache/huggingface:/root/.cache/huggingface", ` --env "HF_TOKEN={{HF_TOKEN}}"`, ...cellEnv.map((e) => ` --env ${e}`), diff --git a/docs_new/src/snippets/configs/thinkingmachines/inkling-small-benchmarks.jsx b/docs_new/src/snippets/configs/thinkingmachines/inkling-small-benchmarks.jsx index c78f67281..98c73fc66 100644 --- a/docs_new/src/snippets/configs/thinkingmachines/inkling-small-benchmarks.jsx +++ b/docs_new/src/snippets/configs/thinkingmachines/inkling-small-benchmarks.jsx @@ -15,6 +15,8 @@ export const benchmarks = [ { match: { hw: "h200" , variant: "default" , quant: "nvfp4" , strategy: "balanced" , nodes: "single" }, sglang_version: "dev-inkling-dspark (b7252cc)", accuracy: { aime26_pct: 95.00, bfcl_pct: 76.02, mmau_pct: 74.70 } }, + { match: { hw: "dgx-spark", variant: "default" , quant: "nvfp4" , strategy: "balanced" , nodes: "multi-2" }, + sglang_version: "dev-inkling-small-dgx-spark" }, { match: { hw: "mi350x" , variant: "default" , quant: "bf16" , strategy: "balanced" , nodes: "single" } }, { match: { hw: "mi355x" , variant: "default" , quant: "bf16" , strategy: "balanced" , nodes: "single" } }, { match: { hw: "b200" , variant: "default" , quant: "nvfp4" , strategy: "mtp" , nodes: "single" }, diff --git a/docs_new/src/snippets/configs/thinkingmachines/inkling-small.jsx b/docs_new/src/snippets/configs/thinkingmachines/inkling-small.jsx index 481be33b8..dcc57fd47 100644 --- a/docs_new/src/snippets/configs/thinkingmachines/inkling-small.jsx +++ b/docs_new/src/snippets/configs/thinkingmachines/inkling-small.jsx @@ -9,6 +9,7 @@ export const config = { // Platform list inherited from the Inkling recipes (same architecture family). supportedHardware: [ "h200", "b200", "b300", "gb200", "gb300", + "dgx-spark", "mi350x", "mi355x", ], @@ -71,13 +72,15 @@ export const config = { // NVIDIA: two multi-arch CUDA builds (dev-inkling-dspark for CUDA 13, // dev-cu12-inkling-dspark for CUDA 12) — pick by your CUDA version, not by GPU. // Panel defaults to cu13. AMD: dev-rocm720-mi35x-inkling-dspark (sglang-rocm repo). - // All tiers ship from the same images, DSpark included. + // DGX Spark uses a dedicated arm64 CUDA 13 image with NCCL 2.30.7. + // All tiers ship from the same images, DSpark included (except DGX Spark). dockerImages: { h200: "lmsysorg/sglang:dev-inkling-dspark", b200: "lmsysorg/sglang:dev-inkling-dspark", b300: "lmsysorg/sglang:dev-inkling-dspark", gb200: "lmsysorg/sglang:dev-inkling-dspark", gb300: "lmsysorg/sglang:dev-inkling-dspark", + "dgx-spark": "lmsysorg/sglang:dev-inkling-small-dgx-spark", mi350x: "lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark", mi355x: "lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark", }, @@ -89,12 +92,16 @@ export const config = { playgroundFeatures: { // ----- Card: "Attention Parallelism" ----- - // TP only. Inkling-Small needs TP=8 to hold the 1M-token SWA + Mamba/sconv pools - // (TP=4 can't fit — see §2). TP=16 is cross-node (multi-node path). + // TP only. Datacenter NVFP4 recipes need TP=8 (or TP=4 on GB200/GB300) to hold + // the 1M-token SWA + Mamba/sconv pools. DGX Spark uses TP=2 across 2 nodes + // (1 GPU each). TP=16 is the cross-node datacenter path. attention: { knobs: [ { id: "tp", label: "TP", values: [ - null, 4, 8, + null, + { value: 2, disable: { hw: ["h200", "b200", "b300", "gb200", "gb300", "mi350x", "mi355x"] }, + disableReason: "TP=2 is the DGX Spark multi-node recipe (1 GPU × 2 nodes)." }, + 4, 8, { value: 16, disable: { nodes: ["single"] }, disableReason: "TP=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." }, ]}, @@ -103,7 +110,7 @@ export const config = { // ----- Card: "MoE Parallelism" ----- // Blackwell (SM100) runs the FlashInfer TRT-LLM routed FP4 experts; Hopper (SM90) - // has no FP4 runner and falls back to Marlin W4A16. + // and DGX Spark (SM121) fall back to Marlin W4A16. moe: { backend: { options: [ @@ -112,16 +119,16 @@ export const config = { { id: "flashinfer_trtllm_routed", label: "FlashInfer TRT-LLM (routed FP4)", flags: ["--moe-runner-backend flashinfer_trtllm_routed"], requiresHw: ["b200", "b300", "gb200", "gb300"], - hide: { hw: ["mi350x", "mi355x"] } }, + hide: { hw: ["mi350x", "mi355x", "dgx-spark"] } }, { id: "marlin", label: "Marlin (W4A16)", flags: ["--moe-runner-backend marlin"], hide: { hw: ["mi350x", "mi355x"] } }, { id: "aiter", label: "AITER", flags: ["--moe-runner-backend aiter"], - hide: { hw: ["h200", "b200", "b300", "gb200", "gb300"] } }, + hide: { hw: ["h200", "b200", "b300", "gb200", "gb300", "dgx-spark"] } }, { id: "triton", label: "Triton", flags: ["--moe-runner-backend triton"], - hide: { hw: ["h200", "b200", "b300", "gb200", "gb300"] } }, + hide: { hw: ["h200", "b200", "b300", "gb200", "gb300", "dgx-spark"] } }, ], }, }, @@ -146,12 +153,12 @@ export const config = { ], }, - // ----- Card: "PD Disaggregation" ----- NVIDIA only; Mooncake MNNVL env gated to GB200/GB300. + // ----- Card: "PD Disaggregation" ----- NVIDIA datacenter only; Mooncake MNNVL env gated to GB200/GB300. pdDisagg: { modes: [ { id: "off", label: "Off" }, - { id: "prefill", label: "Prefill role", hide: { hw: ["mi350x", "mi355x"] } }, - { id: "decode", label: "Decode role", hide: { hw: ["mi350x", "mi355x"] } }, + { id: "prefill", label: "Prefill role", hide: { hw: ["mi350x", "mi355x", "dgx-spark"] } }, + { id: "decode", label: "Decode role", hide: { hw: ["mi350x", "mi355x", "dgx-spark"] } }, ], transferBackends: [ { id: "mooncake", label: "Mooncake", @@ -336,6 +343,38 @@ export const config = { "--port {{PORT}}", ], }, + // ==================================================================== + // NVIDIA DGX Spark (GB10 / SM121) + NVFP4 — 2× Spark over ConnectX-7. + // 1 GPU per node → TP=2 across 2 nodes. Marlin W4A16 + Triton attention; + // prefill CUDA graphs disabled on this platform. + // ==================================================================== + { + match: { hw: "dgx-spark", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "multi-2" }, + verified: true, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 2", + "--quantization modelopt_fp4", + "--attention-backend triton", + "--page-size 128", + "--fp4-gemm-backend marlin", + "--moe-runner-backend marlin", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.85", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--disable-prefill-cuda-graph", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, // AMD ROCm (MI350X / MI355X) + BF16 — verified, TP=8. `--moe-runner-backend` // sits right after `--tp` so the Playground AITER override (re-inserted at // that anchor) reproduces this command exactly.