feat(cookbook): add DGX Spark support for Inkling-Small (#33131)
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
yvbbrjdr
co-authored by
Zijie Xia
Claude Opus 5
Zijie Xia
Claude Opus 5
parent
3e0f7c3f30
commit
e4c4faf8a2
@@ -109,6 +109,7 @@ this table (RTX PRO 6000, GH200, future chips) goes in the model's own `config.h
|
||||
| B300 | NVIDIA | 288GB | `lmsysorg/sglang:<ver>` (or `-cu130` when required) |
|
||||
| GB200 | NVIDIA | 192GB | `lmsysorg/sglang:<ver>` (or `-cu130`) |
|
||||
| GB300 | NVIDIA | 288GB | `lmsysorg/sglang:<ver>` (or `-cu130`) |
|
||||
| DGX Spark | NVIDIA | 128GB (unified) | `lmsysorg/sglang:<ver>` — needs a **CUDA 13** build |
|
||||
| MI300X | AMD | 192GB | `lmsysorg/sglang:<ver>-rocm720-mi30x` |
|
||||
| MI325X | AMD | 256GB | `lmsysorg/sglang:<ver>-rocm720-mi30x` |
|
||||
| MI350X | AMD | 288GB | `lmsysorg/sglang:<ver>-rocm720-mi35x` |
|
||||
@@ -119,8 +120,9 @@ this table (RTX PRO 6000, GH200, future chips) goes in the model's own `config.h
|
||||
`sglang_version`; the engine falls back to `lmsysorg/sglang:dev` for any unmapped hw.
|
||||
- **TP sizing** (sanity-check recipes): `weight_GB / gpu_mem`, round up to a power of 2,
|
||||
~20–30% headroom. BF16 ≈ params×2 GB, FP8 ≈ ×1, FP4 ≈ ×0.5. MoE → **total** weight, not
|
||||
active params. FP4 is Blackwell-only (B200/B300/GB200/GB300). GB200/GB300 single-node
|
||||
hosts are typically **4 GPUs** (TP=4 ceiling).
|
||||
active params. FP4 is Blackwell-only (B200/B300/GB200/GB300/DGX Spark). GB200/GB300
|
||||
single-node hosts are typically **4 GPUs** (TP=4 ceiling); a DGX Spark is **1 GPU**, so
|
||||
its only multi-GPU topology is TP=2 across 2 nodes.
|
||||
- **Platform flags**: Blackwell may need `--attention-backend trtllm_mha`; AMD typically
|
||||
needs `--attention-backend triton` + env `SGLANG_USE_AITER=1` /
|
||||
`SGLANG_ROCM_FUSED_DECODE_MLA=0` (check AITER TP constraints, e.g. `heads_per_gpu % 16 == 0`).
|
||||
|
||||
@@ -29,7 +29,7 @@ the full contract):
|
||||
|---|---|---|
|
||||
| `modelName` | string | Display label only. Not used for HF slug — see `modelNames`. |
|
||||
| `supportedHardware` | `string[]` | Which hw ids appear in the catalog. Subset of `HARDWARE_CATALOG` (in `_deployment.jsx`) ∪ `config.hardware`. Listing an id makes its button appear; if no cell uses it, the engine greys it out automatically. |
|
||||
| `hardware` | `{id,label,vram,vendor}[]` | Optional. GPUs the shared `HARDWARE_CATALOG` doesn't carry (workstation / desktop / future chips, e.g. RTX PRO 6000). The engine merges these into the catalog, so a model-specific GPU is config data — **no engine-catalog edit**. Also add the id to `supportedHardware`. |
|
||||
| `hardware` | `{id,label,vram,vendor}[]` | Optional. GPUs the shared `HARDWARE_CATALOG` doesn't carry (workstation / desktop / future chips, e.g. RTX PRO 6000). The engine merges these into the catalog, so a model-specific GPU is config data — **no engine-catalog edit**. Also add the id to `supportedHardware`. A catalog entry (shared or per-model) may add `multiNodeDockerFlags: string[]` — `docker run` flags the platform's multi-node fabric needs, emitted into the Docker command for multi-node cells only (e.g. DGX Spark's ConnectX-7 RDMA: `--ulimit memlock=-1:-1 --cap-add IPC_LOCK --device /dev/infiniband`). Platform-invariant, so it lives on the hardware entry, not in each model's config. |
|
||||
| `variants` | `{id, label, subtitle?}[]` | 2nd-dim option list. Use `default` / single-element if the model has no variant axis. |
|
||||
| `quantizations` | `{id, label}[]` | 3rd-dim option list. |
|
||||
| `strategies` | `{id, label}[]` | 4th-dim option list. Canonical ids: `low-latency` / `balanced` / `high-throughput` (never model-specific ids like `mtp`). **The count follows the page's operating points**: one recipe → a single `balanced`; two → `low-latency` + `high-throughput`; three → the full trio (the ideal). Tiers apply per (hw × variant × quant) combination — a single-recipe combination parks under its semantically honest tier (clear slant → that tier, e.g. DSv4's RTX 6000 → `low-latency`; no slant → `balanced`, e.g. Qwen3.5's Xeon); the page's list is the union and the engine greys unused chips per selection. Never invent a recipe just to fill chips. When two recipes differ by MTP / speculative decoding, the assignment is deterministic: spec ON → `low-latency`, spec OFF → `high-throughput` (at saturation the draft+verify overhead outweighs the speedup — same reason DSv4's high-throughput recipes disable MTP). The recurring markers in the other direction: dp-attention ON (MLA-attention models) and EP / DP+EP ON (MoE models) → `high-throughput`. |
|
||||
@@ -43,7 +43,7 @@ the full contract):
|
||||
|
||||
| Field | Type | Purpose |
|
||||
|---|---|---|
|
||||
| `multiNodeHints` | `{[hwId]: string[]}` | Lines prepended as `# ...` comments to multi-node commands (env-var hints). Per-hw, and only for hw whose **cluster fabric needs manual NIC config** (e.g. `gb200` NVL72/MNNVL → NVSHMEM/Gloo hints). NOT every multi-N hw needs an entry — standard-IB DeepEP (h200) auto-detects the HCA, and Marlin multi-node (h100) uses no DeepEP/NVSHMEM at all. |
|
||||
| `multiNodeHints` | `{[hwId]: string[]}` | Lines prepended as `# ...` comments to multi-node commands (env-var hints). Per-hw, and only for hw whose **cluster fabric needs manual NIC config** (e.g. `gb200` NVL72/MNNVL → NVSHMEM/Gloo hints). NOT every multi-N hw needs an entry — standard-IB DeepEP (h200) auto-detects the HCA, and Marlin multi-node (h100) uses no DeepEP/NVSHMEM at all. Hints render above **both** run modes, so keep them mode-agnostic: `docker run` flags belong in the hardware entry's `multiNodeDockerFlags` (above), which the engine puts in the command itself. |
|
||||
| `dockerImages` | `{[key]: string}` | Image for `docker run` framing, keyed by `hw\|quant` (most specific) then `hw`. Use a `hw\|quant` key only when one quant on a shared GPU needs a different image (e.g. an NVFP4 dev build on b300/gb300 while FP8/BF16 stay on the release image); otherwise key by plain `hw`. **Ask the user which sglang build the recipes ran on; don't guess a supporting release.** Falls back to `lmsysorg/sglang:dev` if missing — also the sensible default when unsure. |
|
||||
| `playgroundFeatures` | `{[axisId]: {...}}` | Opts into the Playground widget. See §2.3. |
|
||||
| `benchmarkCommands` | `{speed: string, accuracy: {[accKey]: string \| {[variant]: string}}, numPromptsByConc?: {[c]: number}}` | Powers the benchmark card's **"⚡ Reproduce"** modal. `speed` is ONE `bench_serving` template; the engine fills `{{DATASET}}`/`{{ISL}}`/`{{OSL}}` from each cell's `speed[].workload`, the chip-picked `{{MAX_CONCURRENCY}}`, and `{{NUM_PROMPTS}}` (resolved `workload.num_prompts ?? numPromptsByConc[c] ?? max(c*2, 200)`). `accuracy` maps an accuracy field (e.g. `gsm8k_pct`) to a per-eval template — a string, OR a `{flash, pro, …}` object keyed by variant when the command differs per variant (e.g. GPQA/AIME `--max-tokens`). The modal renders a chip per eval (one command area, like Speed). Both also use `{{MODEL_NAME}}` + `{{CURL_HOST}}`/`{{CURL_PORT}}` like `curl`. `speed` should carry `--flush-cache` (bench_serving's `random` prompts are deterministic — warm reruns hit the radix cache and inflate throughput; measure cache-cold). Optional; the button only appears when this AND `benchmarks` are present. |
|
||||
|
||||
@@ -37,7 +37,8 @@ Caveats discovered in the pilot:
|
||||
`--expert-parallel-size N` to `--ep N` (alias, see table above) so the knob
|
||||
can recognize/strip it.
|
||||
- `multiNodeHints` only for hw whose fabric needs manual NIC env (gb200-class);
|
||||
standard-IB H100 multi-node needs none.
|
||||
standard-IB H100 multi-node needs none. Hints show on both run modes, so
|
||||
`docker run` flags go in the hardware entry's `multiNodeDockerFlags` instead.
|
||||
- `dockerImages`: only the tags the legacy page pinned. CPU/Xeon stays unmapped
|
||||
(`:dev` fallback) with a "install from source" tip.
|
||||
- **Accuracy-degrading flags** (`--kv-cache-dtype fp8_e4m3`, W4A4-style
|
||||
|
||||
@@ -45,7 +45,10 @@ than restating.
|
||||
- No `!(x in y)` anywhere (Mintlify AST walker crashes) — use `obj.key === undefined`.
|
||||
- `supportedHardware` ⊆ `HARDWARE_CATALOG` (in `_deployment.jsx`) ∪ `config.hardware`. A
|
||||
model-specific GPU the shared catalog lacks must be declared in `config.hardware`
|
||||
(`{id,label,vram,vendor}`), **not** added to the engine catalog.
|
||||
(`{id,label,vram,vendor}`), **not** added to the engine catalog — unless the platform is
|
||||
genuinely shared, which is an engine change the PR body has to argue (and the
|
||||
`cookbook-add-model` hardware table moves with it). A new catalog entry is inert for
|
||||
other pages: each filters the catalog by its own `supportedHardware`.
|
||||
- `placeholders` declares every `{{KEY}}` used in `curl` or any cell.
|
||||
- `modelNames` covers every cell (by `hw|variant|quant` triple or `variant|quant` pair).
|
||||
- `strategies` count matches the page's operating points — 1 recipe → a single `balanced`;
|
||||
@@ -66,6 +69,9 @@ than restating.
|
||||
a different image (e.g. an FP4 dev build) — don't flag those.
|
||||
- `multiNodeHints` present ONLY for hw whose fabric needs manual NIC env (e.g. `gb200`
|
||||
NVL72) — NOT every `multi-N` hw (standard-IB DeepEP / Marlin multi-node don't need it).
|
||||
Hints render above BOTH run modes, so flag one that reads as docker-only ("add these
|
||||
docker run flags") — `docker run` flags belong in the hardware entry's
|
||||
`multiNodeDockerFlags`, which the engine emits into the Docker command itself.
|
||||
- `github.cookbookModel` is set to the model's HF id (`<hf-org>/<model-slug>`). The issue
|
||||
template's `model` field is a free-form input prefilled from this value; if the config
|
||||
omits the `github` block, the engine falls back to `deepseek-ai/deepseek-v4` and the
|
||||
|
||||
@@ -31,11 +31,12 @@ Then run the **Python** output of the command panel below.
|
||||
|
||||
<Note>The Inkling-Small images are being published to [`lmsysorg/sglang`](https://hub.docker.com/r/lmsysorg/sglang/tags) — watch the tag list for status.</Note>
|
||||
|
||||
There are two multi-arch (amd64 / arm64) CUDA builds plus a ROCm build; pick the CUDA build by your CUDA version, not your GPU:
|
||||
There are two multi-arch (amd64 / arm64) CUDA builds plus a ROCm build; pick the CUDA build by your CUDA version, not your GPU. DGX Spark (GB10) uses a dedicated arm64 CUDA 13 image:
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:dev-inkling-dspark # CUDA 13
|
||||
docker pull lmsysorg/sglang:dev-cu12-inkling-dspark # CUDA 12
|
||||
docker pull lmsysorg/sglang:dev-inkling-small-dgx-spark # DGX Spark (GB10 / SM121)
|
||||
docker pull lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark # AMD MI350X / MI355X
|
||||
```
|
||||
|
||||
@@ -85,6 +86,8 @@ import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
**Multimodal.** The recipes pass `--enable-multimodal` so the server accepts image and audio inputs alongside text — drop it for text-only serving.
|
||||
|
||||
**DGX Spark (2× GB10).** The verified cell runs NVFP4 with TP=2 across two Sparks over ConnectX-7 (1 GPU per node). Use the `dev-inkling-small-dgx-spark` image, Triton attention + Marlin FP4/MoE, and `--disable-prefill-cuda-graph`. The Docker command already carries the ConnectX-7 flags `--ulimit memlock=-1:-1 --cap-add IPC_LOCK --device /dev/infiniband`.
|
||||
|
||||
**Memory pool ratios.** `--swa-full-tokens-ratio` and `--mamba-full-memory-ratio` (both default `0.1`) size the SWA and Mamba/sconv state pools; tune them to your workload's usage.
|
||||
|
||||
**MTP needs `--enable-multi-layer-eagle`.** The MTP recipe drives Inkling-Small's multi-layer draft head; without this flag the standard EAGLE worker runs against it and outputs garbage.
|
||||
|
||||
@@ -9,7 +9,9 @@
|
||||
// hardware optional — per-model GPUs the shared HARDWARE_CATALOG lacks:
|
||||
// {id, label, vram, vendor}[] merged into the catalog at render
|
||||
// (so a model-specific GPU never needs an engine-catalog edit);
|
||||
// vendor picks the selector group: blackwell | hopper | amd
|
||||
// vendor picks the selector group: blackwell | hopper | amd.
|
||||
// `multiNodeDockerFlags: string[]` (either source) adds
|
||||
// `docker run` flags the platform's fabric needs
|
||||
// variants/quantizations/strategies/nodesOptions LEGACY 4-dim option lists,
|
||||
// used when `matchDims` is absent (nodesOptions id is
|
||||
// `single` or `multi-N` → --nnodes N)
|
||||
@@ -79,6 +81,12 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
{ id: "gb300", label: "GB300", vram: "288GB" },
|
||||
{ id: "b200", label: "B200", vram: "192GB" },
|
||||
{ id: "gb200", label: "GB200", vram: "192GB" },
|
||||
// GB10 Grace Blackwell — 128 GB coherent unified system memory (not discrete VRAM).
|
||||
// Multi-node runs over ConnectX-7 RDMA (pinned memory + IB passthrough).
|
||||
{ id: "dgx-spark", label: "DGX Spark", vram: "128GB",
|
||||
multiNodeDockerFlags: [
|
||||
"--ulimit memlock=-1:-1", "--cap-add IPC_LOCK", "--device /dev/infiniband",
|
||||
] },
|
||||
],
|
||||
hopper: [
|
||||
{ id: "h200", label: "H200", vram: "141GB" },
|
||||
@@ -690,6 +698,16 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
const extra = (config.hardware || []).find((h) => h.id === hwId);
|
||||
return (extra && extra.vendor) || "nvidia";
|
||||
};
|
||||
// `config.hardware` overrides by id, as in buildHardwareGroups.
|
||||
const fabricFlagsOf = (hwId) => {
|
||||
const extra = (config.hardware || []).find((h) => h.id === hwId);
|
||||
if (extra) return extra.multiNodeDockerFlags || [];
|
||||
for (const list of Object.values(HARDWARE_CATALOG)) {
|
||||
const hit = list.find((h) => h.id === hwId);
|
||||
if (hit) return hit.multiNodeDockerFlags || [];
|
||||
}
|
||||
return [];
|
||||
};
|
||||
const gpuAccessLines = vendorOf(sel.hw) === "amd"
|
||||
? [
|
||||
"docker run",
|
||||
@@ -708,6 +726,7 @@ export const Deployment = ({ config, benchmarks }) => {
|
||||
// (--dist-init-addr) and NCCL/GLOO traffic are reachable; single-node
|
||||
// just maps the serve port.
|
||||
multinode ? " --network host" : ` -p ${servePort}:${servePort}`,
|
||||
...(multinode ? fabricFlagsOf(sel.hw).map((f) => " " + f) : []),
|
||||
" -v ~/.cache/huggingface:/root/.cache/huggingface",
|
||||
// HF token only for gated checkpoints — configs that declare an HF_TOKEN placeholder.
|
||||
...(config.placeholders && config.placeholders.HF_TOKEN
|
||||
|
||||
@@ -1392,10 +1392,19 @@ export const Playground = ({ config }) => {
|
||||
|| di[`${sel.hw}|${sel.quant}`] || di[sel.hw] || "lmsysorg/sglang:dev";
|
||||
const portFlag = f.find((x) => x.split(/[\s=]/)[0] === "--port");
|
||||
const servePort = portFlag ? portFlag.slice("--port".length).trim() : "{{PORT}}";
|
||||
// Mirrors `multiNodeDockerFlags` on the _deployment.jsx HARDWARE_CATALOG
|
||||
// (Mintlify strips module state, so the engines cannot share it).
|
||||
const HW_MULTINODE_DOCKER_FLAGS = {
|
||||
"dgx-spark": [
|
||||
"--ulimit memlock=-1:-1", "--cap-add IPC_LOCK", "--device /dev/infiniband",
|
||||
],
|
||||
};
|
||||
const fabricFlags = HW_MULTINODE_DOCKER_FLAGS[sel.hw] || [];
|
||||
const dockerLines = [
|
||||
"docker run --gpus all",
|
||||
" --shm-size 32g",
|
||||
(multinode || pdMode) ? " --network host" : ` -p ${servePort}:${servePort}`,
|
||||
...(multinode ? fabricFlags.map((x) => " " + x) : []),
|
||||
" -v ~/.cache/huggingface:/root/.cache/huggingface",
|
||||
` --env "HF_TOKEN={{HF_TOKEN}}"`,
|
||||
...cellEnv.map((e) => ` --env ${e}`),
|
||||
|
||||
@@ -15,6 +15,8 @@ export const benchmarks = [
|
||||
{ match: { hw: "h200" , variant: "default" , quant: "nvfp4" , strategy: "balanced" , nodes: "single" },
|
||||
sglang_version: "dev-inkling-dspark (b7252cc)",
|
||||
accuracy: { aime26_pct: 95.00, bfcl_pct: 76.02, mmau_pct: 74.70 } },
|
||||
{ match: { hw: "dgx-spark", variant: "default" , quant: "nvfp4" , strategy: "balanced" , nodes: "multi-2" },
|
||||
sglang_version: "dev-inkling-small-dgx-spark" },
|
||||
{ match: { hw: "mi350x" , variant: "default" , quant: "bf16" , strategy: "balanced" , nodes: "single" } },
|
||||
{ match: { hw: "mi355x" , variant: "default" , quant: "bf16" , strategy: "balanced" , nodes: "single" } },
|
||||
{ match: { hw: "b200" , variant: "default" , quant: "nvfp4" , strategy: "mtp" , nodes: "single" },
|
||||
|
||||
@@ -9,6 +9,7 @@ export const config = {
|
||||
// Platform list inherited from the Inkling recipes (same architecture family).
|
||||
supportedHardware: [
|
||||
"h200", "b200", "b300", "gb200", "gb300",
|
||||
"dgx-spark",
|
||||
"mi350x", "mi355x",
|
||||
],
|
||||
|
||||
@@ -71,13 +72,15 @@ export const config = {
|
||||
// NVIDIA: two multi-arch CUDA builds (dev-inkling-dspark for CUDA 13,
|
||||
// dev-cu12-inkling-dspark for CUDA 12) — pick by your CUDA version, not by GPU.
|
||||
// Panel defaults to cu13. AMD: dev-rocm720-mi35x-inkling-dspark (sglang-rocm repo).
|
||||
// All tiers ship from the same images, DSpark included.
|
||||
// DGX Spark uses a dedicated arm64 CUDA 13 image with NCCL 2.30.7.
|
||||
// All tiers ship from the same images, DSpark included (except DGX Spark).
|
||||
dockerImages: {
|
||||
h200: "lmsysorg/sglang:dev-inkling-dspark",
|
||||
b200: "lmsysorg/sglang:dev-inkling-dspark",
|
||||
b300: "lmsysorg/sglang:dev-inkling-dspark",
|
||||
gb200: "lmsysorg/sglang:dev-inkling-dspark",
|
||||
gb300: "lmsysorg/sglang:dev-inkling-dspark",
|
||||
"dgx-spark": "lmsysorg/sglang:dev-inkling-small-dgx-spark",
|
||||
mi350x: "lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark",
|
||||
mi355x: "lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark",
|
||||
},
|
||||
@@ -89,12 +92,16 @@ export const config = {
|
||||
playgroundFeatures: {
|
||||
|
||||
// ----- Card: "Attention Parallelism" -----
|
||||
// TP only. Inkling-Small needs TP=8 to hold the 1M-token SWA + Mamba/sconv pools
|
||||
// (TP=4 can't fit — see §2). TP=16 is cross-node (multi-node path).
|
||||
// TP only. Datacenter NVFP4 recipes need TP=8 (or TP=4 on GB200/GB300) to hold
|
||||
// the 1M-token SWA + Mamba/sconv pools. DGX Spark uses TP=2 across 2 nodes
|
||||
// (1 GPU each). TP=16 is the cross-node datacenter path.
|
||||
attention: {
|
||||
knobs: [
|
||||
{ id: "tp", label: "TP", values: [
|
||||
null, 4, 8,
|
||||
null,
|
||||
{ value: 2, disable: { hw: ["h200", "b200", "b300", "gb200", "gb300", "mi350x", "mi355x"] },
|
||||
disableReason: "TP=2 is the DGX Spark multi-node recipe (1 GPU × 2 nodes)." },
|
||||
4, 8,
|
||||
{ value: 16, disable: { nodes: ["single"] },
|
||||
disableReason: "TP=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." },
|
||||
]},
|
||||
@@ -103,7 +110,7 @@ export const config = {
|
||||
|
||||
// ----- Card: "MoE Parallelism" -----
|
||||
// Blackwell (SM100) runs the FlashInfer TRT-LLM routed FP4 experts; Hopper (SM90)
|
||||
// has no FP4 runner and falls back to Marlin W4A16.
|
||||
// and DGX Spark (SM121) fall back to Marlin W4A16.
|
||||
moe: {
|
||||
backend: {
|
||||
options: [
|
||||
@@ -112,16 +119,16 @@ export const config = {
|
||||
{ id: "flashinfer_trtllm_routed", label: "FlashInfer TRT-LLM (routed FP4)",
|
||||
flags: ["--moe-runner-backend flashinfer_trtllm_routed"],
|
||||
requiresHw: ["b200", "b300", "gb200", "gb300"],
|
||||
hide: { hw: ["mi350x", "mi355x"] } },
|
||||
hide: { hw: ["mi350x", "mi355x", "dgx-spark"] } },
|
||||
{ id: "marlin", label: "Marlin (W4A16)",
|
||||
flags: ["--moe-runner-backend marlin"],
|
||||
hide: { hw: ["mi350x", "mi355x"] } },
|
||||
{ id: "aiter", label: "AITER",
|
||||
flags: ["--moe-runner-backend aiter"],
|
||||
hide: { hw: ["h200", "b200", "b300", "gb200", "gb300"] } },
|
||||
hide: { hw: ["h200", "b200", "b300", "gb200", "gb300", "dgx-spark"] } },
|
||||
{ id: "triton", label: "Triton",
|
||||
flags: ["--moe-runner-backend triton"],
|
||||
hide: { hw: ["h200", "b200", "b300", "gb200", "gb300"] } },
|
||||
hide: { hw: ["h200", "b200", "b300", "gb200", "gb300", "dgx-spark"] } },
|
||||
],
|
||||
},
|
||||
},
|
||||
@@ -146,12 +153,12 @@ export const config = {
|
||||
],
|
||||
},
|
||||
|
||||
// ----- Card: "PD Disaggregation" ----- NVIDIA only; Mooncake MNNVL env gated to GB200/GB300.
|
||||
// ----- Card: "PD Disaggregation" ----- NVIDIA datacenter only; Mooncake MNNVL env gated to GB200/GB300.
|
||||
pdDisagg: {
|
||||
modes: [
|
||||
{ id: "off", label: "Off" },
|
||||
{ id: "prefill", label: "Prefill role", hide: { hw: ["mi350x", "mi355x"] } },
|
||||
{ id: "decode", label: "Decode role", hide: { hw: ["mi350x", "mi355x"] } },
|
||||
{ id: "prefill", label: "Prefill role", hide: { hw: ["mi350x", "mi355x", "dgx-spark"] } },
|
||||
{ id: "decode", label: "Decode role", hide: { hw: ["mi350x", "mi355x", "dgx-spark"] } },
|
||||
],
|
||||
transferBackends: [
|
||||
{ id: "mooncake", label: "Mooncake",
|
||||
@@ -336,6 +343,38 @@ export const config = {
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
// ====================================================================
|
||||
// NVIDIA DGX Spark (GB10 / SM121) + NVFP4 — 2× Spark over ConnectX-7.
|
||||
// 1 GPU per node → TP=2 across 2 nodes. Marlin W4A16 + Triton attention;
|
||||
// prefill CUDA graphs disabled on this platform.
|
||||
// ====================================================================
|
||||
{
|
||||
match: { hw: "dgx-spark", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "multi-2" },
|
||||
verified: true,
|
||||
env: [
|
||||
"SGLANG_ENABLE_UNIFIED_RADIX_TREE=1",
|
||||
],
|
||||
flags: [
|
||||
"--trust-remote-code",
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--tp 2",
|
||||
"--quantization modelopt_fp4",
|
||||
"--attention-backend triton",
|
||||
"--page-size 128",
|
||||
"--fp4-gemm-backend marlin",
|
||||
"--moe-runner-backend marlin",
|
||||
"--mamba-radix-cache-strategy extra_buffer",
|
||||
"--mem-fraction-static 0.85",
|
||||
"--swa-full-tokens-ratio 0.1",
|
||||
"--mamba-full-memory-ratio 0.1",
|
||||
"--enable-multimodal",
|
||||
"--disable-prefill-cuda-graph",
|
||||
"--reasoning-parser inkling",
|
||||
"--tool-call-parser inkling",
|
||||
"--host {{HOST_IP}}",
|
||||
"--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
// AMD ROCm (MI350X / MI355X) + BF16 — verified, TP=8. `--moe-runner-backend`
|
||||
// sits right after `--tp` so the Playground AITER override (re-inserted at
|
||||
// that anchor) reproduces this command exactly.
|
||||
|
||||
Reference in New Issue
Block a user