feat(cookbook): add DGX Spark support for Inkling-Small (#33131)

Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
yvbbrjdr
2026-08-01 00:25:25 +00:00
committed by GitHub
co-authored by Zijie Xia Claude Opus 5
parent 3e0f7c3f30
commit e4c4faf8a2
9 changed files with 100 additions and 19 deletions
+4 -2
View File
@@ -109,6 +109,7 @@ this table (RTX PRO 6000, GH200, future chips) goes in the model's own `config.h
| B300 | NVIDIA | 288GB | `lmsysorg/sglang:<ver>` (or `-cu130` when required) |
| GB200 | NVIDIA | 192GB | `lmsysorg/sglang:<ver>` (or `-cu130`) |
| GB300 | NVIDIA | 288GB | `lmsysorg/sglang:<ver>` (or `-cu130`) |
| DGX Spark | NVIDIA | 128GB (unified) | `lmsysorg/sglang:<ver>` — needs a **CUDA 13** build |
| MI300X | AMD | 192GB | `lmsysorg/sglang:<ver>-rocm720-mi30x` |
| MI325X | AMD | 256GB | `lmsysorg/sglang:<ver>-rocm720-mi30x` |
| MI350X | AMD | 288GB | `lmsysorg/sglang:<ver>-rocm720-mi35x` |
@@ -119,8 +120,9 @@ this table (RTX PRO 6000, GH200, future chips) goes in the model's own `config.h
`sglang_version`; the engine falls back to `lmsysorg/sglang:dev` for any unmapped hw.
- **TP sizing** (sanity-check recipes): `weight_GB / gpu_mem`, round up to a power of 2,
~20–30% headroom. BF16 ≈ params×2 GB, FP8 ≈ ×1, FP4 ≈ ×0.5. MoE → **total** weight, not
active params. FP4 is Blackwell-only (B200/B300/GB200/GB300). GB200/GB300 single-node
hosts are typically **4 GPUs** (TP=4 ceiling).
active params. FP4 is Blackwell-only (B200/B300/GB200/GB300/DGX Spark). GB200/GB300
single-node hosts are typically **4 GPUs** (TP=4 ceiling); a DGX Spark is **1 GPU**, so
its only multi-GPU topology is TP=2 across 2 nodes.
- **Platform flags**: Blackwell may need `--attention-backend trtllm_mha`; AMD typically
needs `--attention-backend triton` + env `SGLANG_USE_AITER=1` /
`SGLANG_ROCM_FUSED_DECODE_MLA=0` (check AITER TP constraints, e.g. `heads_per_gpu % 16 == 0`).
@@ -29,7 +29,7 @@ the full contract):
|---|---|---|
| `modelName` | string | Display label only. Not used for HF slug — see `modelNames`. |
| `supportedHardware` | `string[]` | Which hw ids appear in the catalog. Subset of `HARDWARE_CATALOG` (in `_deployment.jsx`) ∪ `config.hardware`. Listing an id makes its button appear; if no cell uses it, the engine greys it out automatically. |
| `hardware` | `{id,label,vram,vendor}[]` | Optional. GPUs the shared `HARDWARE_CATALOG` doesn't carry (workstation / desktop / future chips, e.g. RTX PRO 6000). The engine merges these into the catalog, so a model-specific GPU is config data — **no engine-catalog edit**. Also add the id to `supportedHardware`. |
| `hardware` | `{id,label,vram,vendor}[]` | Optional. GPUs the shared `HARDWARE_CATALOG` doesn't carry (workstation / desktop / future chips, e.g. RTX PRO 6000). The engine merges these into the catalog, so a model-specific GPU is config data — **no engine-catalog edit**. Also add the id to `supportedHardware`. A catalog entry (shared or per-model) may add `multiNodeDockerFlags: string[]` — `docker run` flags the platform's multi-node fabric needs, emitted into the Docker command for multi-node cells only (e.g. DGX Spark's ConnectX-7 RDMA: `--ulimit memlock=-1:-1 --cap-add IPC_LOCK --device /dev/infiniband`). Platform-invariant, so it lives on the hardware entry, not in each model's config. |
| `variants` | `{id, label, subtitle?}[]` | 2nd-dim option list. Use `default` / single-element if the model has no variant axis. |
| `quantizations` | `{id, label}[]` | 3rd-dim option list. |
| `strategies` | `{id, label}[]` | 4th-dim option list. Canonical ids: `low-latency` / `balanced` / `high-throughput` (never model-specific ids like `mtp`). **The count follows the page's operating points**: one recipe → a single `balanced`; two → `low-latency` + `high-throughput`; three → the full trio (the ideal). Tiers apply per (hw × variant × quant) combination — a single-recipe combination parks under its semantically honest tier (clear slant → that tier, e.g. DSv4's RTX 6000 → `low-latency`; no slant → `balanced`, e.g. Qwen3.5's Xeon); the page's list is the union and the engine greys unused chips per selection. Never invent a recipe just to fill chips. When two recipes differ by MTP / speculative decoding, the assignment is deterministic: spec ON → `low-latency`, spec OFF → `high-throughput` (at saturation the draft+verify overhead outweighs the speedup — same reason DSv4's high-throughput recipes disable MTP). The recurring markers in the other direction: dp-attention ON (MLA-attention models) and EP / DP+EP ON (MoE models) → `high-throughput`. |
@@ -43,7 +43,7 @@ the full contract):
| Field | Type | Purpose |
|---|---|---|
| `multiNodeHints` | `{[hwId]: string[]}` | Lines prepended as `# ...` comments to multi-node commands (env-var hints). Per-hw, and only for hw whose **cluster fabric needs manual NIC config** (e.g. `gb200` NVL72/MNNVL → NVSHMEM/Gloo hints). NOT every multi-N hw needs an entry — standard-IB DeepEP (h200) auto-detects the HCA, and Marlin multi-node (h100) uses no DeepEP/NVSHMEM at all. |
| `multiNodeHints` | `{[hwId]: string[]}` | Lines prepended as `# ...` comments to multi-node commands (env-var hints). Per-hw, and only for hw whose **cluster fabric needs manual NIC config** (e.g. `gb200` NVL72/MNNVL → NVSHMEM/Gloo hints). NOT every multi-N hw needs an entry — standard-IB DeepEP (h200) auto-detects the HCA, and Marlin multi-node (h100) uses no DeepEP/NVSHMEM at all. Hints render above **both** run modes, so keep them mode-agnostic: `docker run` flags belong in the hardware entry's `multiNodeDockerFlags` (above), which the engine puts in the command itself. |
| `dockerImages` | `{[key]: string}` | Image for `docker run` framing, keyed by `hw\|quant` (most specific) then `hw`. Use a `hw\|quant` key only when one quant on a shared GPU needs a different image (e.g. an NVFP4 dev build on b300/gb300 while FP8/BF16 stay on the release image); otherwise key by plain `hw`. **Ask the user which sglang build the recipes ran on; don't guess a supporting release.** Falls back to `lmsysorg/sglang:dev` if missing — also the sensible default when unsure. |
| `playgroundFeatures` | `{[axisId]: {...}}` | Opts into the Playground widget. See §2.3. |
| `benchmarkCommands` | `{speed: string, accuracy: {[accKey]: string \| {[variant]: string}}, numPromptsByConc?: {[c]: number}}` | Powers the benchmark card's **"⚡ Reproduce"** modal. `speed` is ONE `bench_serving` template; the engine fills `{{DATASET}}`/`{{ISL}}`/`{{OSL}}` from each cell's `speed[].workload`, the chip-picked `{{MAX_CONCURRENCY}}`, and `{{NUM_PROMPTS}}` (resolved `workload.num_prompts ?? numPromptsByConc[c] ?? max(c*2, 200)`). `accuracy` maps an accuracy field (e.g. `gsm8k_pct`) to a per-eval template — a string, OR a `{flash, pro, …}` object keyed by variant when the command differs per variant (e.g. GPQA/AIME `--max-tokens`). The modal renders a chip per eval (one command area, like Speed). Both also use `{{MODEL_NAME}}` + `{{CURL_HOST}}`/`{{CURL_PORT}}` like `curl`. `speed` should carry `--flush-cache` (bench_serving's `random` prompts are deterministic — warm reruns hit the radix cache and inflate throughput; measure cache-cold). Optional; the button only appears when this AND `benchmarks` are present. |
@@ -37,7 +37,8 @@ Caveats discovered in the pilot:
`--expert-parallel-size N` to `--ep N` (alias, see table above) so the knob
can recognize/strip it.
- `multiNodeHints` only for hw whose fabric needs manual NIC env (gb200-class);
standard-IB H100 multi-node needs none.
standard-IB H100 multi-node needs none. Hints show on both run modes, so
`docker run` flags go in the hardware entry's `multiNodeDockerFlags` instead.
- `dockerImages`: only the tags the legacy page pinned. CPU/Xeon stays unmapped
(`:dev` fallback) with a "install from source" tip.
- **Accuracy-degrading flags** (`--kv-cache-dtype fp8_e4m3`, W4A4-style
+7 -1
View File
@@ -45,7 +45,10 @@ than restating.
- No `!(x in y)` anywhere (Mintlify AST walker crashes) — use `obj.key === undefined`.
- `supportedHardware` ⊆ `HARDWARE_CATALOG` (in `_deployment.jsx`) ∪ `config.hardware`. A
model-specific GPU the shared catalog lacks must be declared in `config.hardware`
(`{id,label,vram,vendor}`), **not** added to the engine catalog.
(`{id,label,vram,vendor}`), **not** added to the engine catalog — unless the platform is
genuinely shared, which is an engine change the PR body has to argue (and the
`cookbook-add-model` hardware table moves with it). A new catalog entry is inert for
other pages: each filters the catalog by its own `supportedHardware`.
- `placeholders` declares every `{{KEY}}` used in `curl` or any cell.
- `modelNames` covers every cell (by `hw|variant|quant` triple or `variant|quant` pair).
- `strategies` count matches the page's operating points — 1 recipe → a single `balanced`;
@@ -66,6 +69,9 @@ than restating.
a different image (e.g. an FP4 dev build) — don't flag those.
- `multiNodeHints` present ONLY for hw whose fabric needs manual NIC env (e.g. `gb200`
NVL72) — NOT every `multi-N` hw (standard-IB DeepEP / Marlin multi-node don't need it).
Hints render above BOTH run modes, so flag one that reads as docker-only ("add these
docker run flags") — `docker run` flags belong in the hardware entry's
`multiNodeDockerFlags`, which the engine emits into the Docker command itself.
- `github.cookbookModel` is set to the model's HF id (`<hf-org>/<model-slug>`). The issue
template's `model` field is a free-form input prefilled from this value; if the config
omits the `github` block, the engine falls back to `deepseek-ai/deepseek-v4` and the