minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

feat(cookbook): add DGX Spark support for Inkling-Small (#33131)

Browse Source 
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
yvbbrjdr
2026-08-01 00:25:25 +00:00
committed by GitHub
co-authored by Zijie Xia Claude Opus 5
parent 3e0f7c3f30
commit e4c4faf8a2
9 changed files with 100 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/skills/cookbook-add-model/SKILL.md
+4 -2
View File
@@ -109,6 +109,7 @@ this table (RTX PRO 6000, GH200, future chips) goes in the model's own `config.h
| B300 | NVIDIA | 288GB | `lmsysorg/sglang:<ver>` (or `-cu130` when required) |
| GB200 | NVIDIA | 192GB | `lmsysorg/sglang:<ver>` (or `-cu130`) |
| GB300 | NVIDIA | 288GB | `lmsysorg/sglang:<ver>` (or `-cu130`) |
| DGX Spark | NVIDIA | 128GB (unified) | `lmsysorg/sglang:<ver>` — needs a **CUDA 13** build |
| MI300X | AMD | 192GB | `lmsysorg/sglang:<ver>-rocm720-mi30x` |
| MI325X | AMD | 256GB | `lmsysorg/sglang:<ver>-rocm720-mi30x` |
| MI350X | AMD | 288GB | `lmsysorg/sglang:<ver>-rocm720-mi35x` |
@@ -119,8 +120,9 @@ this table (RTX PRO 6000, GH200, future chips) goes in the model's own `config.h
`sglang_version`; the engine falls back to `lmsysorg/sglang:dev` for any unmapped hw.
- **TP sizing** (sanity-check recipes): `weight_GB / gpu_mem`, round up to a power of 2,
~20–30% headroom. BF16 ≈ params×2 GB, FP8 ≈ ×1, FP4 ≈ ×0.5. MoE → **total** weight, not
active params. FP4 is Blackwell-only (B200/B300/GB200/GB300). GB200/GB300 single-node
hosts are typically **4 GPUs** (TP=4 ceiling).
active params. FP4 is Blackwell-only (B200/B300/GB200/GB300/DGX Spark). GB200/GB300
single-node hosts are typically **4 GPUs** (TP=4 ceiling); a DGX Spark is **1 GPU**, so
its only multi-GPU topology is TP=2 across 2 nodes.
- **Platform flags**: Blackwell may need `--attention-backend trtllm_mha`; AMD typically
needs `--attention-backend triton` + env `SGLANG_USE_AITER=1` /
`SGLANG_ROCM_FUSED_DECODE_MLA=0` (check AITER TP constraints, e.g. `heads_per_gpu % 16 == 0`).
.claude/skills/cookbook-add-model/references/authoring-reference.md
+2 -2
View File
@@ -29,7 +29,7 @@ the full contract):
|---|---|---|
| `modelName` | string | Display label only. Not used for HF slug — see `modelNames`. |
| `supportedHardware` | `string[]` | Which hw ids appear in the catalog. Subset of `HARDWARE_CATALOG` (in `_deployment.jsx`) ∪ `config.hardware`. Listing an id makes its button appear; if no cell uses it, the engine greys it out automatically. |
| `hardware` | `{id,label,vram,vendor}[]` | Optional. GPUs the shared `HARDWARE_CATALOG` doesn't carry (workstation / desktop / future chips, e.g. RTX PRO 6000). The engine merges these into the catalog, so a model-specific GPU is config data — **no engine-catalog edit**. Also add the id to `supportedHardware`. |
| `hardware` | `{id,label,vram,vendor}[]` | Optional. GPUs the shared `HARDWARE_CATALOG` doesn't carry (workstation / desktop / future chips, e.g. RTX PRO 6000). The engine merges these into the catalog, so a model-specific GPU is config data — **no engine-catalog edit**. Also add the id to `supportedHardware`. A catalog entry (shared or per-model) may add `multiNodeDockerFlags: string[]` — `docker run` flags the platform's multi-node fabric needs, emitted into the Docker command for multi-node cells only (e.g. DGX Spark's ConnectX-7 RDMA: `--ulimit memlock=-1:-1 --cap-add IPC_LOCK --device /dev/infiniband`). Platform-invariant, so it lives on the hardware entry, not in each model's config. |
| `variants` | `{id, label, subtitle?}[]` | 2nd-dim option list. Use `default` / single-element if the model has no variant axis. |
| `quantizations` | `{id, label}[]` | 3rd-dim option list. |
| `strategies` | `{id, label}[]` | 4th-dim option list. Canonical ids: `low-latency` / `balanced` / `high-throughput` (never model-specific ids like `mtp`). **The count follows the page's operating points**: one recipe → a single `balanced`; two → `low-latency` + `high-throughput`; three → the full trio (the ideal). Tiers apply per (hw × variant × quant) combination — a single-recipe combination parks under its semantically honest tier (clear slant → that tier, e.g. DSv4's RTX 6000 → `low-latency`; no slant → `balanced`, e.g. Qwen3.5's Xeon); the page's list is the union and the engine greys unused chips per selection. Never invent a recipe just to fill chips. When two recipes differ by MTP / speculative decoding, the assignment is deterministic: spec ON → `low-latency`, spec OFF → `high-throughput` (at saturation the draft+verify overhead outweighs the speedup — same reason DSv4's high-throughput recipes disable MTP). The recurring markers in the other direction: dp-attention ON (MLA-attention models) and EP / DP+EP ON (MoE models) → `high-throughput`. |
@@ -43,7 +43,7 @@ the full contract):
| Field | Type | Purpose |
|---|---|---|
| `multiNodeHints` | `{[hwId]: string[]}` | Lines prepended as `# ...` comments to multi-node commands (env-var hints). Per-hw, and only for hw whose **cluster fabric needs manual NIC config** (e.g. `gb200` NVL72/MNNVL → NVSHMEM/Gloo hints). NOT every multi-N hw needs an entry — standard-IB DeepEP (h200) auto-detects the HCA, and Marlin multi-node (h100) uses no DeepEP/NVSHMEM at all. |
| `multiNodeHints` | `{[hwId]: string[]}` | Lines prepended as `# ...` comments to multi-node commands (env-var hints). Per-hw, and only for hw whose **cluster fabric needs manual NIC config** (e.g. `gb200` NVL72/MNNVL → NVSHMEM/Gloo hints). NOT every multi-N hw needs an entry — standard-IB DeepEP (h200) auto-detects the HCA, and Marlin multi-node (h100) uses no DeepEP/NVSHMEM at all. Hints render above **both** run modes, so keep them mode-agnostic: `docker run` flags belong in the hardware entry's `multiNodeDockerFlags` (above), which the engine puts in the command itself. |
| `dockerImages` | `{[key]: string}` | Image for `docker run` framing, keyed by `hw\|quant` (most specific) then `hw`. Use a `hw\|quant` key only when one quant on a shared GPU needs a different image (e.g. an NVFP4 dev build on b300/gb300 while FP8/BF16 stay on the release image); otherwise key by plain `hw`. **Ask the user which sglang build the recipes ran on; don't guess a supporting release.** Falls back to `lmsysorg/sglang:dev` if missing — also the sensible default when unsure. |
| `playgroundFeatures` | `{[axisId]: {...}}` | Opts into the Playground widget. See §2.3. |
| `benchmarkCommands` | `{speed: string, accuracy: {[accKey]: string \| {[variant]: string}}, numPromptsByConc?: {[c]: number}}` | Powers the benchmark card's **"⚡ Reproduce"** modal. `speed` is ONE `bench_serving` template; the engine fills `{{DATASET}}`/`{{ISL}}`/`{{OSL}}` from each cell's `speed[].workload`, the chip-picked `{{MAX_CONCURRENCY}}`, and `{{NUM_PROMPTS}}` (resolved `workload.num_prompts ?? numPromptsByConc[c] ?? max(c*2, 200)`). `accuracy` maps an accuracy field (e.g. `gsm8k_pct`) to a per-eval template — a string, OR a `{flash, pro, …}` object keyed by variant when the command differs per variant (e.g. GPQA/AIME `--max-tokens`). The modal renders a chip per eval (one command area, like Speed). Both also use `{{MODEL_NAME}}` + `{{CURL_HOST}}`/`{{CURL_PORT}}` like `curl`. `speed` should carry `--flush-cache` (bench_serving's `random` prompts are deterministic — warm reruns hit the radix cache and inflate throughput; measure cache-cold). Optional; the button only appears when this AND `benchmarks` are present. |