diff --git a/docs_new/cookbook/autoregressive/ThinkingMachines/Inkling-Small.mdx b/docs_new/cookbook/autoregressive/ThinkingMachines/Inkling-Small.mdx new file mode 100644 index 000000000..ed89c3e10 --- /dev/null +++ b/docs_new/cookbook/autoregressive/ThinkingMachines/Inkling-Small.mdx @@ -0,0 +1,309 @@ +--- +title: Inkling-Small +description: "Deploy Inkling-Small with SGLang — launch commands, tuning, and multimodal / reasoning / tool-calling usage for Thinking Machines' Inkling-Small Mixture-of-Experts model." +tag: NEW +--- + +## Deployment + + + + + +For all install methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). + + + + + +Inkling-Small has merged to `main` but isn't in a `pip` release yet — install from source: + +```bash Command +pip install --upgrade pip +pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' +``` + +Then run the **Python** output of the command panel below. + + + + + +The Inkling-Small images are being published to [`lmsysorg/sglang`](https://hub.docker.com/r/lmsysorg/sglang/tags) — watch the tag list for status. + +There are two multi-arch (amd64 / arm64) CUDA builds plus a ROCm build; pick the CUDA build by your CUDA version, not your GPU: + +```bash Command +docker pull lmsysorg/sglang:dev-inkling-dspark # CUDA 13 +docker pull lmsysorg/sglang:dev-cu12-inkling-dspark # CUDA 12 +docker pull lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark # AMD MI350X / MI355X +``` + +For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. + + + + + + + +Pick your hardware to generate the launch command. Each platform ships a **Balanced** recipe plus **MTP** and **DSpark** (speculative decoding) tiers and a **Long Context (MXFP8 KV)** tier where validated; the **LoRA** variant serves adapters on top of the frozen base model. Set `MAX_LORAS` to the number of distinct adapters you serve (1 is fastest for single-adapter serving). + +import { Deployment } from "/src/snippets/_deployment.jsx"; +import { config } from "/src/snippets/configs/thinkingmachines/inkling-small.jsx"; +import { benchmarks } from "/src/snippets/configs/thinkingmachines/inkling-small-benchmarks.jsx"; + + + +
+

Panel controls (top of the command box):

+
    +
  • ⧉ Copy — copies the current command to your clipboard.
    • +
  • $ cURL — a sample request against localhost:30000 to confirm the server is up.
    • +
  • ⚙ Env — edits the placeholders (HOST_IP, PORT, NODE_RANK, NODE0_IP) the command and cURL share.
    • +
  • Verified / Not Verified badge — green when the (hw, variant, quant, strategy, nodes) combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.
    • +
+
+ +## Playground + +The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The base is read live from your Deploy selection — only your overrides change. + +Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. Any change flips the badge to **Not Verified** until the new configuration is run end-to-end. + +import { Playground } from "/src/snippets/_playground.jsx"; + + + +## 1. Model Introduction + +**Inkling-Small** is a Mixture-of-Experts model from Thinking Machines with **open weights** (BF16 and NVFP4 checkpoints below), in the same architecture family as Inkling. It handles text, image, and audio inputs natively, and exposes a **variable reasoning-effort** control to trade latency and cost against answer quality. This page covers serving Inkling-Small on SGLang, including its **MTP** speculative-decoding path and long-context prefix caching (unified radix cache + HiCache). + +**Resources:** HuggingFace — [Inkling-Small](https://huggingface.co/thinkingmachines/Inkling-Small) (BF16) · [Inkling-Small-NVFP4](https://huggingface.co/thinkingmachines/Inkling-Small-NVFP4). + +## 2. Configuration Tips + +**Multimodal.** The recipes pass `--enable-multimodal` so the server accepts image and audio inputs alongside text — drop it for text-only serving. + +**Memory pool ratios.** `--swa-full-tokens-ratio` and `--mamba-full-memory-ratio` (both default `0.1`) size the SWA and Mamba/sconv state pools; tune them to your workload's usage. + +**MTP needs `--enable-multi-layer-eagle`.** The MTP recipe drives Inkling-Small's multi-layer draft head; without this flag the standard EAGLE worker runs against it and outputs garbage. + +**Reasoning effort.** Pass `reasoning_effort` as one of the named levels below; requests that omit it default to `high`, and `max` is the strongest. Each level maps to an internal effort value (max at `0.99`): + + + + + + + + + + + + + + + + + +
reasoning_effortvalue
none0.0
minimal0.1
low0.2
medium0.7
high0.9
xhigh0.99
max0.99
+ +## 3. Advanced Usage + +### 3.1 Reasoning + +Enable the `inkling` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer into `reasoning_content` vs `content`. + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") + +resp = client.chat.completions.create( + model="thinkingmachines/Inkling-Small-NVFP4", + messages=[{"role": "user", "content": "What is 17 times 24?"}], + extra_body={"chat_template_kwargs": {"thinking": True}}, +) +msg = resp.choices[0].message +print("Reasoning:", getattr(msg, "reasoning_content", None)) +print("Answer:", msg.content) +``` + + + + + +```text Output +Reasoning: The user is asking for the product of 17 and 24. Let me calculate that. + +17 × 24 + +I can break this down: +17 × 20 = 340 +17 × 4 = 68 +340 + 68 = 408 + +Alternatively: +24 × 10 = 240 +24 × 7 = 168 +240 + 168 = 408 + +So the answer is 408. +Answer: 17 times 24 is **408**. + +Here's a quick breakdown: +- 17 × 20 = 340 +- 17 × 4 = 68 +- 340 + 68 = **408** +``` + + + +### 3.2 Tool Calling + +Enable the `inkling` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") + +tools = [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get the current weather for a location", + "parameters": { + "type": "object", + "properties": {"location": {"type": "string", "description": "The city name"}}, + "required": ["location"], + }, + }, + } +] + +resp = client.chat.completions.create( + model="thinkingmachines/Inkling-Small-NVFP4", + messages=[{"role": "user", "content": "What's the weather in Beijing?"}], + tools=tools, +) +msg = resp.choices[0].message +print("Reasoning:", getattr(msg, "reasoning_content", None)) +print("Content:", msg.content) +print("Tool calls:", msg.tool_calls) +``` + + + + + +```text Output +Reasoning: The user is asking for the weather in Beijing. I have a tool called `get_weather` that can get the current weather for a location. Let me call it with "Beijing" as the location. +Content: +Tool calls: [ChatCompletionMessageFunctionToolCall(id='call_98f772f3a0044f45b80c5ba5', function=Function(arguments='{"location": "Beijing"}', name='get_weather'), type='function', index=0)] +``` + + + +### 3.3 Multimodal Input (Image + Audio) + +Inkling-Small is multimodal: a single user message can mix **text**, **images**, and **audio**. Pass each media item as its own content part — `image_url` for images, `audio_url` for audio — with the `url` set to either an HTTP(S) link or a base64 `data:` URI. The server must be started with `--enable-multimodal` (already included in every recipe above). + + + +```python Example +import base64 +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") + +with open("image.png", "rb") as f: + image_b64 = base64.b64encode(f.read()).decode() +with open("audio.wav", "rb") as f: + audio_b64 = base64.b64encode(f.read()).decode() + +resp = client.chat.completions.create( + model="thinkingmachines/Inkling-Small-NVFP4", + messages=[ + { + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}}, + {"type": "audio_url", "audio_url": {"url": f"data:audio/wav;base64,{audio_b64}"}}, + {"type": "text", "text": "Describe the image, then transcribe the audio."}, + ], + } + ], + max_tokens=1024, +) +print(resp.choices[0].message.content) +``` + + + + +Images and audio can be sent as public HTTP(S) URLs instead of base64 — e.g. `{"type": "image_url", "image_url": {"url": "https://.../photo.jpg"}}`. Use one content part per media item; mix as many as the context budget allows. + + +### 3.4 LoRA (Serving Adapters) + +The **LoRA** deploy variant serves adapters on top of the frozen base model. Its launch command adds `--enable-lora --lora-paths lora0={{ADAPTER_PATH}} --max-loras-per-batch {{MAX_LORAS}}` — each adapter is registered under the **name** to the left of `=` (here `lora0`). Adapters can also be added/removed at runtime via the `POST /load_lora_adapter` endpoint. To serve several adapters, pass multiple `--lora-paths name=path` at launch and reference each by its name. + +Pick the adapter per request by that name — either in the `model` field with `base-model:adapter` syntax (recommended), or explicitly via `lora_path` in `extra_body`: + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") + +# Option A (recommended): ":" in the model field +resp = client.chat.completions.create( + model="thinkingmachines/Inkling-Small-NVFP4:lora0", + messages=[{"role": "user", "content": "Summarize the changelog."}], +) + +# Option B: explicit lora_path via extra_body +resp = client.chat.completions.create( + model="thinkingmachines/Inkling-Small-NVFP4", + messages=[{"role": "user", "content": "Summarize the changelog."}], + extra_body={"lora_path": "lora0"}, +) + +print(resp.choices[0].message.content) +``` + + + + +One adapter per request — omit the `:adapter` suffix (and `lora_path`) to hit the base model. Different requests **in the same batch** may use different adapters; the number of *distinct* adapters co-resident in a batch is capped by `--max-loras-per-batch` (the `MAX_LORAS` field, default `1`). If both `model:adapter` and `lora_path` are supplied, the `model` suffix takes precedence. + + +### 3.5 HiCache (Hierarchical KV Caching) + +Inkling-Small serves on SGLang's **unified radix cache**: the historically separate full-attention, SWA, and Mamba/sconv caches are combined into one radix tree with typed components, and native HiCache offloads cold prefix pages across tiers (GPU HBM → host DRAM → disk / remote). This expands effective prefix-cache capacity for multi-turn and long-context workloads. + +To enable HiCache, open the **HiCache** card in the [Playground above](#playground) and flip **Enable**, then pick a storage backend (`file` / `mooncake` / `nixl`) for the L3 tier. The Write policy defaults to `write_through`. + +### 3.6 Long Context (MXFP8 KV) + +The **Long Context** deploy strategy adds `--kv-cache-dtype mxfp8` on top of the Balanced recipe. KV entries are stored as block-scaled MXFP8 instead of BF16, so the SWA + Mamba/sconv memory pool holds roughly 2x as many tokens on the same GPU. Use it when you're context-bound or concurrency-bound. + +**Blackwell only.** MXFP8 KV cache requires Blackwell (B200 / B300 / GB200 / GB300), it's not offered on Hopper (H200). + +The tradeoff is not just a ~5% decode latency penalty from the extra quantize/dequantize work versus BF16 KV — storing KV in MXFP8 also introduces some accuracy loss at long context lengths. Treat it as a capacity lever, not a speed one — stay on **Balanced** if you have headroom in the memory pool and just want lower latency or maximum output quality. + +To try it, select the **Long Context** strategy in the Deploy panel above for any NVFP4 cell; the panel regenerates the launch command with `--kv-cache-dtype mxfp8` inserted. Verified end-to-end on B200, B300, and GB300. + +### 3.7 DSpark (Speculative Decoding) + +The **DSpark** deploy strategy is the second speculative-decoding path for Inkling-Small. Unlike **MTP**, which drives Inkling-Small's own multi-layer draft head, DSpark runs a **separate draft checkpoint** — `RadixArk/Inkling-Small-DSpark-Preview` — served unquantized alongside the NVFP4 target. + +DSpark support ships in the images listed in §1 (`dev-inkling-dspark` for CUDA 13, `dev-cu12-inkling-dspark` for CUDA 12), so no separate build is needed. Verified end-to-end on B200 (TP=8, NVFP4). diff --git a/docs_new/cookbook/autoregressive/ThinkingMachines/Inkling.mdx b/docs_new/cookbook/autoregressive/ThinkingMachines/Inkling.mdx index 334145279..3ef94f69b 100644 --- a/docs_new/cookbook/autoregressive/ThinkingMachines/Inkling.mdx +++ b/docs_new/cookbook/autoregressive/ThinkingMachines/Inkling.mdx @@ -34,10 +34,9 @@ Then run the **Python** output of the command panel below. There are two multi-arch (amd64 / arm64) CUDA builds plus a ROCm build; pick the CUDA build by your CUDA version, not your GPU: ```bash Command -docker pull lmsysorg/sglang:inkling-cu13 # CUDA 13 -docker pull lmsysorg/sglang:inkling-cu12 # CUDA 12 -docker pull lmsysorg/sglang:inkling-rocm700-mi35x # AMD MI350X / MI355X -docker pull lmsysorg/sglang:dev-cu13-inkling-dspark # CUDA 13 + DSpark support +docker pull lmsysorg/sglang:dev-inkling-dspark # CUDA 13 +docker pull lmsysorg/sglang:dev-cu12-inkling-dspark # CUDA 12 +docker pull lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark # AMD MI350X / MI355X ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. @@ -307,4 +306,4 @@ To try it, select the **Long Context** strategy in the Deploy panel above for an The **DSpark** deploy strategy is the second speculative-decoding path for Inkling. Unlike **MTP**, which drives Inkling's own multi-layer draft head, DSpark runs a **separate draft checkpoint** — `RadixArk/Inkling-DSpark-Preview` — served unquantized alongside the NVFP4 target. -It needs an SGLang build with DSpark support — `lmsysorg/sglang:dev-cu13-inkling-dspark`, which the Deploy panel's **Docker** command uses for this tier; the `inkling-cu13` / `inkling-cu12` images don't carry DSpark yet. Verified end-to-end on B200 (TP=8, NVFP4). +DSpark support ships in the images listed in §1 (`dev-inkling-dspark` for CUDA 13, `dev-cu12-inkling-dspark` for CUDA 12), so no separate build is needed. Verified end-to-end on B200 (TP=8, NVFP4). diff --git a/docs_new/docs.json b/docs_new/docs.json index c71d23f62..20e2eb946 100644 --- a/docs_new/docs.json +++ b/docs_new/docs.json @@ -987,7 +987,8 @@ { "group": "Thinking Machines", "pages": [ - "cookbook/autoregressive/ThinkingMachines/Inkling" + "cookbook/autoregressive/ThinkingMachines/Inkling", + "cookbook/autoregressive/ThinkingMachines/Inkling-Small" ] }, { diff --git a/docs_new/src/snippets/_deployment.jsx b/docs_new/src/snippets/_deployment.jsx index 5a74937dc..024c00f29 100644 --- a/docs_new/src/snippets/_deployment.jsx +++ b/docs_new/src/snippets/_deployment.jsx @@ -163,8 +163,8 @@ export const Deployment = ({ config, benchmarks }) => { color: isDark ? "#e5e7eb" : "#374151", whiteSpace: "pre-wrap", overflowX: "auto", margin: 0, }, - // Amber callout under the command when speculative decoding (MTP) is on - // but --max-running-requests isn't set (SGLang then caps it at 48). + // Amber callout under the command when speculative decoding (MTP, DSpark, ...) + // is on but --max-running-requests isn't set (SGLang then caps it at 48). mtpWarn: { margin: "8px 0 0", padding: "8px 12px", borderRadius: "8px", fontSize: "12px", lineHeight: "1.45", @@ -1176,13 +1176,37 @@ export const Deployment = ({ config, benchmarks }) => { return { ...cell, flags }; })(); const command = renderCommand(cellWithRatio, sel, env, runMode); - // MTP hint on the EFFECTIVE flags — speculation arrives via the Spec Decode - // overlay, never the cell. SGLang resets --max-running-requests to 48 when - // spec is on and it's unset. + // Speculative-decoding hint on the EFFECTIVE flags — speculation can arrive via + // the Spec Decode overlay as well as the cell. SGLang resets + // --max-running-requests to 48 when spec is on and it's unset; verified for both + // EAGLE/MTP and DSPARK (server_args reports max_running_requests=48 either way). const effFlags = cell ? [...overlayStrip(cell.flags, sel), ...overlayFlags(sel)] : []; - const mtpHint = - effFlags.some((f) => f.split(/[\s=]/)[0] === "--speculative-algorithm") && - !effFlags.some((f) => f.split(/[\s=]/)[0] === "--max-running-requests"); + const specAlgoFlag = effFlags.find( + (f) => f.split(/[\s=]/)[0] === "--speculative-algorithm"); + const specMrrFlag = effFlags.find( + (f) => f.split(/[\s=]/)[0] === "--max-running-requests"); + // Two cases, both worth surfacing when speculation is on: + // mtpHint — the flag is MISSING, so SGLang silently caps at 48 (a hazard) + // specPinnedHint— the recipe PINS it, which is safe but is a fixed number the + // reader still has to match to their own concurrency + const mtpHint = !!specAlgoFlag && !specMrrFlag; + const specPinnedHint = !!specAlgoFlag && !!specMrrFlag; + const specMrrValue = specMrrFlag + ? (specMrrFlag.split(/[\s=]/).filter(Boolean)[1] || "") + : ""; + // Name the algorithm in the banner rather than hardcoding "MTP" — the same reset + // applies to DSpark and friends, and a DSpark user reading "(MTP)" would be + // misled. The cookbook calls the EAGLE-based path MTP, so keep that mapping. + const SPEC_ALGO_LABEL = { + EAGLE: "MTP", EAGLE3: "MTP", FROZEN_KV_MTP: "MTP", + DSPARK: "DSpark", DFLASH: "DFlash", NGRAM: "N-gram", + STANDALONE: "standalone draft", + }; + const specAlgoName = (() => { + if (!specAlgoFlag) return "MTP"; + const v = specAlgoFlag.split(/[\s=]/).filter(Boolean)[1] || ""; + return SPEC_ALGO_LABEL[v.toUpperCase()] || v || "MTP"; + })(); // cell.warn may embed [label](#anchor) links — rendered as scrollIntoView // buttons, not hrefs, so the hash (which carries the selection) isn't overwritten. const renderWarn = (text) => { @@ -1413,7 +1437,12 @@ export const Deployment = ({ config, benchmarks }) => { {cell && cell.warn &&
⚠️ {renderWarn(cell.warn)}
} {mtpHint && (
- ⚠️ Speculative decoding (MTP) is on — SGLang resets --max-running-requests to 48 when it isn't set. Add --max-running-requests <N> sized for your target concurrency. + ⚠️ Speculative decoding ({specAlgoName}) is on — SGLang resets --max-running-requests to 48 when it isn't set. Add --max-running-requests <N> sized for your target concurrency. +
+ )} + {specPinnedHint && ( +
+ ℹ️ Speculative decoding ({specAlgoName}) is on and this recipe pins --max-running-requests to {specMrrValue}. Adjust it to match your target concurrency — if you remove the flag, SGLang falls back to 48.
)} )} diff --git a/docs_new/src/snippets/configs/thinkingmachines/inkling-small-benchmarks.jsx b/docs_new/src/snippets/configs/thinkingmachines/inkling-small-benchmarks.jsx new file mode 100644 index 000000000..c78f67281 --- /dev/null +++ b/docs_new/src/snippets/configs/thinkingmachines/inkling-small-benchmarks.jsx @@ -0,0 +1,72 @@ +// One entry per cell `match` tuple. `accuracy` is keyed to +// config.accuracyLabels in inkling-small.jsx. + +export const benchmarks = [ + { match: { hw: "b200" , variant: "default" , quant: "nvfp4" , strategy: "balanced" , nodes: "single" }, + sglang_version: "dev-inkling-dspark (b7252cc)", + accuracy: { aime26_pct: 95.42, bfcl_pct: 76.54, mmau_pct: 76.30 } }, + { match: { hw: "b300" , variant: "default" , quant: "nvfp4" , strategy: "balanced" , nodes: "single" }, + sglang_version: "dev (cb12a15)", + accuracy: { gsm8k_pct: 96.29 } }, + { match: { hw: "gb200" , variant: "default" , quant: "nvfp4" , strategy: "balanced" , nodes: "single" } }, + { match: { hw: "gb300" , variant: "default" , quant: "nvfp4" , strategy: "balanced" , nodes: "single" }, + sglang_version: "dev (cb12a15)", + accuracy: { gsm8k_pct: 96.66 } }, + { match: { hw: "h200" , variant: "default" , quant: "nvfp4" , strategy: "balanced" , nodes: "single" }, + sglang_version: "dev-inkling-dspark (b7252cc)", + accuracy: { aime26_pct: 95.00, bfcl_pct: 76.02, mmau_pct: 74.70 } }, + { match: { hw: "mi350x" , variant: "default" , quant: "bf16" , strategy: "balanced" , nodes: "single" } }, + { match: { hw: "mi355x" , variant: "default" , quant: "bf16" , strategy: "balanced" , nodes: "single" } }, + { match: { hw: "b200" , variant: "default" , quant: "nvfp4" , strategy: "mtp" , nodes: "single" }, + sglang_version: "dev-inkling-dspark (b7252cc)", + accuracy: { aime26_pct: 96.25, bfcl_pct: 77.57, mmau_pct: 77.20 } }, + { match: { hw: "b300" , variant: "default" , quant: "nvfp4" , strategy: "mtp" , nodes: "single" }, + sglang_version: "dev-cu13-inkling-dspark (86ccfef)", + accuracy: { gsm8k_pct: 96.06 } }, + { match: { hw: "gb200" , variant: "default" , quant: "nvfp4" , strategy: "mtp" , nodes: "single" } }, + { match: { hw: "gb300" , variant: "default" , quant: "nvfp4" , strategy: "mtp" , nodes: "single" }, + sglang_version: "dev-cu13-inkling-dspark (86ccfef)", + accuracy: { gsm8k_pct: 96.44 } }, + { match: { hw: "h200" , variant: "default" , quant: "nvfp4" , strategy: "mtp" , nodes: "single" }, + sglang_version: "dev-inkling-dspark (b7252cc)", + accuracy: { aime26_pct: 95.83, bfcl_pct: 76.09, mmau_pct: 76.80 } }, + { match: { hw: "b200" , variant: "default" , quant: "nvfp4" , strategy: "dspark" , nodes: "single" }, + sglang_version: "dev-inkling-dspark (b7252cc)", + accuracy: { aime26_pct: 95.83, bfcl_pct: 76.31, mmau_pct: 76.80 } }, + { match: { hw: "b300" , variant: "default" , quant: "nvfp4" , strategy: "dspark" , nodes: "single" }, + sglang_version: "dev-cu13-inkling-dspark (86ccfef)", + accuracy: { gsm8k_pct: 96.21 } }, + { match: { hw: "gb300" , variant: "default" , quant: "nvfp4" , strategy: "dspark" , nodes: "single" }, + sglang_version: "dev-cu13-inkling-dspark (86ccfef)", + accuracy: { gsm8k_pct: 95.83 } }, + { match: { hw: "h200" , variant: "default" , quant: "nvfp4" , strategy: "dspark" , nodes: "single" }, + sglang_version: "dev-inkling-dspark (b7252cc)", + accuracy: { aime26_pct: 96.25, bfcl_pct: 76.68, mmau_pct: 76.50 } }, + { match: { hw: "b200" , variant: "default" , quant: "nvfp4" , strategy: "long_context" , nodes: "single" }, + sglang_version: "dev (8fbf960)", + accuracy: { gsm8k_pct: 96.13 } }, + { match: { hw: "b300" , variant: "default" , quant: "nvfp4" , strategy: "long_context" , nodes: "single" }, + sglang_version: "dev (cb12a15)", + accuracy: { gsm8k_pct: 95.91 } }, + { match: { hw: "gb200" , variant: "default" , quant: "nvfp4" , strategy: "long_context" , nodes: "single" } }, + { match: { hw: "gb300" , variant: "default" , quant: "nvfp4" , strategy: "long_context" , nodes: "single" }, + sglang_version: "dev (cb12a15)", + accuracy: { gsm8k_pct: 96.21 } }, + { match: { hw: "gb300" , variant: "default" , quant: "bf16" , strategy: "balanced" , nodes: "multi-2" } }, + { match: { hw: "gb300" , variant: "default" , quant: "bf16" , strategy: "mtp" , nodes: "multi-2" } }, + { match: { hw: "b300" , variant: "default" , quant: "bf16" , strategy: "balanced" , nodes: "single" }, + sglang_version: "dev (cb12a15)", + accuracy: { gsm8k_pct: 96.29 } }, + { match: { hw: "b300" , variant: "default" , quant: "bf16" , strategy: "mtp" , nodes: "single" }, + sglang_version: "dev-cu13-inkling-dspark (86ccfef)", + accuracy: { gsm8k_pct: 96.36 } }, + { match: { hw: "b200" , variant: "default" , quant: "bf16" , strategy: "balanced" , nodes: "multi-2" } }, + { match: { hw: "b200" , variant: "default" , quant: "bf16" , strategy: "mtp" , nodes: "multi-2" } }, + { match: { hw: "b200" , variant: "lora" , quant: "nvfp4" , strategy: "balanced" , nodes: "single" } }, + { match: { hw: "b300" , variant: "lora" , quant: "nvfp4" , strategy: "balanced" , nodes: "single" } }, + { match: { hw: "gb200" , variant: "lora" , quant: "nvfp4" , strategy: "balanced" , nodes: "single" } }, + { match: { hw: "gb300" , variant: "lora" , quant: "nvfp4" , strategy: "balanced" , nodes: "single" } }, + { match: { hw: "h200" , variant: "lora" , quant: "nvfp4" , strategy: "balanced" , nodes: "single" } }, + { match: { hw: "gb300" , variant: "lora" , quant: "bf16" , strategy: "balanced" , nodes: "single" } }, + { match: { hw: "h200" , variant: "lora" , quant: "bf16" , strategy: "balanced" , nodes: "single" } }, +]; diff --git a/docs_new/src/snippets/configs/thinkingmachines/inkling-small.jsx b/docs_new/src/snippets/configs/thinkingmachines/inkling-small.jsx new file mode 100644 index 000000000..481be33b8 --- /dev/null +++ b/docs_new/src/snippets/configs/thinkingmachines/inkling-small.jsx @@ -0,0 +1,1260 @@ +// Single `export const config` literal — no spreads/calls/IIFE (Mintlify re-evals at hydration). +// Cells are denormalized: no `--nnodes`/`--node-rank`/`--dist-init-addr` literals — engine injects them. +// +// `{{MODEL_NAME}}` resolves to `modelNames` below (HF repos under the thinkingmachines org). + +export const config = { + modelName: "Inkling-Small", + + // Platform list inherited from the Inkling recipes (same architecture family). + supportedHardware: [ + "h200", "b200", "b300", "gb200", "gb300", + "mi350x", "mi355x", + ], + + variants: [ + { id: "default", label: "Default" }, + { id: "lora", label: "LoRA" }, + ], + quantizations: [ + { id: "nvfp4", label: "NVFP4" }, + { id: "bf16", label: "BF16" }, + ], + // One validated serving recipe per hardware -> a single honest `balanced` tier. + strategies: [ + { id: "balanced", label: "Balanced" }, + { id: "mtp", label: "MTP" }, + { id: "dspark", label: "DSpark" }, + { id: "long_context", label: "Long Context (MXFP8 KV)" }, + ], + nodesOptions: [ + { id: "single", label: "Single Node" }, + { id: "multi-2", label: "Multi-Nodes" }, + ], + + accuracyLabels: [ + ["gsm8k_pct", "GSM8K", "%"], + ["bfcl_pct", "BFCL (EXACT)", "%"], + ["mmau_pct", "MMAU", "%"], + ["mmmu_pro_pct", "MMMU-Pro", "%"], + ["aime25_pct", "AIME25 (pass@1)", "%"], + ["aime26_pct", "AIME26 (pass@1)", "%"], + ["niah_512k_pct", "NIAH @512K", "%"], + ["niah_1m_pct", "NIAH @1M", "%"], + ["hle_pct", "HLE", "%"], + ], + + // HF repos under the thinkingmachines org. + modelNames: { + "default|nvfp4": "thinkingmachines/Inkling-Small-NVFP4", + "default|bf16": "thinkingmachines/Inkling-Small", + "lora|nvfp4": "thinkingmachines/Inkling-Small-NVFP4", + "lora|bf16": "thinkingmachines/Inkling-Small", + }, + + placeholders: { + HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" }, + PORT: { target: "command", label: "Bind port", default: "30000" }, + NODE0_IP: { target: "command", label: "Head node IP", default: "" }, + NODE_RANK: { target: "command", label: "This node rank", default: "" }, + HF_TOKEN: { target: "command", label: "HF token (Docker)", default: "" }, + ADAPTER_PATH: { target: "command", label: "LoRA adapter dir", default: "" }, + MAX_LORAS: { target: "command", label: "Max LoRAs per batch", default: "1" }, + CURL_HOST: { target: "curl", label: "Server host", default: "localhost" }, + CURL_PORT: { target: "curl", label: "Server port", default: "30000" }, + }, + + curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\ +-H 'Content-Type: application/json' \\ +-d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"Hello"}] }'`, + + // NVIDIA: two multi-arch CUDA builds (dev-inkling-dspark for CUDA 13, + // dev-cu12-inkling-dspark for CUDA 12) — pick by your CUDA version, not by GPU. + // Panel defaults to cu13. AMD: dev-rocm720-mi35x-inkling-dspark (sglang-rocm repo). + // All tiers ship from the same images, DSpark included. + dockerImages: { + h200: "lmsysorg/sglang:dev-inkling-dspark", + b200: "lmsysorg/sglang:dev-inkling-dspark", + b300: "lmsysorg/sglang:dev-inkling-dspark", + gb200: "lmsysorg/sglang:dev-inkling-dspark", + gb300: "lmsysorg/sglang:dev-inkling-dspark", + mi350x: "lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark", + mi355x: "lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark", + }, + + github: { + cookbookModel: "thinkingmachines/inkling-small", + }, + + playgroundFeatures: { + + // ----- Card: "Attention Parallelism" ----- + // TP only. Inkling-Small needs TP=8 to hold the 1M-token SWA + Mamba/sconv pools + // (TP=4 can't fit — see §2). TP=16 is cross-node (multi-node path). + attention: { + knobs: [ + { id: "tp", label: "TP", values: [ + null, 4, 8, + { value: 16, disable: { nodes: ["single"] }, + disableReason: "TP=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." }, + ]}, + ], + }, + + // ----- Card: "MoE Parallelism" ----- + // Blackwell (SM100) runs the FlashInfer TRT-LLM routed FP4 experts; Hopper (SM90) + // has no FP4 runner and falls back to Marlin W4A16. + moe: { + backend: { + options: [ + { id: null, label: "Inherited" }, + // NVIDIA backends hidden on AMD; AITER/Triton hidden on NVIDIA. + { id: "flashinfer_trtllm_routed", label: "FlashInfer TRT-LLM (routed FP4)", + flags: ["--moe-runner-backend flashinfer_trtllm_routed"], + requiresHw: ["b200", "b300", "gb200", "gb300"], + hide: { hw: ["mi350x", "mi355x"] } }, + { id: "marlin", label: "Marlin (W4A16)", + flags: ["--moe-runner-backend marlin"], + hide: { hw: ["mi350x", "mi355x"] } }, + { id: "aiter", label: "AITER", + flags: ["--moe-runner-backend aiter"], + hide: { hw: ["h200", "b200", "b300", "gb200", "gb300"] } }, + { id: "triton", label: "Triton", + flags: ["--moe-runner-backend triton"], + hide: { hw: ["h200", "b200", "b300", "gb200", "gb300"] } }, + ], + }, + }, + + // ----- Card: "Parsers" ----- + parsers: { + items: [ + { id: "reasoning", label: "Reasoning Parser", flag: "--reasoning-parser inkling" }, + { id: "toolCall", label: "Tool Call Parser", flag: "--tool-call-parser inkling" }, + ], + }, + + // ----- Card: "Speculative Decoding" ----- Inkling-Small ships an MTP draft head. + speculative: { + options: [ + { id: "current", label: "Inherited from base" }, + { id: "off", label: "Off (greedy)" }, + { id: "mtp", label: "EAGLE / MTP 8-1-9", + flags: ["--speculative-algorithm EAGLE", "--speculative-num-steps 8", + "--speculative-eagle-topk 1", "--speculative-num-draft-tokens 9", + "--enable-multi-layer-eagle", "--speculative-use-rejection-sampling"] }, + ], + }, + + // ----- Card: "PD Disaggregation" ----- NVIDIA only; Mooncake MNNVL env gated to GB200/GB300. + pdDisagg: { + modes: [ + { id: "off", label: "Off" }, + { id: "prefill", label: "Prefill role", hide: { hw: ["mi350x", "mi355x"] } }, + { id: "decode", label: "Decode role", hide: { hw: ["mi350x", "mi355x"] } }, + ], + transferBackends: [ + { id: "mooncake", label: "Mooncake", + env: [ + "MC_FORCE_MNNVL=1", + "NCCL_MNNVL_ENABLE=1", + "NCCL_CUMEM_ENABLE=1", + "SGLANG_MOONCAKE_CUSTOM_MEM_POOL=True", + ], + envWhen: { hw: ["gb200", "gb300"] } }, + ], + // Router fronting both roles; 8998 = prefill bootstrap port (default). + router: { + port: 30080, + command: +`python3 -m sglang_router.launch_router \\ + --pd-disaggregation \\ + --prefill http://:{{PREFILL_PORT}} 8998 \\ + --decode http://:{{DECODE_PORT}} \\ + --host 0.0.0.0 --port {{ROUTER_PORT}} \\ + --disable-circuit-breaker \\ + --health-check-interval-secs 999999`, + }, + }, + + // ----- Card: "Hierarchical KV Cache" ----- Native HiCache over the unified radix tree. + hicache: { + backends: [ + { id: null, label: "Auto" }, + { id: "file", label: "File" }, + { id: "mooncake", label: "Mooncake" }, + { id: "nixl", label: "NiXL" }, + ], + writePolicies: [ + { id: "auto", label: "Auto" }, + { id: "write_through", label: "Write-through" }, + { id: "write_back", label: "Write-back" }, + ], + }, + }, + + cells: [ + // ==================================================================== + // NVIDIA Blackwell (SM100) + NVFP4 — FlashInfer TRT-LLM routed FP4 experts. + // B200 / B300 / GB300 verified; GB200 same-arch extrapolation. + // ==================================================================== + { + match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "single" }, + verified: true, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--quantization modelopt_fp4", + "--attention-backend fa4", + "--page-size 128", + "--fp4-gemm-backend flashinfer_trtllm", + "--moe-runner-backend flashinfer_trtllm_routed", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.85", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "single" }, + verified: true, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--quantization modelopt_fp4", + "--attention-backend fa4", + "--page-size 128", + "--fp4-gemm-backend flashinfer_trtllm", + "--moe-runner-backend flashinfer_trtllm_routed", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.85", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb200", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "single" }, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--quantization modelopt_fp4", + "--attention-backend fa4", + "--page-size 128", + "--fp4-gemm-backend flashinfer_trtllm", + "--moe-runner-backend flashinfer_trtllm_routed", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.85", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "single" }, + verified: true, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--quantization modelopt_fp4", + "--attention-backend fa4", + "--page-size 128", + "--fp4-gemm-backend flashinfer_trtllm", + "--moe-runner-backend flashinfer_trtllm_routed", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.85", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + // ==================================================================== + // NVIDIA Hopper (SM90) + NVFP4 — no FP4 MoE runner on Hopper -> Marlin W4A16. + // fa4 SplitKV auto-sets num_splits=1 on SM90. H200 verified. + // ==================================================================== + { + match: { hw: "h200", variant: "default", quant: "nvfp4", strategy: "balanced", nodes: "single" }, + verified: true, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--quantization modelopt_fp4", + "--attention-backend fa4", + "--page-size 128", + "--fp4-gemm-backend marlin", + "--moe-runner-backend marlin", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.85", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + // AMD ROCm (MI350X / MI355X) + BF16 — verified, TP=8. `--moe-runner-backend` + // sits right after `--tp` so the Playground AITER override (re-inserted at + // that anchor) reproduces this command exactly. + { + match: { hw: "mi350x", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" }, + verified: true, + env: [ + "SGLANG_USE_AITER=1", + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--moe-runner-backend aiter", + "--attention-backend triton", + "--disable-custom-all-reduce", + "--disable-prefill-cuda-graph", + "--mamba-radix-cache-strategy extra_buffer", + "--page-size 128", + "--mem-fraction-static 0.87", + "--swa-full-tokens-ratio 0.2", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "mi355x", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" }, + verified: true, + env: [ + "SGLANG_USE_AITER=1", + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--moe-runner-backend aiter", + "--attention-backend triton", + "--disable-custom-all-reduce", + "--disable-prefill-cuda-graph", + "--mamba-radix-cache-strategy extra_buffer", + "--page-size 128", + "--mem-fraction-static 0.87", + "--swa-full-tokens-ratio 0.2", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // Long Context (MXFP8 KV) — block-scaled KV cache shrinks the per-token + // KV footprint, raising how many tokens fit in the memory pool (longer + // context / more concurrent sequences) vs the default BF16 KV. Same base + // command as Balanced + `--kv-cache-dtype mxfp8`. B200 / B300 / GB300 + // verified end-to-end. + // ==================================================================== + { + match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "long_context", nodes: "single" }, + verified: true, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--quantization modelopt_fp4", + "--attention-backend fa4", + "--page-size 128", + "--fp4-gemm-backend flashinfer_trtllm", + "--moe-runner-backend flashinfer_trtllm_routed", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.85", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--kv-cache-dtype mxfp8", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "default", quant: "nvfp4", strategy: "long_context", nodes: "single" }, + verified: true, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--quantization modelopt_fp4", + "--attention-backend fa4", + "--page-size 128", + "--fp4-gemm-backend flashinfer_trtllm", + "--moe-runner-backend flashinfer_trtllm_routed", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.85", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--kv-cache-dtype mxfp8", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb200", variant: "default", quant: "nvfp4", strategy: "long_context", nodes: "single" }, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--quantization modelopt_fp4", + "--attention-backend fa4", + "--page-size 128", + "--fp4-gemm-backend flashinfer_trtllm", + "--moe-runner-backend flashinfer_trtllm_routed", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.85", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--kv-cache-dtype mxfp8", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "nvfp4", strategy: "long_context", nodes: "single" }, + verified: true, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--quantization modelopt_fp4", + "--attention-backend fa4", + "--page-size 128", + "--fp4-gemm-backend flashinfer_trtllm", + "--moe-runner-backend flashinfer_trtllm_routed", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.85", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--kv-cache-dtype mxfp8", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // MTP (speculative decoding) — Inkling-Small's multi-layer MTP draft head. + // --enable-multi-layer-eagle is REQUIRED (without it the standard EAGLE + // worker runs against the multi-layer draft and outputs garbage). + // B200 / B300 / GB300 / H200 verified end-to-end. + // ==================================================================== + { + match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "mtp", nodes: "single" }, + verified: true, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--quantization modelopt_fp4", + "--attention-backend fa4", + "--page-size 128", + "--fp4-gemm-backend flashinfer_trtllm", + "--moe-runner-backend flashinfer_trtllm_routed", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.60", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 8", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 9", + "--enable-multi-layer-eagle", + "--speculative-use-rejection-sampling", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "default", quant: "nvfp4", strategy: "mtp", nodes: "single" }, + verified: true, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--quantization modelopt_fp4", + "--attention-backend fa4", + "--page-size 128", + "--fp4-gemm-backend flashinfer_trtllm", + "--moe-runner-backend flashinfer_trtllm_routed", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.55", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 8", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 9", + "--enable-multi-layer-eagle", + "--speculative-use-rejection-sampling", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb200", variant: "default", quant: "nvfp4", strategy: "mtp", nodes: "single" }, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--quantization modelopt_fp4", + "--attention-backend fa4", + "--page-size 128", + "--fp4-gemm-backend flashinfer_trtllm", + "--moe-runner-backend flashinfer_trtllm_routed", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.75", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 8", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 9", + "--enable-multi-layer-eagle", + "--speculative-use-rejection-sampling", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "nvfp4", strategy: "mtp", nodes: "single" }, + verified: true, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--quantization modelopt_fp4", + "--attention-backend fa4", + "--page-size 128", + "--fp4-gemm-backend flashinfer_trtllm", + "--moe-runner-backend flashinfer_trtllm_routed", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.60", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 8", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 9", + "--enable-multi-layer-eagle", + "--speculative-use-rejection-sampling", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", variant: "default", quant: "nvfp4", strategy: "mtp", nodes: "single" }, + verified: true, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--quantization modelopt_fp4", + "--attention-backend fa4", + "--page-size 128", + "--fp4-gemm-backend marlin", + "--moe-runner-backend marlin", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.60", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 8", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 9", + "--enable-multi-layer-eagle", + "--speculative-use-rejection-sampling", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + // ==================================================================== + // DSpark (speculative decoding) — a separate draft checkpoint served + // unquantized, instead of Inkling-Small's own MTP head. The draft weights + // sit outside the FP4 target, hence mem-fraction 0.68. + // ==================================================================== + { + match: { hw: "b200", variant: "default", quant: "nvfp4", strategy: "dspark", nodes: "single" }, + verified: true, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--quantization modelopt_fp4", + "--attention-backend fa4", + "--page-size 128", + "--fp4-gemm-backend flashinfer_trtllm", + "--moe-runner-backend flashinfer_trtllm_routed", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.68", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--max-running-requests 68", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--skip-server-warmup", + "--speculative-algorithm DSPARK", + "--speculative-draft-model-path RadixArk/Inkling-Small-DSpark-Preview", + "--speculative-draft-model-quantization unquant", + "--chunked-prefill-size 8192", + "--cuda-graph-max-bs-prefill 8192", + "--disable-flashinfer-autotune", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "default", quant: "nvfp4", strategy: "dspark", nodes: "single" }, + verified: true, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--quantization modelopt_fp4", + "--attention-backend fa4", + "--page-size 128", + "--fp4-gemm-backend flashinfer_trtllm", + "--moe-runner-backend flashinfer_trtllm_routed", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.68", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--max-running-requests 68", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--skip-server-warmup", + "--speculative-algorithm DSPARK", + "--speculative-draft-model-path RadixArk/Inkling-Small-DSpark-Preview", + "--speculative-draft-model-quantization unquant", + "--chunked-prefill-size 8192", + "--cuda-graph-max-bs-prefill 8192", + "--disable-flashinfer-autotune", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "nvfp4", strategy: "dspark", nodes: "single" }, + verified: true, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--quantization modelopt_fp4", + "--attention-backend fa4", + "--page-size 128", + "--fp4-gemm-backend flashinfer_trtllm", + "--moe-runner-backend flashinfer_trtllm_routed", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.68", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--max-running-requests 68", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--skip-server-warmup", + "--speculative-algorithm DSPARK", + "--speculative-draft-model-path RadixArk/Inkling-Small-DSpark-Preview", + "--speculative-draft-model-quantization unquant", + "--chunked-prefill-size 8192", + "--cuda-graph-max-bs-prefill 8192", + "--disable-flashinfer-autotune", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", variant: "default", quant: "nvfp4", strategy: "dspark", nodes: "single" }, + verified: true, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--quantization modelopt_fp4", + "--attention-backend fa4", + "--page-size 128", + "--fp4-gemm-backend marlin", + "--moe-runner-backend marlin", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.60", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--max-running-requests 68", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--skip-server-warmup", + "--speculative-algorithm DSPARK", + "--speculative-draft-model-path RadixArk/Inkling-Small-DSpark-Preview", + "--speculative-draft-model-quantization unquant", + "--chunked-prefill-size 8192", + "--cuda-graph-max-bs-prefill 8192", + "--disable-flashinfer-autotune", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + + // ==================================================================== + // GB300 BF16 — 2x GB300 nodes (4 GPUs each) over MNNVL. The NCCL_MNNVL / + // NVLS / CUMEM envs are required: 2-node NCCL init hangs without them. + // MTP on BF16 requires the v3 MTP checkpoint + an SGLang revision with + // v3 MTP support. + // ==================================================================== + { + match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "multi-2" }, + env: [ + "NCCL_MNNVL_ENABLE=1", + "NCCL_NVLS_ENABLE=1", + "NCCL_CUMEM_ENABLE=1", + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dist-timeout 3600", + "--moe-runner-backend flashinfer_trtllm_routed", + "--attention-backend fa4", + "--disable-custom-all-reduce", + "--enable-torch-symm-mem", + "--page-size 128", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.87", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "default", quant: "bf16", strategy: "mtp", nodes: "multi-2" }, + env: [ + "NCCL_MNNVL_ENABLE=1", + "NCCL_NVLS_ENABLE=1", + "NCCL_CUMEM_ENABLE=1", + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--dist-timeout 3600", + "--moe-runner-backend flashinfer_trtllm_routed", + "--attention-backend fa4", + "--disable-custom-all-reduce", + "--enable-torch-symm-mem", + "--page-size 128", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.87", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 8", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 9", + "--enable-multi-layer-eagle", + "--speculative-use-rejection-sampling", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" }, + verified: true, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--attention-backend fa4", + "--page-size 128", + "--moe-runner-backend flashinfer_trtllm_routed", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + // BF16 weights are large on B300 — 0.85 caps the token pool near ~315k + // and rejects longer requests; 0.93 fits the full 1M context. + "--mem-fraction-static 0.93", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "default", quant: "bf16", strategy: "mtp", nodes: "single" }, + verified: true, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--attention-backend fa4", + "--page-size 128", + "--moe-runner-backend flashinfer_trtllm_routed", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.65", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 8", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 9", + "--enable-multi-layer-eagle", + "--speculative-use-rejection-sampling", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "bf16", strategy: "balanced", nodes: "multi-2" }, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 16", + "--moe-runner-backend flashinfer_trtllm_routed", + "--attention-backend fa4", + "--disable-custom-all-reduce", + "--page-size 128", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.87", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b200", variant: "default", quant: "bf16", strategy: "mtp", nodes: "multi-2" }, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 16", + "--moe-runner-backend flashinfer_trtllm_routed", + "--attention-backend fa4", + "--disable-custom-all-reduce", + "--page-size 128", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.87", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--speculative-algorithm EAGLE", + "--speculative-num-steps 8", + "--speculative-eagle-topk 1", + "--speculative-num-draft-tokens 9", + "--enable-multi-layer-eagle", + "--speculative-use-rejection-sampling", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + // ==================================================================== + // LoRA serving. LoRA prefill currently requires CUDA graphs to be disabled. + // Set MAX_LORAS to the number of distinct adapters served (1 is fastest + // for single-adapter serving). Verified cells are marked individually + // after end-to-end validation. + // ==================================================================== + { + match: { hw: "b200", variant: "lora", quant: "nvfp4", strategy: "balanced", nodes: "single" }, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + "SGLANG_EXPERIMENTAL_LORA_OPTI=1", + "SGLANG_OPT_LORA_OVERLAP_MAIN_ALLOC=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--quantization modelopt_fp4", + "--attention-backend fa4", + "--page-size 128", + "--fp4-gemm-backend marlin", + "--moe-runner-backend experimental_sgl_marlin", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.80", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--enable-lora", + "--disable-prefill-cuda-graph", + "--lora-backend triton", + "--lora-use-virtual-experts", + "--max-loras-per-batch {{MAX_LORAS}}", + "--lora-paths lora0={{ADAPTER_PATH}}", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "b300", variant: "lora", quant: "nvfp4", strategy: "balanced", nodes: "single" }, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + "SGLANG_EXPERIMENTAL_LORA_OPTI=1", + "SGLANG_OPT_LORA_OVERLAP_MAIN_ALLOC=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--quantization modelopt_fp4", + "--attention-backend fa4", + "--page-size 128", + "--fp4-gemm-backend marlin", + "--moe-runner-backend experimental_sgl_marlin", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.80", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--enable-lora", + "--disable-prefill-cuda-graph", + "--lora-backend triton", + "--lora-use-virtual-experts", + "--max-loras-per-batch {{MAX_LORAS}}", + "--lora-paths lora0={{ADAPTER_PATH}}", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb200", variant: "lora", quant: "nvfp4", strategy: "balanced", nodes: "single" }, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + "SGLANG_EXPERIMENTAL_LORA_OPTI=1", + "SGLANG_OPT_LORA_OVERLAP_MAIN_ALLOC=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--quantization modelopt_fp4", + "--attention-backend fa4", + "--page-size 128", + "--fp4-gemm-backend marlin", + "--moe-runner-backend experimental_sgl_marlin", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.80", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--enable-lora", + "--disable-prefill-cuda-graph", + "--lora-backend triton", + "--lora-use-virtual-experts", + "--max-loras-per-batch {{MAX_LORAS}}", + "--lora-paths lora0={{ADAPTER_PATH}}", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "lora", quant: "nvfp4", strategy: "balanced", nodes: "single" }, + verified: true, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + "SGLANG_EXPERIMENTAL_LORA_OPTI=1", + "SGLANG_OPT_LORA_OVERLAP_MAIN_ALLOC=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--quantization modelopt_fp4", + "--attention-backend fa4", + "--fp4-gemm-backend marlin", + "--moe-runner-backend experimental_sgl_marlin", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.80", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--enable-lora", + "--disable-prefill-cuda-graph", + "--lora-backend triton", + "--lora-use-virtual-experts", + "--max-loras-per-batch {{MAX_LORAS}}", + "--lora-paths lora0={{ADAPTER_PATH}}", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", variant: "lora", quant: "nvfp4", strategy: "balanced", nodes: "single" }, + verified: true, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + "SGLANG_EXPERIMENTAL_LORA_OPTI=1", + "SGLANG_OPT_LORA_OVERLAP_MAIN_ALLOC=1", + "SGLANG_OPT_USE_INKLING_SHEARED_BIAS=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--quantization modelopt_fp4", + "--attention-backend fa4", + "--page-size 128", + "--fp4-gemm-backend marlin", + "--moe-runner-backend experimental_sgl_marlin", + "--disable-custom-all-reduce", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.85", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--enable-lora", + "--disable-prefill-cuda-graph", + "--lora-backend triton", + "--lora-use-virtual-experts", + "--max-loras-per-batch {{MAX_LORAS}}", + "--lora-paths lora0={{ADAPTER_PATH}}", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "gb300", variant: "lora", quant: "bf16", strategy: "balanced", nodes: "single" }, + verified: true, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + "SGLANG_EXPERIMENTAL_LORA_OPTI=1", + "SGLANG_OPT_LORA_OVERLAP_MAIN_ALLOC=1", + "SGLANG_OPT_USE_JIT_KERNEL_MOE_ALIGN=1", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 4", + "--moe-runner-backend experimental_sgl_trtllm", + "--attention-backend fa4", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.87", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--enable-lora", + "--disable-prefill-cuda-graph", + "--lora-backend triton", + "--lora-use-virtual-experts", + "--max-loras-per-batch {{MAX_LORAS}}", + "--lora-paths lora0={{ADAPTER_PATH}}", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + { + match: { hw: "h200", variant: "lora", quant: "bf16", strategy: "balanced", nodes: "single" }, + verified: true, + env: [ + "SGLANG_ENABLE_UNIFIED_RADIX_TREE=1", + "SGLANG_EXPERIMENTAL_LORA_OPTI=1", + "SGLANG_OPT_LORA_OVERLAP_MAIN_ALLOC=1", + "SGLANG_OPT_USE_JIT_KERNEL_MOE_ALIGN=1", + "SGLANG_OPT_USE_INKLING_SHEARED_BIAS=0", + ], + flags: [ + "--trust-remote-code", + "--model-path {{MODEL_NAME}}", + "--tp 8", + "--moe-runner-backend triton", + "--attention-backend fa4", + "--page-size 128", + "--disable-custom-all-reduce", + "--enable-torch-symm-mem", + "--mamba-radix-cache-strategy extra_buffer", + "--mem-fraction-static 0.87", + "--swa-full-tokens-ratio 0.1", + "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", + "--reasoning-parser inkling", + "--tool-call-parser inkling", + "--enable-lora", + "--disable-prefill-cuda-graph", + "--lora-backend triton", + "--lora-use-virtual-experts", + "--max-loras-per-batch {{MAX_LORAS}}", + "--lora-paths lora0={{ADAPTER_PATH}}", + "--host {{HOST_IP}}", + "--port {{PORT}}", + ], + }, + ], +}; diff --git a/docs_new/src/snippets/configs/thinkingmachines/inkling.jsx b/docs_new/src/snippets/configs/thinkingmachines/inkling.jsx index 319f70320..05de1befa 100644 --- a/docs_new/src/snippets/configs/thinkingmachines/inkling.jsx +++ b/docs_new/src/snippets/configs/thinkingmachines/inkling.jsx @@ -72,19 +72,18 @@ export const config = { -H 'Content-Type: application/json' \\ -d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"Hello"}] }'`, - // NVIDIA: two multi-arch CUDA builds (inkling-cu12 / inkling-cu13) — pick by your - // CUDA version, not by GPU. AMD: inkling-rocm700-mi35x. Panel defaults to cu13. - // The DSpark tier needs its own preview build (DSpark isn't in the inkling-cu1x - // images yet), so it takes a `hw|quant|strategy` key. + // NVIDIA: two multi-arch CUDA builds (dev-inkling-dspark for CUDA 13, + // dev-cu12-inkling-dspark for CUDA 12) — pick by your CUDA version, not by GPU. + // Panel defaults to cu13. AMD: dev-rocm720-mi35x-inkling-dspark (sglang-rocm repo). + // All tiers ship from the same images, DSpark included. dockerImages: { - "b200|nvfp4|dspark": "lmsysorg/sglang:dev-cu13-inkling-dspark", - h200: "lmsysorg/sglang:inkling-cu13", - b200: "lmsysorg/sglang:inkling-cu13", - b300: "lmsysorg/sglang:inkling-cu13", - gb200: "lmsysorg/sglang:inkling-cu13", - gb300: "lmsysorg/sglang:inkling-cu13", - mi350x: "lmsysorg/sglang:inkling-rocm700-mi35x", - mi355x: "lmsysorg/sglang:inkling-rocm700-mi35x", + h200: "lmsysorg/sglang:dev-inkling-dspark", + b200: "lmsysorg/sglang:dev-inkling-dspark", + b300: "lmsysorg/sglang:dev-inkling-dspark", + gb200: "lmsysorg/sglang:dev-inkling-dspark", + gb300: "lmsysorg/sglang:dev-inkling-dspark", + mi350x: "lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark", + mi355x: "lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark", }, github: { @@ -689,9 +688,8 @@ export const config = { // ==================================================================== // DSpark (speculative decoding) — separate draft checkpoint // (RadixArk/Inkling-DSpark-Preview, served unquantized) instead of Inkling's - // own MTP head, so it needs a build carrying DSpark support: the - // `dev-cu13-inkling-dspark` image (see the dockerImages key above). - // The draft weights sit outside the FP4 target, hence mem-fraction 0.68. + // own MTP head. The draft weights sit outside the FP4 target, hence + // mem-fraction 0.68. // B200 verified end-to-end. // ==================================================================== { @@ -714,6 +712,7 @@ export const config = { "--mem-fraction-static 0.68", "--swa-full-tokens-ratio 0.1", "--mamba-full-memory-ratio 0.1", + "--enable-multimodal", "--max-running-requests 68", "--reasoning-parser inkling", "--tool-call-parser inkling",