minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

Add Intern-S2-Mobius cookbook (#33820)

Browse Source 
Co-authored-by: Justin Tong <justintong0323@outlook.com>
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
This commit is contained in:
Xinyuan Tong
2026-08-10 13:13:35 -07:00
committed by GitHub
co-authored by Justin Tong Zijie Xia
parent a2161ce682
commit e54c153ba6
6 changed files with 541 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/InternLM/Intern-S2-Mobius.mdx
+274
View File
@@ -0,0 +1,274 @@
---
title: Intern-S2-Mobius
description: "Deploy Intern-S2-Mobius with SGLang — InternLM's Mobius-v0 multimodal model with a globally shared Knowledge Memory, hybrid GDN + full attention, MTP (NEXTN) speculative decoding, and 256K context, on NVIDIA H200 and B200."
tag: NEW
---
## Deployment
<a id="install" />
<Accordion title="Install SGLang">
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
<Tabs>
<Tab title="Python (pip / uv)">
```bash Command
pip install --upgrade pip
pip install uv
uv pip install sglang
```
Then run the **Python** output of the command panel below in that environment.
</Tab>
<Tab title="Docker">
```bash Command
docker pull lmsysorg/sglang:dev
```
Intern-S2-Mobius was upstreamed in PR [#33691](https://github.com/sgl-project/sglang/pull/33691) (merged 2026-08-08) — it lives on `lmsysorg/sglang:dev` (nightly) until the next release cut. For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
</Tab>
</Tabs>
</Accordion>
Pick your hardware + recipe to generate the launch command. The two serving strategies cover the main operating points:
- **Low-Latency** — MTP (NEXTN) speculative decoding on. Fastest reply for a single user.
- **High-Throughput** — spec off, more tokens per second when many users share the server.
import { Deployment } from "/src/snippets/_deployment.jsx";
import { config } from "/src/snippets/configs/internlm/intern-s2-mobius.jsx";
import { benchmarks } from "/src/snippets/configs/internlm/intern-s2-mobius-benchmarks.jsx";
<Deployment config={config} benchmarks={benchmarks} />
<Note>
Speed numbers are measured with `--random-range-ratio 1.0`, `--flush-cache`, on 2×H200 TP=2 against `main @ e0828ee3` + PR [#33691](https://github.com/sgl-project/sglang/pull/33691) head (since merged 2026-08-08 — `lmsysorg/sglang:dev` is the live equivalent). GSM8K is the full 1319-example test split; GPQA is Diamond 198 problems × 8 repeats (pass@1 avg-of-8). Both ran with no server-side sampling override, so the checkpoint's `generation_config.json` defaults applied (temperature 1.0, top_p 0.95, top_k 20). The B200 recipes are inferred from the H200 ones and unverified — same flags, just a TP=2 or TP=1 Blackwell equivalent.
</Note>
## Playground
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.
import { Playground } from "/src/snippets/_playground.jsx";
<Playground config={config} />
## 1. Model Introduction
**Intern-S2-Mobius** is InternLM's 35B scientific multimodal foundation model built on the **Mobius-v0** architecture (continually pre-trained from Qwen3.5-35B, then SFT and RL post-trained). Instead of binding knowledge storage and reasoning computation layer by layer as conventional Transformers do, Mobius organizes knowledge into a **globally shared Memory** that multiple **Reasoners** iteratively query against, yielding two native capabilities:
- **Backward Residual Connection** — shallow and deep reasoning stages can reach knowledge across the model rather than relying only on forward layer-wise flow.
- **Dynamic Latent Reasoning** — recurrent latent iteration refines hidden states before decoding, internalizing part of the deliberation process and shrinking visible chain-of-thought. The reported result is roughly a **4× end-to-end inference speedup** over the Qwen3.5-35B baseline while holding comparable scores on general reasoning benchmarks and improving on scientific tasks (Biology-Instructions, Mol-Instructions, MolecularIQ).
On the serving side the model is a hybrid: 30 of 40 transformer layers use **GDN (Gated Delta Net) linear attention** (`kimi-linear`-family), with a **full-attention** layer every 4th layer (`full_attention_interval: 4` → **10 full-attention** layers), and the bottom of the stack is MoE-routed (2,560 routed experts × 512 intermediate, 8 active per token); a separate **MoE-256 / top-8 MTP (NEXTN) layer** feeds speculative decoding. It takes images via a vision tower and recognizes the standard `<|vision_start|>…<|vision_end|>` + `<|image_pad|>` markers. Context length is **262,144** tokens.
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "32%"}} />
<col style={{width: "32%"}} />
<col style={{width: "16%"}} />
<col style={{width: "20%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Variant</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Architecture</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Context</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>License</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong><a href="https://huggingface.co/internlm/Intern-S2-Mobius">Intern-S2-Mobius</a></strong></td>
<td style={{padding: "9px 12px"}}>Mobius-v0 · GDN ×30 + full ×10 · MoE-2560 / top-8 · MTP · BF16</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>262,144</td>
<td style={{padding: "9px 12px"}}><a href="https://huggingface.co/internlm/Intern-S2-Mobius/blob/main/LICENSE">Apache-2.0</a></td>
</tr>
</tbody>
</table>
**Recommended generation:** `temperature=0.8`, `top_p=1.0`, `top_k=50`, `min_p=0.0` — the values the [model card](https://huggingface.co/internlm/Intern-S2-Mobius) recommends. Note these are *not* what the checkpoint ships in `generation_config.json` (`temperature=1.0`, `top_p=0.95`, `top_k=20`), and SGLang applies that file by default (`--sampling-defaults model`) — so send the recommended values explicitly per request if you want them.
**Resources:** [HuggingFace](https://huggingface.co/internlm/Intern-S2-Mobius) · [GitHub (InternLM/Intern-S2-Mobius)](https://github.com/InternLM/Intern-S2-Mobius).
## 2. Configuration Tips
- **Trust remote code is required.** Intern-S2-Mobius ships a custom `configuration_interns2_mobius.py` / `modeling_interns2_mobius.py` on its HF repo; every recipe adds `--trust-remote-code`.
- **Speculative decoding schedule.** The checkpoint ships one MTP layer. Enable MTP for the lowest latency (`--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`). We measured accept-length ~3.9/4 draft tokens at 8K-in / 1K-out, roughly tripling single-stream decode speed (median TPOT 9.79 ms → 3.13 ms at conc=1, 14.26 ms → 6.84 ms at conc=16) and roughly doubling mid-concurrency total throughput (9358 → 18029 tokens/s at conc=16, 21395 → 26033 tokens/s at conc=64). The high-throughput recipe stays spec-off because once you can batch wide, its saturation point is higher (34786 tokens/s at conc=256 vs the spec recipe's peak at conc=64).
- **Mamba pool sizing.** GDN layers live in a separate Mamba state pool; the `--mamba-full-memory-ratio` (defaults to 0.9) controls the split between the 10 full-attention layers' KV pool and the 30 GDN layers' conv+SSM state pool. Default split comfortably handles conc=64 on a 2×H200 node; if you need higher concurrency than `--max-running-requests` allows for your workload, raise `--mamba-full-memory-ratio` slightly (each +1% mamba ratio costs full-attn KV).
- **Vision input.** Images are accepted via the standard `image_url` chat message type. Vision tokens are counted into the prompt (`prompt_tokens_details.image_tokens` shows the count), and the model honors `<|vision_start|>` / `<|vision_end|>` boundaries exactly.
- **B200 sizing.** B200 (192 GB HBM) fits the BF16 weights + KV + Mamba pool on a **single GPU** with `--tp 1`. The B200 cells in the panel inherit the H200 recipe with only `--tp` changed — unverified; treat them as a starting point until the Intern-S2-Mobius team publishes a Blackwell measurement.
## 3. Advanced Usage
<Note>
The outputs below are verbatim captures from a live server (sampling per the checkpoint's `generation_config.json`, temperature 1.0). Re-running the same request yields a semantically equivalent but textually different trace — treat them as representative, not deterministic.
</Note>
### 3.1 Reasoning
InternS2-Mobius is a hybrid-reasoning model — thinking traces start with "Thinking Process:" before the final answer. Enable the **`qwen3` reasoning parser** (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to split thinking into `message.reasoning_content` and the answer into `message.content`.
<Accordion title="Reasoning Example (Python)">
```python Example
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
model="internlm/Intern-S2-Mobius",
messages=[{"role": "user", "content": "What is 15% of 240?"}],
)
msg = resp.choices[0].message
print("Reasoning:", getattr(msg, "reasoning_content", None))
print("Answer:", msg.content)
```
</Accordion>
<Accordion title="Example Output">
```text Output
Reasoning: Thinking Process:
1. **Identify the core question:** The user is asking for 15% of 240.
2. **Determine the calculation method:** To find a percentage of a
number, multiply the number by the percentage expressed as a decimal
or fraction.
* Percentage: 15%
* Decimal: 0.15
* Fraction: 15/100
3. **Perform the calculation:** $240 \times 0.15$
* Method 1: $240 \times 0.10 = 24$ (10%) and
$240 \times 0.05 = 12$ (5%). Then add them: $24 + 12 = 36$.
* Method 2: $240 \times 15 = 3600$. Divide by 100 -> 36.
4. **Verify the result:** The calculation is correct.
5. **Formulate the answer:** State the final number clearly.cw
Answer:
15% of 240 is **36**.
Here is the math:
$240 \times 0.15 = 36$
```
</Accordion>
### 3.2 Tool Calling
Enable the **`qwen3_coder` tool-call parser** (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. Intern-S2-Mobius emits `<tool_call>…<function=name>…<parameter=key>…value…</parameter>…` — this is exactly the format `qwen3_coder` parses; without the parser the call is left as raw text in `content`. On this thinking-mode model the turn also fills `reasoning_content`, so print both fields.
**Auto-resolution works out of the box.** Intern-S2-Mobius's chat template contains the `<function=` / `<parameter=` markers the auto-detector keys on, so `--reasoning-parser auto --tool-call-parser auto` resolves to `qwen3` / `qwen3_coder` without any extra config (verified on this build by tailing the server log's "Auto-detected …" lines + a live `tools` request). You can pass the literal `qwen3_coder` slug, but you don't have to.
<Accordion title="Tool Calling Example (Python)">
```python Example
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
tools = [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string", "description": "The city name"},
},
"required": ["location"],
},
},
}]
resp = client.chat.completions.create(
model="internlm/Intern-S2-Mobius",
messages=[{"role": "user", "content": "What is the weather in Beijing?"}],
tools=tools,
)
msg = resp.choices[0].message
print("Reasoning:", getattr(msg, "reasoning_content", None))
print("Tool calls:", msg.tool_calls)
```
</Accordion>
<Accordion title="Example Output">
```text Output
Reasoning: The user is asking for the weather in Beijing. I have access
to a get_weather function that can get the current weather for a
location. The function requires a "location" parameter which should be
the city name. In this case, the user specified "Beijing", so I should
use that as the location parameter.
Tool calls: [
{
"id": "call_545b5956b4c3457286261490",
"index": 0,
"type": "function",
"function": {"name": "get_weather", "arguments": "{\"location\": \"Beijing\"}"}
}
]
finish_reason: tool_calls
```
</Accordion>
### 3.3 Vision Input
Intern-S2-Mobius takes images via the OpenAI-compatible `image_url` content type. Vision input works with the same server the Deploy panel produces — no extra model-specific flags needed.
<Accordion title="Vision Example (Python)">
```python Example
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
model="internlm/Intern-S2-Mobius",
messages=[{
"role": "user",
"content": [
{"type": "image_url",
"image_url": {"url": "https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg"}},
{"type": "text", "text": "Describe this image in one sentence."},
],
}],
)
msg = resp.choices[0].message
print("Reasoning:", getattr(msg, "reasoning_content", None))
print("Answer:", msg.content)
```
</Accordion>
<Accordion title="Example Output">
```text Output
Reasoning: The user wants a one-sentence description of the image.
Key elements: Tiger, lying down, grass, looking at camera.
Drafting: A tiger is lying in the green grass looking directly at the
camera.
Refining for flow and detail: A majestic tiger with striking orange and
black stripes rests calmly on a bed of lush green grass, staring
intently directly at the viewer.
Answer:
A majestic tiger with striking orange and black stripes rests calmly on a
bed of lush green grass, staring intently directly at the viewer.
```
</Accordion>
docs/cookbook/autoregressive/InternLM/Intern-S2-Preview.mdx
-1
View File
@@ -2,7 +2,6 @@
title: Intern-S2-Preview
metatags:
description: "Deploy Intern-S2-Preview with SGLang"
tag: NEW
---
## 1. Model Introduction
docs/cookbook/autoregressive/intro.mdx
+1 -1
View File
@@ -112,7 +112,7 @@ metatags:
<Card
title="InternLM"
mode="card"
href="/cookbook/autoregressive/InternLM/Intern-S2-Preview"
href="/cookbook/autoregressive/InternLM/Intern-S2-Mobius"
img="/cards/logos/internlm.png"
/>
<Card
docs/docs.json
+1
View File
@@ -1351,6 +1351,7 @@
{
"group": "InternLM",
"pages": [
"cookbook/autoregressive/InternLM/Intern-S2-Mobius",
"cookbook/autoregressive/InternLM/Intern-S2-Preview",
"cookbook/autoregressive/InternLM/Intern-S1"
]
docs/src/snippets/configs/internlm/intern-s2-mobius-benchmarks.jsx
+50
View File
@@ -0,0 +1,50 @@
// Intern-S2-Mobius per-cell benchmark numbers, keyed by the same `match` tuple as
// intern-s2-mobius.jsx cells. All H200 numbers measured in this work on 2xH200
// (TP=2, sglang main @ e0828ee3 + PR #33691 head — model landed in main 2026-08-08, so lmsysorg/sglang:dev is equivalent now). Speed workload = random 8K-in /
// 1K-out, --random-range-ratio 1.0, --flush-cache, warmup 8-16 prompts, varied seed.
// ttft_ms / tpot_ms are P50. tokens_per_sec_per_gpu = total (in+out)/GPU.
export const benchmarks = [
// ==== H200, low-latency (MTP NEXTN 3-1-4) ====
// EAGLE MTP doubles-to-triples single-stream decode and lifts mid-concurrency
// total throughput; at conc=64 the spec recipe still wins on total throughput
// (26033 vs 21395 tok/s) but the no-spec recipe at conc=256 wins overall
// (34786 — its saturation point). Spec accept_length ~3.9/4.
{
match: { hw: "h200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },
sglang_version: "main @ e0828ee3",
speed: [
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
ttft_ms: 177.91, tpot_ms: 3.13, tokens_per_sec_per_gpu: 1283.2 },
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
ttft_ms: 1139.88, tpot_ms: 6.84, tokens_per_sec_per_gpu: 9014.6 },
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
ttft_ms: 4440.38, tpot_ms: 12.17, tokens_per_sec_per_gpu: 13016.5 },
],
accuracy: { gsm8k_pct: 96.66, gpqa_pct: 79.23 },
},
// ==== H200, high-throughput (no speculative decoding) ====
// Saturates at conc=1024 submitted (server clamps max_running to ~730 — the
// real concurrency observed). Spec-off at the largest batch sizes reaches
// ~35k total tok/s, >1.3× the spec-on peak at conc=64, which is why the
// high-throughput recipe stays spec-free.
{
match: { hw: "h200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" },
sglang_version: "main @ e0828ee3",
speed: [
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
ttft_ms: 1166.47, tpot_ms: 14.26, tokens_per_sec_per_gpu: 4679.2 },
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 64 },
ttft_ms: 4112.65, tpot_ms: 22.91, tokens_per_sec_per_gpu: 10697.7 },
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 256 },
ttft_ms: 16290.63, tpot_ms: 50.31, tokens_per_sec_per_gpu: 17393.4 },
{ workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1024 },
ttft_ms: 182448.22, tpot_ms: 123.82, tokens_per_sec_per_gpu: 11875.5 },
],
accuracy: { gsm8k_pct: 96.82 },
},
// ==== B200 recipes are inferred from the H200 ones — benchmarks pending. ====
{ match: { hw: "b200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" } },
{ match: { hw: "b200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" } },
];
docs/src/snippets/configs/internlm/intern-s2-mobius.jsx
+215
View File
@@ -0,0 +1,215 @@
// Single `export const config` literal — no spreads/calls/IIFE (Mintlify re-evals at hydration).
// Cells are denormalized: no `--nnodes`/`--node-rank`/`--dist-init-addr`/`--host`/`--port` literals — engine injects them.
export const config = {
modelName: "Intern-S2-Mobius",
supportedHardware: ["h200", "b200"],
// Single released checkpoint — no variant axis.
variants: [
{ id: "default", label: "Intern-S2-Mobius", subtitle: "Mobius-v0 · BF16" },
],
quantizations: [
{ id: "bf16", label: "BF16" },
],
strategies: [
{ id: "low-latency", label: "Low-Latency" },
{ id: "high-throughput", label: "High-Throughput" },
],
nodesOptions: [
{ id: "single", label: "Single Node" },
],
modelNames: {
"default|bf16": "internlm/Intern-S2-Mobius",
},
placeholders: {
HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" },
PORT: { target: "command", label: "Bind port", default: "30000" },
CURL_HOST: { target: "curl", label: "Server host", default: "localhost" },
CURL_PORT: { target: "curl", label: "Server port", default: "30000" },
},
curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\
-H 'Content-Type: application/json' \\
-d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"Hello"}] }'`,
// Reproduce commands for the Benchmark card's "⚡ Reproduce" modal.
benchmarkCommands: {
speed:
`python3 -m sglang.bench_serving \\
--backend sglang \\
--host {{CURL_HOST}} --port {{CURL_PORT}} \\
--model {{MODEL_NAME}} \\
--dataset-name {{DATASET}} \\
--random-input-len {{ISL}} --random-output-len {{OSL}} \\
--random-range-ratio 1.0 \\
--num-prompts {{NUM_PROMPTS}} --max-concurrency {{MAX_CONCURRENCY}} \\
--warmup-requests 8 --flush-cache`,
accuracy: {
gsm8k_pct:
`# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval
sgl-eval run gsm8k \\
--base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1 \\
--num-threads 32`,
gpqa_pct:
`# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval
sgl-eval run gpqa \\
--base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1 \\
--num-threads 16`,
},
numPromptsByConc: { 1: 8, 16: 32, 64: 128, 256: 512, 1024: 1024 },
},
// Per-variant accuracy applied to every cell; per-cell `accuracy` overrides.
// Measured on 2xH200 with the low-latency (EAGLE NEXTN 3-1-4) recipe, with no
// server-side sampling override, so the checkpoint's generation_config.json
// defaults apply throughout (temp=1.0, top_p=0.95, top_k=20). NOTE: the model
// card separately recommends temp=0.8 / top_p=1.0 / top_k=50 / min_p=0.0 —
// those are NOT what generation_config.json ships, so they only apply when the
// client sends them explicitly.
// gsm8k : full 1319-example test split.
// gpqa : Diamond, 198 problems × 8 repeats, pass@1 avg-of-8 = 79.23% ± 1.49,
// pass@8 = 88.38 %, majority@8 = 80.56 %, stop_rate = 100 %.
defaultAccuracy: {
default: { gsm8k_pct: 96.66, gpqa_pct: 79.23 },
},
accuracyLabels: [
["gpqa_pct", "GPQA Diamond", "%"],
["gsm8k_pct", "GSM8K (1-shot)", "%"],
],
dockerImages: {
h200: "lmsysorg/sglang:dev",
b200: "lmsysorg/sglang:dev",
},
github: {
cookbookModel: "internlm/Intern-S2-Mobius",
},
playgroundFeatures: {
// ----- Card: "Attention Parallelism" -----
attention: {
knobs: [
{ id: "tp", label: "TP", values: [null, 1, 2, 4, 8] },
{ id: "cp", label: "CP", values: [null, 1, 2, 4] },
],
},
// No "MoE Parallelism" card. The routed experts do not live per-layer: all 40
// layers query 4 globally shared expert banks (`meta_mlp`, config `num_blocks: 4`
// — models/interns2_mobius.py), so EP has nothing to shard. The runtime enforces
// that: server_args._handle_model_specific_adjustments raises for this arch on
// `--ep-size != 1` (and `--pp-size != 1`), so an EP chip would emit a command
// that cannot start. `--moe-a2a-backend deepep` is out for the same reason, and
// arg_groups/overrides.py pins moe_runner_backend to triton_kernel as the only
// runner validated for the 2,560-expert bank.
// ----- Card: "Parsers" -----
parsers: {
items: [
{ id: "reasoning", label: "Reasoning Parser", flag: "--reasoning-parser qwen3" },
{ id: "toolCall", label: "Tool Call Parser", flag: "--tool-call-parser qwen3_coder" },
],
},
// ----- Card: "Speculative Decoding" ----- MTP (NEXTN) is the cook-worthy preset.
speculative: {
options: [
{ id: "current", label: "Inherited from base" },
{ id: "off", label: "Off (greedy)" },
{ id: "mtp-314", label: "MTP / NEXTN 3-1-4 (recommended)",
flags: ["--speculative-algorithm NEXTN", "--speculative-num-steps 3",
"--speculative-eagle-topk 1", "--speculative-num-draft-tokens 4"] },
{ id: "mtp-213", label: "MTP / NEXTN 2-1-3 (lighter draft)",
flags: ["--speculative-algorithm NEXTN", "--speculative-num-steps 2",
"--speculative-eagle-topk 1", "--speculative-num-draft-tokens 3"] },
],
},
},
cells: [
// ==== H200, 2 GPUs, BF16, low-latency (MTP NEXTN on) — VERIFIED ====
// GSM8K 1319 leg: 96.66 % acc / 100 % stop. Bench 8K-in / 1K-out (see
// intern-s2-mobius-benchmarks.jsx for the full 1/16/64 sweep; per conc=16
// spec reaches 18029 total tok/s vs 9358 no-spec).
{
match: { hw: "h200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },
verified: true,
env: [],
flags: [
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 2",
"--mem-fraction-static 0.8",
"--context-length 262144",
"--reasoning-parser qwen3",
"--speculative-algorithm NEXTN",
"--speculative-num-steps 3",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 4",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
// ==== H200, 2 GPUs, BF16, high-throughput (no speculative) — VERIFIED ====
// GSM8K 1319 leg: 96.82 % acc / 100 % stop. Bench 8K-in / 1K-out — the
// spec-off recipe scales cleanly to conc=256 (34786 tok/s total at
// saturation), >1.3× the spec-on peak at conc=64. See benchmarks.jsx.
{
match: { hw: "h200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" },
verified: true,
env: [],
flags: [
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 2",
"--mem-fraction-static 0.8",
"--context-length 262144",
"--reasoning-parser qwen3",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
// ==== B200, 2 GPUs, BF16, low-latency (MTP NEXTN on) — INFERRED from H200 ====
{
match: { hw: "b200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },
verified: false,
env: [],
flags: [
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 2",
"--mem-fraction-static 0.8",
"--context-length 262144",
"--reasoning-parser qwen3",
"--speculative-algorithm NEXTN",
"--speculative-num-steps 3",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 4",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
// ==== B200, 1 GPU, BF16, high-throughput — INFERRED (single 192 GB HBM fits 73 GB weights + KV) ====
{
match: { hw: "b200", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" },
verified: false,
env: [],
flags: [
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 1",
"--mem-fraction-static 0.8",
"--context-length 262144",
"--reasoning-parser qwen3",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
],
};