minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

docs: enable MiMo V2.5 MTP cookbook path (#23945)

Browse Source
This commit is contained in:
Xinyuan Tong
2026-04-28 10:22:19 -07:00
committed by GitHub
parent 3fce8f2009
commit e458a9248f
2 changed files with 90 additions and 88 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/cookbook/autoregressive/Xiaomi/MiMo-V2.5.mdx
+84 -80
View File
@@ -43,7 +43,7 @@ tag: NEW
**Key Features:** **Key Features:**
- **Hybrid Attention Architecture**: Interleaves Sliding Window Attention (SWA) and Global Attention (GA) for reduced KV cache while preserving long-context capability. - **Hybrid Attention Architecture**: Interleaves Sliding Window Attention (SWA) and Global Attention (GA) for reduced KV cache while preserving long-context capability.
- **Multi-Token Prediction (MTP)**: 3-layer MTP module accelerates decoding (329M params on V2.5; V2.5-Pro supports EAGLE speculative decoding on top of MTP). - **Multi-Token Prediction (MTP)**: 3-layer MTP module accelerates decoding. Both variants support EAGLE speculative decoding with MTP weights.
- **1M-Token Context**: Both variants support up to 1 million token context windows. - **1M-Token Context**: Both variants support up to 1 million token context windows.
- **Agentic Capabilities**: Post-training with large-scale agentic RL achieves strong performance on coding, reasoning, and tool-use benchmarks. - **Agentic Capabilities**: Post-training with large-scale agentic RL achieves strong performance on coding, reasoning, and tool-use benchmarks.
- **MiMo-V2.5 Multimodal** (V2.5 only): Native omnimodal architecture with a 729M-param ViT Vision Encoder (28 layers: 24 SWA + 4 Full) and a 261M-param Audio Transformer (24 layers: 12 SWA + 12 Full); supports image, video, and audio understanding via standard OpenAI-compatible multimodal API. - **MiMo-V2.5 Multimodal** (V2.5 only): Native omnimodal architecture with a 729M-param ViT Vision Encoder (28 layers: 24 SWA + 4 Full) and a 261M-param Audio Transformer (24 layers: 12 SWA + 12 Full); supports image, video, and audio understanding via standard OpenAI-compatible multimodal API.
@@ -84,9 +84,10 @@ import { MiMoV25Deployment } from '/src/snippets/autoregressive/mimo-v25-deploym
- EAGLE speculative decoding (3 steps, topk=1) typically yields a 2–3× decode speedup. Requires `SGLANG_ENABLE_SPEC_V2=1`; on Hopper also pass `--enable-multi-layer-eagle`. - EAGLE speculative decoding (3 steps, topk=1) typically yields a 2–3× decode speedup. Requires `SGLANG_ENABLE_SPEC_V2=1`; on Hopper also pass `--enable-multi-layer-eagle`.
**MiMo-V2.5 (310B):** **MiMo-V2.5 (310B):**
- The checkpoint has a TP=4-interleaved fused `qkv_proj`; attention-TP per DP group **must** be 4. So DP-attention is always required (`--dp = TP / 4`), and total GPUs must be a multiple of 4. A bare `--tp 8` without `--dp 2` will fail to load with `MiMoV2Omni fused qkv_proj checkpoint is TP=4-interleaved; got tp_size=8`. - The checkpoint has a TP=4-interleaved fused `qkv_proj`; attention-TP per DP group **must** be 4. Use `--dp = TP / 4`; for TP > 4 this also requires DP-attention. Total GPUs must be a multiple of 4. A bare `--tp 8` without `--dp 2` will fail to load with `MiMoV2 fused qkv_proj checkpoint is TP=4-interleaved; got attention tp_size=8`.
- Single-node deployments: H100/H200 8× GPUs (`--tp 8 --dp 2`), B200 4× GPUs (`--tp 4`, dp=1, no DP-attn flag needed), GB300 4× GPUs (`--tp 4`, single NVL4 node). FP8 quantization. - Single-node deployments: H100/H200 8× GPUs (`--tp 8 --dp 2`), B200 4× GPUs (`--tp 4`, dp=1, no DP-attn flag needed), GB300 4× GPUs (`--tp 4`, single NVL4 node). FP8 quantization.
- `--enable-dp-lm-head` and `--mm-enable-dp-encoder` are required whenever `--enable-dp-attention` is on, to keep LM head and encoder sharding consistent. - `--enable-dp-lm-head` and `--mm-enable-dp-encoder` are required whenever `--enable-dp-attention` is on, to keep LM head and encoder sharding consistent.
- EAGLE MTP uses the checkpoint's MTP weights. For H100/H200, enable `SGLANG_ENABLE_SPEC_V2=1`, `--speculative-algorithm EAGLE`, and `--enable-multi-layer-eagle`.
- **Multimodal**: Supports image, video, and audio understanding; see Section 4.3 for invocation examples. - **Multimodal**: Supports image, video, and audio understanding; see Section 4.3 for invocation examples.
**DeepEP (optional toggle, Hopper-only):** **DeepEP (optional toggle, Hopper-only):**
@@ -324,7 +325,7 @@ The user wants to know the weather in Beijing. I have a function available calle
## 5. Benchmark ## 5. Benchmark
Accuracy numbers come from `sglang.test.run_eval` (GSM8K standard 5-shot, MMMU validation split). Speed numbers come from `sglang.bench_serving` against the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset; each request is configured with 1024 input tokens and 1024 output tokens to represent a typical medium-length conversation. Accuracy numbers come from `sglang.test.run_eval` (GSM8K standard 5-shot, MMMU validation split). Speed numbers come from `sglang.bench_serving` with generated random prompts; text runs use 1024 input tokens and 1024 output tokens per request, and the image run uses 2 random 720p images per request.
### 5.1 Accuracy Benchmark ### 5.1 Accuracy Benchmark
@@ -440,13 +441,13 @@ Pending update — replace with real bench_serving output after the throughput r
- Hardware: NVIDIA H200 GPU (8×) - Hardware: NVIDIA H200 GPU (8×)
- Model: `XiaomiMiMo/MiMo-V2.5` (FP8) - Model: `XiaomiMiMo/MiMo-V2.5` (FP8)
- Tensor Parallelism: 8 (DP-attention with `--dp-size 2`) - Tensor Parallelism: 8 (DP-attention with `--dp 2`)
- Recipe: Balanced (DP-attn) - Recipe: Balanced (DP-attn + EAGLE MTP)
- sglang version: 1.1.2.dev9066 - sglang version: `0.0.0.dev1+g7d99af439` (`lmsysorg/sglang:dev-mimo-v2.5`)
#### 5.3.1 Latency-Sensitive Benchmark #### 5.3.1 Latency-Sensitive Benchmark
- **Model Deployment Command:** see the [command panel above](#3-model-deployment). - **Model Deployment Command:** select MiMo-V2.5, H200, and EAGLE MTP in the [command panel above](#3-model-deployment).
- Benchmark Command: - Benchmark Command:
```shell Command ```shell Command
@@ -469,43 +470,44 @@ Backend: sglang
Traffic request rate: inf Traffic request rate: inf
Max request concurrency: 1 Max request concurrency: 1
Successful requests: 10 Successful requests: 10
Benchmark duration (s): 26.41 Benchmark duration (s): 14.72
Total input tokens: 1997 Total input tokens: 1997
Total input text tokens: 1997 Total input text tokens: 1997
Total generated tokens: 2798 Total generated tokens: 2798
Total generated tokens (retokenized): 2669 Total generated tokens (retokenized): 2697
Request throughput (req/s): 0.38 Request throughput (req/s): 0.68
Input token throughput (tok/s): 75.60 Input token throughput (tok/s): 135.67
Output token throughput (tok/s): 105.93 Output token throughput (tok/s): 190.09
Peak output token throughput (tok/s): 110.00 Peak output token throughput (tok/s): 245.00
Peak concurrent requests: 3 Peak concurrent requests: 3
Total token throughput (tok/s): 181.53 Total token throughput (tok/s): 325.77
Concurrency: 1.00 Concurrency: 1.00
Accept length: 3.08
----------------End-to-End Latency---------------- ----------------End-to-End Latency----------------
Mean E2E Latency (ms): 2639.75 Mean E2E Latency (ms): 1469.98
Median E2E Latency (ms): 2928.14 Median E2E Latency (ms): 1652.84
P90 E2E Latency (ms): 4107.62 P90 E2E Latency (ms): 2210.80
P99 E2E Latency (ms): 4830.83 P99 E2E Latency (ms): 2823.86
---------------Time to First Token---------------- ---------------Time to First Token----------------
Mean TTFT (ms): 73.94 Mean TTFT (ms): 143.89
Median TTFT (ms): 74.09 Median TTFT (ms): 99.25
P99 TTFT (ms): 79.90 P99 TTFT (ms): 481.01
-----Time per Output Token (excl. 1st token)------ -----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 9.19 Mean TPOT (ms): 4.87
Median TPOT (ms): 9.21 Median TPOT (ms): 4.30
P99 TPOT (ms): 9.24 P99 TPOT (ms): 6.64
---------------Inter-Token Latency---------------- ---------------Inter-Token Latency----------------
Mean ITL (ms): 9.20 Mean ITL (ms): 4.76
Median ITL (ms): 9.21 Median ITL (ms): 3.46
P95 ITL (ms): 9.31 P95 ITL (ms): 13.52
P99 ITL (ms): 9.43 P99 ITL (ms): 13.84
Max ITL (ms): 16.43 Max ITL (ms): 74.37
================================================== ==================================================
``` ```
#### 5.3.2 Throughput-Sensitive Benchmark #### 5.3.2 Throughput-Sensitive Benchmark
- **Model Deployment Command:** see the [command panel above](#3-model-deployment). - **Model Deployment Command:** select MiMo-V2.5, H200, and EAGLE MTP in the [command panel above](#3-model-deployment).
- Benchmark Command: - Benchmark Command:
```shell Command ```shell Command
@@ -528,43 +530,44 @@ Backend: sglang
Traffic request rate: inf Traffic request rate: inf
Max request concurrency: 100 Max request concurrency: 100
Successful requests: 1000 Successful requests: 1000
Benchmark duration (s): 87.23 Benchmark duration (s): 93.41
Total input tokens: 302118 Total input tokens: 302118
Total input text tokens: 302118 Total input text tokens: 302118
Total generated tokens: 195775 Total generated tokens: 195775
Total generated tokens (retokenized): 190470 Total generated tokens (retokenized): 188139
Request throughput (req/s): 11.46 Request throughput (req/s): 10.71
Input token throughput (tok/s): 3463.61 Input token throughput (tok/s): 3234.48
Output token throughput (tok/s): 2244.45 Output token throughput (tok/s): 2095.97
Peak output token throughput (tok/s): 4274.00 Peak output token throughput (tok/s): 3019.00
Peak concurrent requests: 122 Peak concurrent requests: 121
Total token throughput (tok/s): 5708.05 Total token throughput (tok/s): 5330.45
Concurrency: 87.80 Concurrency: 91.04
Accept length: 2.95
----------------End-to-End Latency---------------- ----------------End-to-End Latency----------------
Mean E2E Latency (ms): 7658.29 Mean E2E Latency (ms): 8503.45
Median E2E Latency (ms): 5195.82 Median E2E Latency (ms): 7491.96
P90 E2E Latency (ms): 18382.07 P90 E2E Latency (ms): 13706.99
P99 E2E Latency (ms): 32849.04 P99 E2E Latency (ms): 20474.33
---------------Time to First Token---------------- ---------------Time to First Token----------------
Mean TTFT (ms): 188.32 Mean TTFT (ms): 4399.20
Median TTFT (ms): 138.69 Median TTFT (ms): 4333.35
P99 TTFT (ms): 746.89 P99 TTFT (ms): 8004.81
-----Time per Output Token (excl. 1st token)------ -----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 40.13 Mean TPOT (ms): 58.23
Median TPOT (ms): 40.68 Median TPOT (ms): 21.78
P99 TPOT (ms): 80.38 P99 TPOT (ms): 747.79
---------------Inter-Token Latency---------------- ---------------Inter-Token Latency----------------
Mean ITL (ms): 38.42 Mean ITL (ms): 20.06
Median ITL (ms): 21.63 Median ITL (ms): 15.28
P95 ITL (ms): 82.64 P95 ITL (ms): 48.36
P99 ITL (ms): 119.66 P99 ITL (ms): 96.99
Max ITL (ms): 726.36 Max ITL (ms): 969.61
================================================== ==================================================
``` ```
#### 5.3.3 Multimodal (Image) Benchmark #### 5.3.3 Multimodal (Image) Benchmark
- **Model Deployment Command:** see the [command panel above](#3-model-deployment). - **Model Deployment Command:** select MiMo-V2.5, H200, and EAGLE MTP in the [command panel above](#3-model-deployment).
- Benchmark Command: - Benchmark Command:
```shell Command ```shell Command
@@ -590,37 +593,38 @@ Backend: sglang-oai-chat
Traffic request rate: inf Traffic request rate: inf
Max request concurrency: 1 Max request concurrency: 1
Successful requests: 10 Successful requests: 10
Benchmark duration (s): 41.89 Benchmark duration (s): 25.73
Total input tokens: 18514 Total input tokens: 661
Total input text tokens: 874 Total input text tokens: 631
Total input vision tokens: 17640 Total input vision tokens: 30
Total generated tokens: 4220 Total generated tokens: 4220
Total generated tokens (retokenized): 1478 Total generated tokens (retokenized): 0
Request throughput (req/s): 0.24 Request throughput (req/s): 0.39
Input token throughput (tok/s): 442.01 Input token throughput (tok/s): 25.69
Output token throughput (tok/s): 100.75 Output token throughput (tok/s): 164.03
Peak output token throughput (tok/s): 107.00 Peak output token throughput (tok/s): 1.00
Peak concurrent requests: 2 Peak concurrent requests: 2
Total token throughput (tok/s): 542.76 Total token throughput (tok/s): 189.73
Concurrency: 1.00 Concurrency: 1.00
Accept length: 2.94
----------------End-to-End Latency---------------- ----------------End-to-End Latency----------------
Mean E2E Latency (ms): 4186.79 Mean E2E Latency (ms): 2570.74
Median E2E Latency (ms): 3366.20 Median E2E Latency (ms): 2411.92
P90 E2E Latency (ms): 7545.54 P90 E2E Latency (ms): 3711.62
P99 E2E Latency (ms): 9180.85 P99 E2E Latency (ms): 4949.74
---------------Time to First Token---------------- ---------------Time to First Token----------------
Mean TTFT (ms): 1284.90 Mean TTFT (ms): 0.00
Median TTFT (ms): 622.81 Median TTFT (ms): 0.00
P99 TTFT (ms): 5030.79 P99 TTFT (ms): 0.00
-----Time per Output Token (excl. 1st token)------ -----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 7.36 Mean TPOT (ms): 7.31
Median TPOT (ms): 8.45 Median TPOT (ms): 6.17
P99 TPOT (ms): 10.94 P99 TPOT (ms): 17.18
---------------Inter-Token Latency---------------- ---------------Inter-Token Latency----------------
Mean ITL (ms): 9.54 Mean ITL (ms): 0.00
Median ITL (ms): 9.45 Median ITL (ms): 0.00
P95 ITL (ms): 9.58 P95 ITL (ms): 0.00
P99 ITL (ms): 11.12 P99 ITL (ms): 0.00
Max ITL (ms): 37.67 Max ITL (ms): 0.00
================================================== ==================================================
``` ```
docs_new/src/snippets/autoregressive/mimo-v25-deployment.jsx
+6 -8
View File
@@ -15,7 +15,7 @@ export const MiMoV25Deployment = () => {
// GB300 → tp=4, dp=1, single-node, FP8 // GB300 → tp=4, dp=1, single-node, FP8
// //
// Optional toggles: // Optional toggles:
// EAGLE MTP — Pro only. Adds --speculative-* flags + SGLANG_ENABLE_SPEC_V2=1. // EAGLE MTP — adds --speculative-* flags + SGLANG_ENABLE_SPEC_V2=1.
// DeepEP — Hopper only (Blackwell uses flashinfer_trtllm). Adds // DeepEP — Hopper only (Blackwell uses flashinfer_trtllm). Adds
// --moe-a2a-backend deepep + --moe-dense-tp-size 1 // --moe-a2a-backend deepep + --moe-dense-tp-size 1
// (and --ep on Pro) + SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256. // (and --ep on Pro) + SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256.
@@ -44,7 +44,7 @@ export const MiMoV25Deployment = () => {
name: "eagleMtp", name: "eagleMtp",
title: "EAGLE MTP", title: "EAGLE MTP",
items: [ items: [
{ id: "enabled", label: "Enabled", default: true, subtitle: "Pro only" }, { id: "enabled", label: "Enabled", default: true, subtitle: "EAGLE" },
{ id: "disabled", label: "Disabled", default: false }, { id: "disabled", label: "Disabled", default: false },
], ],
}, },
@@ -68,8 +68,8 @@ export const MiMoV25Deployment = () => {
name: "deepep", name: "deepep",
title: "DeepEP", title: "DeepEP",
items: [ items: [
{ id: "disabled", label: "Disabled", default: true, subtitle: "default" },
{ id: "enabled", label: "Enabled", default: false, subtitle: "needs deep_ep" }, { id: "enabled", label: "Enabled", default: false, subtitle: "needs deep_ep" },
{ id: "disabled", label: "Disabled", default: true, subtitle: "default" },
], ],
}, },
reasoningParser: { reasoningParser: {
@@ -126,8 +126,6 @@ export const MiMoV25Deployment = () => {
const blackwell = spec ? spec.blackwell : false; const blackwell = spec ? spec.blackwell : false;
const c = {}; const c = {};
if (!isPro) { if (!isPro) {
// V2.5 has no MTP module, so EAGLE MTP is genuinely unavailable.
c.eagleMtp = { force: "disabled", reason: "EAGLE MTP applies to V2.5-Pro only." };
// V2.5 checkpoint is TP=4-interleaved; tp/dp must equal 4. With dp>1 we // V2.5 checkpoint is TP=4-interleaved; tp/dp must equal 4. With dp>1 we
// must use DP-attention; with dp=1 it must be off (single attention group). // must use DP-attention; with dp=1 it must be off (single attention group).
if (spec && spec.dp > 1) { if (spec && spec.dp > 1) {
@@ -209,9 +207,9 @@ export const MiMoV25Deployment = () => {
const spec = HW_VARIANT_SPEC[specKey]; const spec = HW_VARIANT_SPEC[specKey];
const { slug, tp, multinode, nnodes, blackwell } = spec; const { slug, tp, multinode, nnodes, blackwell } = spec;
const isPro = modelVariant === "pro"; const isPro = modelVariant === "pro";
// Toggles. EAGLE MTP / EP / DeepEP / DP-attn are gated by hardware + variant // Toggles. EP / DeepEP / DP-attn are gated by hardware + variant
// through computeConstraints; here we just read the (already-snapped) value. // through computeConstraints; here we just read the (already-snapped) value.
const useMtp = isPro && eagleMtp === "enabled"; const useMtp = eagleMtp === "enabled";
const useDeepep = !blackwell && deepep === "enabled"; const useDeepep = !blackwell && deepep === "enabled";
const useEp = isPro && !blackwell && expertParallelism === "enabled"; const useEp = isPro && !blackwell && expertParallelism === "enabled";
const useDpAttn = dpAttention === "enabled"; const useDpAttn = dpAttention === "enabled";
@@ -278,7 +276,7 @@ export const MiMoV25Deployment = () => {
} }
if (useMtp) { if (useMtp) {
flags.push(" --speculative-algo EAGLE"); flags.push(" --speculative-algorithm EAGLE");
flags.push(" --speculative-num-steps 3"); flags.push(" --speculative-num-steps 3");
flags.push(" --speculative-eagle-topk 1"); flags.push(" --speculative-eagle-topk 1");
flags.push(" --speculative-num-draft-tokens 4"); flags.push(" --speculative-num-draft-tokens 4");