minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[Docs] Rename docs_new/ to docs/ (#32123)

Browse Source 
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
zijiexia
2026-08-03 16:51:00 -07:00
committed by GitHub
co-authored by Claude Opus 4.8
parent c949e91f18
commit b819d2fb5b
491 changed files with 122 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/MiniMax/MiniMax-M2.5.mdx
+1040
View File
File diff suppressed because it is too large Load Diff
docs/cookbook/autoregressive/MiniMax/MiniMax-M2.7.mdx
+730
View File
@@ -0,0 +1,730 @@
---
title: MiniMax-M2.7
metatags:
description: "Deploy MiniMax-M2.7 with SGLang on NVIDIA GPUs, AMD GPUs, and Intel Xeon CPUs — model self-evolution, professional software engineering, and native agent teams."
---
## 1. Model Introduction
[MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7) is MiniMax's first model deeply participating in its own evolution. Built for real-world productivity, M2.7 excels at building complex agent harnesses and completing highly elaborate productivity tasks, leveraging Agent Teams, complex Skills, and dynamic tool search.
Key highlights:
- **Model Self-Evolution**: During development, M2.7 updates its own memory, builds complex skills for RL experiments, and improves its own learning process. An internal version autonomously optimized a programming scaffold over 100+ rounds, achieving a **30% performance improvement**. On MLE Bench Lite, M2.7 achieved a **66.6% medal rate**.
- **Professional Software Engineering**: Delivers outstanding real-world programming capabilities. On SWE-Pro, M2.7 achieved **56.22%**, with strong results on SWE Multilingual (76.5) and Multi SWE Bench (52.7). On Terminal Bench 2 (57.0%) and NL2Repo (39.8%), M2.7 demonstrates deep understanding of complex engineering systems.
- **Professional Work**: Achieved an ELO score of **1495** on GDPval-AA (highest among open-source models). On Toolathon, M2.7 reached **46.3%** accuracy (global top tier).
- **Native Agent Teams**: Supports multi-agent collaboration with stable role identity and autonomous decision-making.
For more details, see the [official MiniMax-M2.7 blog post](https://www.minimax.io/news/minimax-m27-en).
**License**: [Modified-MIT (MiniMax Model License)](https://github.com/MiniMax-AI/MiniMax-M2.7/blob/main/LICENSE)
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
**Docker Images by Hardware Platform:**
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Hardware Platform</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Docker Image</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>NVIDIA A100 / H100 / H200 / B200</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.10.post1`</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>NVIDIA B300 / GB300</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.10.post1-cu130`</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AMD MI300X / MI325X</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.10.post1-rocm720-mi30x`</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AMD MI355X</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.10.post1-rocm720-mi35x`</td>
</tr>
</tbody>
</table>
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, deployment strategy, and feature capabilities.
import { MiniMaxM27Deployment } from '/src/snippets/autoregressive/minimax-m27-deployment.jsx'
<MiniMaxM27Deployment />
### 3.2 Configuration Tips
**Key Parameters:**
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Recommended Value</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--tool-call-parser`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Tool call parser for function calling support</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`minimax-m2`</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Reasoning parser for thinking mode</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`minimax-append-think`</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--trust-remote-code`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Required for MiniMax model loading</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Always enabled</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--mem-fraction-static`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Static memory fraction for KV cache</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`0.85`</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--tp`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Tensor parallelism size</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`2` / `4` / `8` depending on hardware</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--ep`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Expert parallelism size</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`8` (NVIDIA 8-GPU) or EP=TP (AMD)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--kv-cache-dtype`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>KV cache data type (AMD only)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`fp8_e4m3`</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--attention-backend`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Attention backend (AMD only)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`triton`</td>
</tr>
</tbody>
</table>
**Hardware Requirements: NVIDIA**
- **4-GPU deployment**: Requires 4× high-memory GPUs (e.g., H200, B200, A100, H100) with TP=4
- **8-GPU deployment**: Requires 8× GPUs (e.g., H200, B200, A100, H100) with TP=8 and EP=8
**Hardware Requirements: NVIDIA GB300**
- **2-GPU deployment**: GB300 (275GB per die) can host the model with TP=2
- **4-GPU deployment**: Maximum single-node TP for GB300, recommended for higher throughput
**Hardware Requirements: AMD**
- **2-GPU deployment**: Requires 2× high-memory GPUs (e.g., MI300X, MI325X, MI355X) with TP=2, EP=2
- **4-GPU deployment**: Requires 4× GPUs (e.g., MI300X, MI325X, MI355X) with TP=4, EP=4
- **8-GPU deployment**: Requires 8× GPUs (e.g., MI300X, MI325X, MI355X) with TP=8, EP=8
**Hardware Requirements: Intel Xeon CPU**
- It is recommended to run the model service on a Granite Rapids (GNR) AP 2-Socket server.
- For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
**Deployment Command:**
```bash Command
sglang serve \
--model-path MiniMaxAI/MiniMax-M2.7 \
--tp 4 \
--tool-call-parser minimax-m2 \
--reasoning-parser minimax-append-think \
--trust-remote-code \
--mem-fraction-static 0.85
```
**Testing Deployment:**
After startup, you can test the SGLang OpenAI-compatible API with the following command:
```bash Command
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "MiniMaxAI/MiniMax-M2.7",
"messages": [
{"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]},
{"role": "user", "content": [{"type": "text", "text": "Who won the world series in 2020?"}]}
]
}'
```
**Simple Completion Example:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="MiniMaxAI/MiniMax-M2.7",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Who won the world series in 2020?"}
],
max_tokens=1024
)
print(response.choices[0].message.content)
```
**Example Output**:
```text Output
<think>The user asks: "Who won the World Series in 2020?" That's a simple factual question. The answer: the Los Angeles Dodgers won the 2020 MLB World Series, defeating the Tampa Bay Rays. So answer accordingly.
We must be mindful of policy: it's a factual question about sports. It's allowed. Provide answer with brief context.
We should answer concisely.
Hence final answer: The Los Angeles Dodgers won the 2020 World Series, defeating the Tampa Bay Rays in six games (best-of-seven series). Possibly mention it was played at a neutral site due to COVID-19, at Globe Life Field in Arlington, Texas.
We must avoid disallowed content, no issue.
Thus final.
</think>
The **Los Angeles Dodgers** won the 2020 World Series. They defeated the **Tampa Bay Rays** in six games (4‑2) in a best‑of‑seven series that was played at Globe Life Field in Arlington, Texas, under the MLB bubble‑like arrangements for the COVID‑19 pandemic.
```
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
MiniMax-M2.7 supports Thinking mode. Enable the reasoning parser during deployment to separate the thinking and the content sections:
```bash Command
sglang serve \
--model-path MiniMaxAI/MiniMax-M2.7 \
--tp 4 \
--reasoning-parser minimax-append-think \
--trust-remote-code \
--mem-fraction-static 0.85
```
**Streaming with Thinking Process**
With `minimax-append-think`, the thinking content is wrapped in `<think>...</think>` tags within the `content` field. You can parse these tags on the client side to separate the thinking and content sections:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="MiniMaxAI/MiniMax-M2.7",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
max_tokens=2048,
stream=True
)
# Process the stream, separating <think>...</think> from content
in_think = False
think_printed_header = False
content_printed_header = False
buffer = ""
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if delta.content:
buffer += delta.content
while buffer:
if in_think:
# Look for closing </think> tag
end_idx = buffer.find("</think>")
if end_idx != -1:
print(buffer[:end_idx], end="", flush=True)
buffer = buffer[end_idx + len("</think>"):]
in_think = False
else:
# Still in thinking, print what we have
print(buffer, end="", flush=True)
buffer = ""
else:
# Look for opening <think> tag
start_idx = buffer.find("<think>")
if start_idx != -1:
# Print any content before <think>
before = buffer[:start_idx]
if before:
if not content_printed_header:
print("=============== Content =================", flush=True)
content_printed_header = True
print(before, end="", flush=True)
buffer = buffer[start_idx + len("<think>"):]
in_think = True
if not think_printed_header:
print("=============== Thinking =================", flush=True)
think_printed_header = True
else:
# No <think> tag, print as content
if not content_printed_header and think_printed_header:
print("\n=============== Content =================", flush=True)
content_printed_header = True
print(buffer, end="", flush=True)
buffer = ""
print()
```
**Output Example:**
```text Output
=============== Thinking =================
The user asks: "Solve this problem step by step: What is 15% of 240?" Straightforward. Provide solution: 15% = 15/100 = 0.15. Multiply 240 * 0.15 = 36. Show steps. So answer: 36. Provide explanation.
But also ensure we follow any policy? No issues. Just straightforward.
I'll provide a step-by-step solution.
Also could show fraction: 15% = 15/100 = 3/20, multiply 240 * 3/20 = (240/20)*3 = 12*3 = 36.
Yes. Provide final answer. Also show verification: 10% of 240 is 24, 5% is 12, total 36.
All good.
=============== Content =================
**Step‑by‑step solution**
1. **Convert the percent to a decimal (or a fraction).**
15% = 15/100 = 0.15 = 3/20
2. **Multiply the original number (240) by this decimal/fraction.**
Using the decimal:
240 × 0.15 = 36
Or using the fraction:
240 × 3/20 = (240/20) × 3 = 12 × 3 = 36
3. **Result:**
15% of 240 = **36**
*Check:*
- 10% of 240 = 24
- 5% of 240 = 12
- Adding them: 24 + 12 = 36, which matches the calculation.
```
**Note:** The `minimax-append-think` reasoning parser embeds the thinking process in `<think>...</think>` tags within the `content` field. The code above parses these tags in real-time to display thinking and content separately.
#### 4.2.2 Tool Calling
MiniMax-M2.7 supports tool calling capabilities. Enable the tool call parser:
```bash Command
sglang serve \
--model-path MiniMaxAI/MiniMax-M2.7 \
--tp 4 \
--tool-call-parser minimax-m2 \
--reasoning-parser minimax-append-think \
--trust-remote-code \
--mem-fraction-static 0.85
```
**Python Example:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Non-streaming request
response = client.chat.completions.create(
model="MiniMaxAI/MiniMax-M2.7",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools
)
message = response.choices[0].message
# Check for tool calls
if message.tool_calls:
for tool_call in message.tool_calls:
print(f"Tool Call: {tool_call.function.name}")
print(f" Arguments: {tool_call.function.arguments}")
else:
print(message.content)
```
**Output Example**:
```text Output
Tool Call: get_weather
Arguments: {"location": "Beijing"}
```
**Handling Tool Call Results:**
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="MiniMaxAI/MiniMax-M2.7",
messages=messages
)
print(final_response.choices[0].message.content)
```
**Output Example:**
```text Output
The weather in Beijing is currently 22°C and sunny.
```
## 5. Benchmark
This section uses **industry-standard configurations** for comparable benchmark results.
**Test Environment**:
- Hardware: 2× NVIDIA GB300 (275GB per die)
- Docker Image: `lmsysorg/sglang:v0.5.10.post1-cu130`
- Model: MiniMax-M2.7 (FP8)
- Tensor Parallelism: 2
- SGLang version: 0.5.10.post1
### 5.1 Accuracy Benchmark
**Evaluation Tool**: [NVIDIA NeMo-Skills](https://github.com/NVIDIA-NeMo/Skills)
**Evaluation Settings**: temperature=0.6, top_p=0.95, 8 seeds, max_tokens=120,000, `parse_reasoning=True`
#### 5.1.1 GPQA Diamond
- Dataset: [GPQA Diamond](https://huggingface.co/datasets/Idavidrein/gpqa) (198 questions)
- Prompt: `eval/aai/mcq-4choices` (4-choice multiple choice, matching [Artificial Analysis methodology](https://artificialanalysis.ai/methodology/intelligence-benchmarking))
- Evaluation command:
```bash Command
ns prepare_data gpqa
ns eval \
--cluster=local \
--server_type=openai \
--model=MiniMaxAI/MiniMax-M2.7 \
--server_address=http://localhost:30000/v1 \
--output_dir=./m2.7-eval/ \
--benchmarks=gpqa:8 \
++prompt_config=eval/aai/mcq-4choices \
++inference.tokens_to_generate=120000 \
++inference.temperature=0.6 \
++inference.top_p=0.95 \
++parse_reasoning=True
```
- Test Results:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Evaluation Mode</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Accuracy</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>No Answer</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@1 (avg-of-8)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>84.91%</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>3.54%</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**majority@8**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>**88.89%**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.00%</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@8</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>96.46%</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.00%</td>
</tr>
</tbody>
</table>
#### 5.1.2 AIME 2025
- Dataset: AIME 2025 (30 problems)
- Prompt: `generic/math` (boxed answer format)
- Evaluation command:
```bash Command
ns prepare_data aime25
ns eval \
--cluster=local \
--server_type=openai \
--model=MiniMaxAI/MiniMax-M2.7 \
--server_address=http://localhost:30000/v1 \
--output_dir=./m2.7-eval/ \
--benchmarks=aime25:8 \
++inference.tokens_to_generate=120000 \
++inference.temperature=0.6 \
++inference.top_p=0.95 \
++parse_reasoning=True
```
- Test Results:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Evaluation Mode</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Accuracy</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>No Answer</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@1 (avg-of-8)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>92.50% ± 5.56%</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>2.92%</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**majority@8**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>**97.08%**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.00%</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@8</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>100.00%</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.00%</td>
</tr>
</tbody>
</table>
#### 5.1.3 MMLU-Pro
- Dataset: [MMLU-Pro](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) (12,032 questions, 10-choice)
- Prompt: `eval/aai/mcq-10choices` (10-choice multiple choice)
- Evaluation command:
```bash Command
ns prepare_data mmlu-pro
ns eval \
--cluster=local \
--server_type=openai \
--model=MiniMaxAI/MiniMax-M2.7 \
--server_address=http://localhost:30000/v1 \
--output_dir=./m2.7-eval/ \
--benchmarks=mmlu-pro \
++prompt_config=eval/aai/mcq-10choices \
++inference.tokens_to_generate=32768 \
++inference.temperature=0.0 \
++parse_reasoning=True
```
- Test Results:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Evaluation Mode</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Accuracy</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>No Answer</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@1 (greedy)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>69.41%</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>18.75%</td>
</tr>
</tbody>
</table>
> **Note**: The high no-answer rate is due to the 32K token limit being insufficient for M2.7's extended thinking on some questions. A rerun with 120K tokens is expected to improve accuracy significantly.
#### 5.1.4 GSM8K Benchmark
- Benchmark Method: 8-shot Chain-of-Thought, evaluated via OpenAI-compatible API
- Test Results:
```text Output
GSM8K Results (8-shot CoT)
Model: MiniMaxAI/MiniMax-M2.7
Total: 1319
Correct: 1218
Accuracy: 92.34%
```
### 5.2 Speed Benchmark
#### 5.2.1 Low Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model MiniMaxAI/MiniMax-M2.7 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 34.33
Total input tokens: 6101
Total generated tokens: 4220
Request throughput (req/s): 0.29
Input token throughput (tok/s): 177.71
Output token throughput (tok/s): 122.92
Total token throughput (tok/s): 300.63
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 3431.21
Median E2E Latency (ms): 2742.57
---------------Time to First Token----------------
Mean TTFT (ms): 50.28
Median TTFT (ms): 53.85
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 8.02
Median TPOT (ms): 8.01
---------------Inter-Token Latency----------------
Mean ITL (ms): 8.03
Median ITL (ms): 8.02
==================================================
```
#### 5.2.2 High Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model MiniMaxAI/MiniMax-M2.7 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 100.20
Total input tokens: 249831
Total generated tokens: 252662
Request throughput (req/s): 4.99
Input token throughput (tok/s): 2493.41
Output token throughput (tok/s): 2521.66
Total token throughput (tok/s): 5015.07
Concurrency: 90.19
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 18072.69
Median E2E Latency (ms): 17761.84
---------------Time to First Token----------------
Mean TTFT (ms): 247.94
Median TTFT (ms): 92.05
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 35.75
Median TPOT (ms): 36.67
---------------Inter-Token Latency----------------
Mean ITL (ms): 35.34
Median ITL (ms): 30.55
==================================================
```
docs/cookbook/autoregressive/MiniMax/MiniMax-M2.mdx
+613
View File
@@ -0,0 +1,613 @@
---
title: MiniMax-M2
metatags:
description: "Deploy MiniMax-M2 with SGLang - community contribution guide for MiniMax M2 model deployment."
---
import { MiniMaxM2Deployment } from '/src/snippets/autoregressive/minimax-m2-deployment.jsx';
## 1. Model Introduction
[MiniMax-M2](https://huggingface.co/MiniMaxAI/MiniMax-M2) is a compact, fast, and cost-effective MoE model (230 billion total parameters with 10 billion active parameters) built for elite performance in coding and agentic tasks, all while maintaining powerful general intelligence.
This generation delivers comprehensive upgrades across the board:
- **Superior Intelligence**: MiniMax-M2 demonstrates highly competitive general intelligence across mathematics, science, instruction following, coding, and agentic tool use in [Artificial Analysis](https://artificialanalysis.ai/). Its composite score ranks #1 among open-source models globally.
- **Advanced Coding**: Engineered for end-to-end developer workflows, MiniMax-M2 excels at multi-file edits, coding-run-fix loops, and test-validated repairs. Strong performance on Terminal-Bench and (Multi-)SWE-Bench–style tasks demonstrates practical effectiveness in terminals, IDEs, and CI across languages.
- **Agent Performance**: MiniMax-M2 plans and executes complex, long-horizon toolchains across shell, browser, retrieval, and code runners. In BrowseComp-style evaluations, it consistently locates hard-to-surface sources, maintains evidence traceable, and gracefully recovers from flaky steps.
- **Efficient Design**: With 10 billion activated parameters (230 billion in total), MiniMax-M2 delivers lower latency, lower cost, and higher throughput for interactive agents and batched sampling—perfectly aligned with the shift toward highly deployable models that still shine on coding and agentic tasks.
For more details, please refer to the [official Minimax GitHub Repository](https://github.com/MiniMax-AI).
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. The AMD environment is currently available in SGLang via Docker image install.
### 2.1 AMD Docker
#### 2.1.1 Launch docker
```shell Command
docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x
```
```shell Command
docker run -d -it --ipc=host --network=host --privileged \
--cap-add=CAP_SYS_ADMIN \
--device=/dev/kfd --device=/dev/dri --device=/dev/mem \
--group-add video --cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
-v /:/work \
-e SHELL=/bin/bash \
--name Minimax \
lmsysorg/sglang:v0.5.9-rocm720-mi30x \
/bin/bash
```
#### 2.1.2 Make modifications inside the docker
```shell Command
mv /sgl-workspace/sglang/python/sglang/srt/models/transformers.py \
/sgl-workspace/sglang/python/sglang/srt/models/hf_transformers_model.py
```
#### 2.1.3 Fix torch compile
Comment out the following line: @torch.compile(dynamic=True, backend=get_compiler_backend()) in /sgl-workspace/sglang/python/sglang/srt/models/minimax_m2.py
```shell Command
#@torch.compile(dynamic=True, backend=get_compiler_backend())
```
## 3. Model Deployment
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
<MiniMaxM2Deployment />
#### 3.1.1 NVIDIA GPU Deployment
The interactive command generator above covers AMD deployments. For NVIDIA GPUs (H100/H200/B200), use these explicit commands:
**4-GPU deployment (up to 400K context):**
```bash Command
python -m sglang.launch_server \
--model-path MiniMaxAI/MiniMax-M2 \
--tp-size 4 \
--tool-call-parser minimax-m2 \
--reasoning-parser minimax-append-think \
--host 0.0.0.0 \
--trust-remote-code \
--port 30000 \
--mem-fraction-static 0.85
```
**8-GPU deployment (up to 3M context):**
```bash Command
python -m sglang.launch_server \
--model-path MiniMaxAI/MiniMax-M2 \
--tp-size 8 \
--ep-size 8 \
--tool-call-parser minimax-m2 \
--reasoning-parser minimax-append-think \
--host 0.0.0.0 \
--trust-remote-code \
--port 30000 \
--mem-fraction-static 0.85
```
### 3.2 System Requirements
Recommended configurations — actual requirements depend on workload:
<table style={{width: "100%", borderCollapse: "collapse"}}>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>GPUs</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Context Length Support</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>4× 96 GB GPUs</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Up to 400K tokens</td>
</tr>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8× 144 GB GPUs</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Up to 3M tokens</td>
</tr>
</tbody>
</table>
### 3.3 Testing Deployment
After the server starts, verify with:
```shell Command
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "MiniMaxAI/MiniMax-M2",
"messages": [
{"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]},
{"role": "user", "content": [{"type": "text", "text": "Who won the world series in 2020?"}]}
]
}'
```
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
Server Command:
```shell Command
sglang serve \
--model-path MiniMaxAI/MiniMax-M2 \
--tp-size 4 \
--reasoning-parser minimax-append-think \
--trust-remote-code \
--mem-fraction-static 0.85
```
Test Code:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="MiniMaxAI/MiniMax-M2",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.6,
max_tokens=2048,
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
Output Example:
```text Output
<think>First, the user asks: "What is 15% of 240?" This is a straightforward percentage calculation. I need to solve it step by step as per the instruction.
The problem is: What is 15% of 240?
To find a percentage of a number, I multiply the number by the percentage divided by 100. So, 15% is 15/100, which simplifies to 0.15.
Therefore, 15% of 240 is 240 times 0.15.
Let me calculate that: 240 × 0.15.
I can break it down: 240 × 0.15 = 240 × (15/100) = (240 × 15) / 100.
Now, 240 × 15. 200 × 15 = 3000, and 40 × 15 = 600, so total 3000 + 600 = 3600.
Then, divide by 100: 3600 / 100 = 36.
So, 15% of 240 is 36.
I should confirm this with another method. For example, 10% of 240 is 24, and 5% is half of that, which is 12. Then 15% is 10% + 5% = 24 + 12 = 36. Same answer.
Or, using fractions: 15% = 3/20, so 240 × 3/20 = (240 / 20) × 3 = 12 × 3 = 36.
All methods confirm it's 36.
The user said "solve this problem step by step," so I should present the steps clearly.
Step 1: Understand that "15% of 240" means 15 per hundred of 240.
Step 2: Convert percentage to decimal: 15% = 15/100 = 0.15.
Step 3: Multiply the number by the decimal: 240 × 0.15.
Step 4: Calculate the multiplication: 240 × 0.15 = 36.
Step 5: Therefore, 15% of 240 is 36.
I should also mention that percentage means per hundred, so it's straightforward.
Finally, I need to box the answer as per the instruction.
So, the final answer is 36.
</think>
To find 15% of 240, follow these steps:
1. **Understand the percentage**: "15%" means 15 per hundred, or 15/100.
2. **Convert to a decimal**: 15/100 = 0.15.
3. **Multiply by the number**: 240 × 0.15.
4. **Calculate the result**:
- 240 × 0.15 = 36.
Alternatively, you can break it down:
- 10% of 240 is 24 (since 240 ÷ 10 = 24).
- 5% of 240 is half of 10%, which is 12.
- Therefore, 15% is 10% + 5% = 24 + 12 = 36.
Both methods confirm the result.
**Answer**: 36
```
### 4.2.2 Tool Calling
Server Command:
```shell Command
sglang serve \
--model-path MiniMaxAI/MiniMax-M2 \
--tp-size 4 \
--tool-call-parser minimax-m2 \
--trust-remote-code \
--mem-fraction-static 0.85
```
Test Code:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="MiniMaxAI/MiniMax-M2",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Accumulate tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================\n", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {
'name': None,
'arguments': ''
}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]['name'] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
# Print content
if delta.content:
print(delta.content, end="", flush=True)
# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"🔧 Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
print()
```
Output Example:
```text Output
Alright, the user is asking about the weather in Beijing. This is a straightforward request that I can help with using the get_weather tool that's available to me.
Let me think about what I need to do here. The user wants to know the current weather conditions in Beijing, which is the capital city of China. To provide this information, I need to use the get_weather tool that's been provided to me.
Looking at the tool's parameters, I can see it requires:
1. location - which is required and should be a string representing the city name
2. unit - which is optional and can be either "celsius" or "fahrenheit"
For the location parameter, I'll use "Beijing" since that's what the user asked about.
For the unit parameter, the user didn't specify their preference between celsius and fahrenheit. Since Beijing is in China, which primarily uses celsius, and celsius is the more standard unit internationally, I'll default to celsius. If the user wants the temperature in fahrenheit instead, they can ask in a follow-up message and I can provide that information.
So I need to make a tool call to get_weather with the following parameters:
- location: "Beijing"
- unit: "celsius"
This should return the current weather information for Beijing, which I can then share with the user. I'll format my response using the required XML tags for tool calls as specified in my instructions.
</think>
🔧 Tool Call: get_weather
Arguments: {"location": "Beijing", "unit": "celsius"}
```
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment**:
- Hardware: AMD MI300X GPU(4x)
- Model: MiniMax-M2
- Tensor Parallelism: 4
- sglang version: 0.5.7
**Model Deployment**:
```bash Command
sglang serve \
--model-path MiniMaxAI/MiniMax-M2 \
--tp-size 4 \
--trust-remote-code \
--mem-fraction-static 0.85
```
### 5.1.1 Low Concurrency (Latency-Optimized)
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model MiniMaxAI/MiniMax-M2 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 138.91
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4220
Request throughput (req/s): 0.07
Input token throughput (tok/s): 43.92
Output token throughput (tok/s): 30.38
Peak output token throughput (tok/s): 46.00
Peak concurrent requests: 2
Total token throughput (tok/s): 74.30
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 13887.62
Median E2E Latency (ms): 10377.26
---------------Time to First Token----------------
Mean TTFT (ms): 4528.94
Median TTFT (ms): 385.23
P99 TTFT (ms): 38338.51
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 22.21
Median TPOT (ms): 22.24
P99 TPOT (ms): 22.25
---------------Inter-Token Latency----------------
Mean ITL (ms): 22.23
Median ITL (ms): 22.24
P95 ITL (ms): 22.35
P99 ITL (ms): 22.41
Max ITL (ms): 23.64
==================================================
```
### 5.1.2 Medium Concurrency (Balanced)
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model MiniMaxAI/MiniMax-M2 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 81.07
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 40805
Total generated tokens (retokenized): 40803
Request throughput (req/s): 0.99
Input token throughput (tok/s): 489.29
Output token throughput (tok/s): 503.32
Peak output token throughput (tok/s): 704.00
Peak concurrent requests: 19
Total token throughput (tok/s): 992.61
Concurrency: 13.74
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 13925.95
Median E2E Latency (ms): 14348.75
---------------Time to First Token----------------
Mean TTFT (ms): 532.32
Median TTFT (ms): 147.69
P99 TTFT (ms): 1978.48
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 27.49
Median TPOT (ms): 26.56
P99 TPOT (ms): 46.52
---------------Inter-Token Latency----------------
Mean ITL (ms): 26.31
Median ITL (ms): 23.47
P95 ITL (ms): 24.37
P99 ITL (ms): 125.10
Max ITL (ms): 1192.51
==================================================
```
### 5.1.3 High Concurrency (Throughput-Optimized)
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model MiniMaxAI/MiniMax-M2 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 153.71
Total input tokens: 249831
Total input text tokens: 249831
Total input vision tokens: 0
Total generated tokens: 252662
Total generated tokens (retokenized): 250982
Request throughput (req/s): 3.25
Input token throughput (tok/s): 1625.33
Output token throughput (tok/s): 1643.75
Peak output token throughput (tok/s): 2597.00
Peak concurrent requests: 107
Total token throughput (tok/s): 3269.09
Concurrency: 91.14
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 28017.24
Median E2E Latency (ms): 26865.28
---------------Time to First Token----------------
Mean TTFT (ms): 387.41
Median TTFT (ms): 183.90
P99 TTFT (ms): 1192.44
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 55.23
Median TPOT (ms): 57.84
P99 TPOT (ms): 70.23
---------------Inter-Token Latency----------------
Mean ITL (ms): 54.79
Median ITL (ms): 39.01
P95 ITL (ms): 143.10
P99 ITL (ms): 150.46
Max ITL (ms): 986.14
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 GSM8K Benchmark
- **Server Command**:
```shell Command
sglang serve \
--model-path MiniMaxAI/MiniMax-M2 \
--tp-size 4 \
--trust-remote-code \
--mem-fraction-static 0.85
```
- **Benchmark Command**:
```shell Command
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
```
- **Result**:
- MiniMax-M2
```text Output
Accuracy: 0.950
Invalid: 0.000
Latency: 15.120 s
Output throughput: 1306.711 token/s
```
docs/cookbook/autoregressive/MiniMax/MiniMax-M3.mdx
+506
View File
@@ -0,0 +1,506 @@
---
title: MiniMax-M3
description: "Deploy MiniMax-M3 with SGLang — a ~428B-param (23B activated) multimodal Mixture-of-Experts reasoning model with MiniMax Sparse Attention and 1M context, MXFP8 on NVIDIA Blackwell & AMD Instinct, bf16 on Hopper."
tag: NEW
---
## Deployment
<a id="install" />
<Accordion title="Install SGLang">
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
<Tabs>
<Tab title="Python (pip / uv)">
```bash Command
pip install -U uv
uv venv --python 3.12 && source .venv/bin/activate
# MiniMax-M3 ships in SGLang PR #27944, not yet in a tagged release — install from
# the PR head. The serving runtime is in the base dependencies, so no extra is needed:
git clone https://github.com/sgl-project/sglang.git
cd sglang
git fetch origin pull/27944/head && git checkout FETCH_HEAD
uv pip install -e python
```
Then run the **Python** output of the command panel below in that environment. The **Docker** tab is simpler — its image bundles the CUDA-13 runtime and the #27944 code. Once [PR #27944](https://github.com/sgl-project/sglang/pull/27944) is merged and released, `uv pip install sglang` will pull M3 support directly.
</Tab>
<Tab title="Docker">
```bash Command
# Pull the M3 image the command panel selects for your platform, e.g.:
docker pull lmsysorg/sglang:dev-cu13-minimax-m3
```
The command panel below fills in the right tag per platform: `dev-cu13-minimax-m3` (CUDA 13 — B300, GB200, GB300), `dev-cu12-minimax-m3` (CUDA 12 — Hopper H200), or `dev-minimax-m3` (default). On AMD Instinct it uses the matching ROCm image (MI300X/MI325X → `aigmkt/minimax-m3-sglang-rocm700-mi30x`, MI350X/MI355X → `aigmkt/minimax-m3-sglang-rocm720-mi35x`). For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker), substituting the inner `sglang serve ...` with what the command generator produces.
<Note>
These M3 dev images now **bundle MiniMax's MSA sparse-attention kernel** (`fmha_sm100`), so Blackwell users get the recommended fast path automatically — no manual install needed (see **§2.1**). On a custom image without it, the same recipe still serves on the built-in Triton sparse path.
</Note>
</Tab>
</Tabs>
</Accordion>
Pick your hardware + recipe to generate the launch command.
import { Deployment } from "/src/snippets/_deployment.jsx";
import { config } from "/src/snippets/configs/MiniMaxAI/minimax-m3.jsx";
import { benchmarks } from "/src/snippets/configs/MiniMaxAI/minimax-m3-benchmarks.jsx";
<Deployment config={config} benchmarks={benchmarks} />
## Playground
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.
import { Playground } from "/src/snippets/_playground.jsx";
<Playground config={config} />
## 1. Model Introduction
[MiniMax-M3](https://huggingface.co/MiniMaxAI/MiniMax-M3-MXFP8) is MiniMax's native-multimodal Mixture-of-Experts reasoning model: **~428B total parameters with ~23B activated per token** (128 experts, 4 active per token), 60 layers, and a **1M-token context** over text, image, and video. Its defining feature is **MiniMax Sparse Attention (MSA)** — a block-sparse "lightning indexer" attention that keeps long-context cost low (MiniMax reports ~9× prefill / ~15× decode speedup over M2 at 1M context). This page serves the **MXFP8** variant (`MiniMaxAI/MiniMax-M3-MXFP8`, ~440 GB) on NVIDIA Blackwell and AMD Instinct; on NVIDIA Hopper (H200), use the full-precision **bfloat16** build [`MiniMaxAI/MiniMax-M3`](https://huggingface.co/MiniMaxAI/MiniMax-M3) (§2.4). Released under the **MiniMax Community License**.
Key characteristics as served by SGLang:
- **Multimodal (vision + text)**: accepts interleaved text and images through the OpenAI-compatible chat API (loaded as `MiniMaxM3SparseForConditionalGeneration`). Image input via URL and base64 is validated; video input has not been tested here.
- **Reasoning model**: emits its chain of thought wrapped in `<mm:think>...</mm:think>`. Always launch with **`--reasoning-parser auto`** — it auto-detects the right parser from the chat template, and SGLang then strips the tags and returns the trace separately in `message.reasoning_content`.
- **Native tool calling**: a custom namespace-token XML format, parsed into standard OpenAI `tool_calls`. Always launch with **`--tool-call-parser auto`** — it auto-detects the right parser from the chat template. Single, parallel, and nested (object / array) arguments are supported.
- **Sparse attention**: most layers use M3's "lightning indexer" block-sparse attention (top-k 128-token blocks), which keeps decode cost roughly flat in context length. On Blackwell, MiniMax's open-source [MSA kernel](https://github.com/MiniMax-AI/MSA) accelerates this path further (§2.1).
- **MXFP8 quantization across vendors**: the MXFP8 MoE weights run natively on NVIDIA Blackwell (B200 / B300 / GB200 / GB300) and on AMD Instinct MI350X/MI355X (gfx950 / CDNA4), both of which have hardware MX-scaled matmul. On AMD MI300X/MI325X (gfx942 / CDNA3) — no hardware MX — SGLang converts the weights to block-fp8 `[128,128]` at load and serves them on the tuned ROCm kernels (§2.3). The vision tower stays unquantized.
**Recommended generation**: the model's `generation_config.json` sets `temperature` 1.0 / `top_p` 0.95, which SGLang applies automatically (the default `--sampling-defaults model`). The model card additionally suggests `top_k` 40, but that value is **not** in `generation_config.json`, so SGLang does not apply it by default. `top_k` is a per-request sampling parameter (not a launch flag) — set it per call if you want it, e.g. `extra_body={"top_k": 40}` with the OpenAI client.
**Resources:** [HuggingFace](https://huggingface.co/MiniMaxAI/MiniMax-M3-MXFP8) · [MSA kernel](https://github.com/MiniMax-AI/MSA)
## 2. Configuration Tips
### 2.1 MSA sparse-attention fast path (recommended for Blackwell users)
[MiniMax MSA](https://github.com/MiniMax-AI/MSA) (`fmha_sm100`, MIT-licensed) is the recommended Blackwell kernel for M3's main sparse-attention step — faster and more memory-efficient than the built-in Triton fallback. **It ships pre-installed in the M3 dev image** (`lmsysorg/sglang:dev-minimax-m3`, also published under the `dev-cu13-minimax-m3` tag), so the Blackwell recipe above engages it automatically with no extra setup — `import fmha_sm100` works out of the box and the kernels JIT-compile on first use. It is otherwise purely additive: on a custom image, install it (below) and the recipe engages it automatically; without it the same recipe still serves on the built-in Triton path. The swap is numerically equivalent (cosine ≥ 0.99999 vs Triton), decode stays CUDA-graph-capturable, prefill TTFT drops ~9–12% at 8K–64K context, and the MSA path survives memory configurations where the Triton path OOMs.
**Requirements** (from the [MSA README](https://github.com/MiniMax-AI/MSA#requirements)):
- **GPU**: NVIDIA SM100 family — sm_100 (B200 / GB200) and sm_103 (B300 / GB300).
- **Toolchain**: CUDA Toolkit with `nvcc` ≥ 12.x on `PATH` (or `CUDA_HOME` set) — the kernels are JIT-compiled at first import.
- **Python**: ≥ 3.10; **OS**: Linux — works on both **x86_64 and aarch64 (Grace, e.g. GB200 / GB300)**; the aarch64 build needs no source edits.
<Accordion title="Install MSA (only on a custom image) & verify the gate (Python)">
The M3 Blackwell dev images above already bundle MSA, so you can skip straight to the gate check. The `git clone` / `pip install` steps are only needed on a custom image that doesn't have `fmha_sm100`.
```bash Command
# Only on a custom image: --recursive pulls the CUTLASS submodule required for JIT compilation
git clone --recursive https://github.com/MiniMax-AI/MSA.git msa
cd msa && pip install .
# Verify the SGLang gate (True -> MSA engaged on this device; False -> Triton fallback):
python -c "from sglang.srt.layers.attention.minimax_sparse_ops.msa import msa_available; print(msa_available())"
```
</Accordion>
<Note>
The first import JIT-compiles the kernels, which can take 30 s to a few minutes on a cold `nvcc` cache — this is normal, not a hang. Subsequent server starts hit the JIT cache.
</Note>
<Warning>
**Warm the JIT cache before a multi-GPU launch.** On a *cold* cache, several tensor-parallel ranks racing to JIT-compile MSA's plan kernel can leave one rank loading a half-linked module (`AttributeError: Module has no function 'plan'` at CUDA-graph capture). Run the gate-check `python -c "..."` (or any single-process `fmha_sm100_plan` call) once before launching the server — that compiles the kernel single-process, and every rank then hits the warm cache.
</Warning>
The gate requires `--attention-backend fa4` (MSA's sparse blocks are 128 tokens, so the page size must be 128). SGLang auto-forces `page_size` to 128 for the `fa4` backend — including the combined `--attention-backend fa4` the M3 recipe uses (#28976) — so `--page-size 128` is omitted from the Blackwell cells below. Force the Triton path at any time with the env var `SGLANG_DISABLE_MSA=1`. MSA is a Blackwell (SM100) kernel and does not apply to the AMD ROCm paths.
<Note>
For multimodal (image) serving, keep the same text recipe above — `--attention-backend fa4` (MSA) is unchanged — and add `--mm-attention-backend flashinfer_cudnn` for the vision tower. The text and vision-tower attention backends are independent knobs; MSA only touches the language-model sparse attention, not image handling.
</Note>
### 2.2 Memory and workload tuning
The NVIDIA Blackwell recipes are validated single-node: **B200 at `--tp 8`** and **B300 / GB300 at `--tp 4`** (4-GPU is also the GB200 / GB300 single-node ceiling). GB200 (sm_100, aarch64) is inferred-supported — both of its axes are validated above (B200 is sm_100; GB300 is sm_103 aarch64) — but not directly benchmarked. The AMD recipes use **8-GPU (`--tp 8`)**.
- **Memory**: `--mem-fraction-static` reserves GPU memory for weights + KV pool; the rest is prefill **activation headroom**. The value scales with *free* memory per GPU (card capacity minus per-GPU weight), so it tracks the card more than the TP degree: **`0.65` on B200** (180 GB — less headroom once weights are resident) and **`0.75` on the larger-memory B300 / GB300** (`0.80` on AMD). Lower TP packs more weight per GPU, so a tighter config needs a *lower* value — B200 needs `0.65` even at `--tp 4`. Raising it past the validated value is fine only for low-concurrency single-stream serving; it OOMs under high concurrency or long context.
- **Long context (32K+)**: keep `--mem-fraction-static` at the platform default and raise `--chunked-prefill-size` to `16384`. Decode TPOT stays roughly flat in context length thanks to sparse attention; 1K–128K prompts are validated.
- **Scaling TP**: B200 is documented at `--tp 8`; B300 / GB200 / GB300 at `--tp 4` (the single-node cross-family common denominator). On an 8-GPU B300 host you can also raise to `--tp 8` for more throughput / KV headroom.
- **Expert parallelism**: to trade latency for throughput add `--ep` (see [Expert Parallelism Deployment](../../../docs/advanced_features/expert_parallelism)). On AMD, set `--ep` equal to `--tp`. Shared-experts fusion is automatically disabled when EP > 1; on AMD standard EP the server also disables `--enable-aiter-allreduce-fusion` automatically to preserve accuracy.
- `--trust-remote-code` is required to load the MiniMax config / processor classes.
### 2.3 AMD Instinct (ROCm)
MiniMax-M3 runs on AMD Instinct GPUs through two code paths, by architecture — both selected automatically; you still pass `--quantization mxfp8` either way:
- **MI350X / MI355X (gfx950, CDNA4)** has hardware MX-scaled matmul, so the **MXFP8 weights are served natively**. SGLang auto-detects the checkpoint, selects the Triton MiniMax-M3 MoE path with the packaged tuned MXFP8 configs, and enables AITER fused all-reduce for single-node tensor parallelism. The launch command is the NVIDIA recipe minus the Blackwell-only backend flags.
- **MI300X / MI325X (gfx942, CDNA3)** has **no** hardware MX matmul. SGLang transparently **converts the MXFP8 weights to block-fp8 `[128,128]` at load time**, then serves them with the tuned ROCm block-fp8 kernels (`--attention-backend aiter`, `--moe-runner-backend triton`; the `aiter` runner also works and scores marginally higher). On a cold start the first generation can JIT-compile AITER configs and exceed the default warmup/HTTP timeout, so the recipe adds `--watchdog-timeout 3600 --skip-server-warmup`. The block-fp8 step adds only a small relative error over MXFP8's native `1×32` scaling — negligible on GSM8K (see the benchmark card).
Select an MI300X/MI325X or MI350X/MI355X tile in the command panel above to get the exact launch command for each path.
<Note>
The AMD recipes are validated end-to-end on **text** workloads — chat, reasoning separation, and tool calling. The vision tower was not exercised on ROCm; for image input on AMD, omit the Blackwell `--mm-attention-backend flashinfer_cudnn` flag and let the encoder use the ROCm default backend, and treat vision as unvalidated on that path.
</Note>
### 2.4 Serving on Hopper (H200) with the bf16 build
The MXFP8 kernels are Blackwell-only, so Hopper (H200) serves the full-precision bfloat16 build [`MiniMaxAI/MiniMax-M3`](https://huggingface.co/MiniMaxAI/MiniMax-M3). Select **H200 + BF16** in the Deploy panel above for the exact command — it runs at `--tp 8` (the bf16 weights need a full 8-GPU node). SGLang picks the right backends for Hopper automatically, so the recipe stays minimal:
- **MoE runner**: Triton, auto-selected for bf16 weights.
- **Attention**: FlashAttention-3 with page size 1. MSA (§2.1) is a Blackwell kernel, so M3's sparse step runs on the built-in Triton path here.
- **CUDA graph**: on, with full decode-graph capture.
**High-concurrency throughput (optional).** On Hopper the sparse prefill runs on the Triton path as a separate eager forward, which briefly stalls the in-flight decode batch under heavy concurrent load. Adding `--enable-mixed-chunk --chunked-prefill-size 2048` merges the running decodes into the prefill step instead of preempting them, which recovers roughly **+10% output throughput** and **~10% lower median TPOT** at high concurrency on 8×H200, with no change in accuracy. Leave it off for latency-sensitive low-concurrency serving.
Validated on 8×H200 — reasoning and tool-call auto-detection plus long-context generation. For prefill/decode disaggregation on Hopper, see §3.4.
## 3. Advanced Usage
### 3.1 Reasoning
Launch with `--reasoning-parser auto` (or toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)). The `<mm:think>` trace then lands in `message.reasoning_content`, separate from the final answer in `message.content` — no client-side tag stripping needed.
<Accordion title="Reasoning Example (Python)">
```python Example
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="MiniMaxAI/MiniMax-M3-MXFP8",
messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}],
max_tokens=2048,
)
message = response.choices[0].message
print("=============== Reasoning ===============")
print(message.reasoning_content)
print("=============== Answer ==================")
print(message.content)
```
</Accordion>
<Accordion title="Example Output">
```text Output
=============== Reasoning ===============
15% of 240. 15% = 0.15. 240 * 0.15 = 36. Quick check: 10% is 24, 5% is 12, 24 + 12 = 36.
=============== Answer ==================
15% of 240 is **36**.
(10% of 240 = 24, and 5% of 240 = 12; 24 + 12 = 36.)
```
</Accordion>
When streaming, the trace arrives on `delta.reasoning_content` and the answer on `delta.content`, so the two sections can be rendered separately in real time:
<Accordion title="Streaming Reasoning (Python)">
```python Example
response = client.chat.completions.create(
model="MiniMaxAI/MiniMax-M3-MXFP8",
messages=[{"role": "user", "content": "Solve step by step: what is 15% of 240?"}],
max_tokens=2048,
stream=True,
)
for chunk in response:
if not chunk.choices:
continue
delta = chunk.choices[0].delta
if getattr(delta, "reasoning_content", None):
print(delta.reasoning_content, end="", flush=True) # thinking stream
if delta.content:
print(delta.content, end="", flush=True) # answer stream
print()
```
**Output Example:**
```text Output
[delta.reasoning_content — thinking stream]
Let me solve this step by step.
15% of 240
= 0.15 × 240
= 36
Let me verify: 10% of 240 = 24, 5% of 240 = 12, so 15% = 24 + 12 = 36. ✓
[delta.content — answer stream]
# Solving 15% of 240
## Step 1: Convert the percentage to a decimal
15% = 15/100 = 0.15
## Step 2: Multiply by 240
0.15 × 240 = 36
## Answer
**15% of 240 = 36**
```
</Accordion>
### 3.2 Tool Calling
Launch with `--tool-call-parser auto` (or toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) — it auto-detects M3's tool-call parser from the chat template. M3 emits tool calls in a custom namespace-token XML format:
```text Raw model output
]<]minimax[>[<tool_call>
]<]minimax[>[<invoke name="get_weather">]<]minimax[>[<location>Beijing]<]minimax[>[</location>]<]minimax[>[</invoke>
]<]minimax[>[</tool_call>
```
The parser converts that into the standard OpenAI `tool_calls` structure:
<Accordion title="Tool Calling Example (Python)">
```python Example
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string", "description": "The city name"},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
},
"required": ["location"],
},
},
}
]
response = client.chat.completions.create(
model="MiniMaxAI/MiniMax-M3-MXFP8",
messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
tools=tools,
)
message = response.choices[0].message
if message.tool_calls:
for call in message.tool_calls:
print(f"Tool: {call.function.name}")
print(f"Args: {call.function.arguments}")
```
</Accordion>
<Accordion title="Example Output">
```text Output
Tool: get_weather
Args: {"location": "Beijing"}
```
</Accordion>
Beyond a single flat call, the parser also supports:
- **Parallel calls** — multiple `<invoke>` blocks inside the single `<tool_call>` wrapper, surfaced as multiple `message.tool_calls` entries.
- **Nested object arguments** — an `object`-typed parameter is emitted as nested XML tags and reconstructed into a JSON object.
- **Array arguments** — an `array`-typed parameter uses repeated `<item>` children and is reconstructed into a JSON list.
For example, a tool with object and array parameters round-trips cleanly:
```text Output
create_event {"title": "Design sync", "attendees": ["alice", "bob"], "location": {"room": "R2", "floor": 3}}
```
To return a tool result, append the assistant's `tool_calls` turn plus a matching `tool` message and ask the model to continue — the follow-up answer may place text in `reasoning_content` as well as `content`, so print both.
### 3.3 Multimodal (Vision) Input
Images go through the standard OpenAI `image_url` content type. The vision tower is always loaded; for image serving add `--mm-attention-backend flashinfer_cudnn` (the vision-tower backend) to the Blackwell deployment recipe — the text `--attention-backend` is unchanged (§2.1 note). On AMD, omit `--mm-attention-backend` and let the encoder use the ROCm default (vision is unvalidated on ROCm — §2.3).
<Accordion title="Vision Example (Python)">
```python Example
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="MiniMaxAI/MiniMax-M3-MXFP8",
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"
},
},
{"type": "text", "text": "Describe this image in detail."},
],
}
],
max_tokens=1024,
)
print(response.choices[0].message.content)
```
**Output Example:**
```text Output
This image captures a striking and unusual urban scene on what appears to be a busy New York City street.
**Main Subject:**
A man stands on the rear bumper of a yellow taxi cab (an SUV-style cab, likely a Ford Escape hybrid), operating a full-sized ironing board set up across the back of the vehicle. He is wearing a bright yellow long-sleeved shirt and dark pants, and is actively ironing a blue garment, holding an iron in his right hand.
**Vehicles:**
- The yellow SUV taxi on the right is stationary, its rear hatch serving as the ironing platform.
- A second yellow taxi (a sedan) drives past on the left, captured with motion blur.
**Setting:**
Tall city buildings with classic urban architecture, an American flag, and white lane markings — a bustling downtown area, possibly Midtown Manhattan.
```
</Accordion>
Notes:
- If the server cannot fetch external URLs, embed the image as a base64 `data:image/png;base64,...` URI — SGLang decodes it server-side.
- Multiple images per message are supported; add more `image_url` entries to the `content` list.
- Reasoning and tool calling work the same way for multimodal requests — a vision prompt can still produce a `<mm:think>` trace and/or tool calls.
### 3.4 Prefill-Decode (PD) Disaggregation
[PD disaggregation](../../../docs/advanced_features/pd_disaggregation) runs prefill and decode on **separate** SGLang servers linked by an RDMA KV-transfer fabric (mooncake or NIXL), fronted by the PD router. M3 needs one thing beyond a dense model: alongside the main KV cache, every sparse "lightning-indexer" layer keeps a **K-only index buffer**, and that buffer must reach the decode server too — otherwise sparse attention reads stale state. SGLang transfers it alongside the main KV — reusing the same page mapping — so M3 disaggregates correctly with no extra flags.
**Supported topology** (the released MiniMax-M3, whose sparse layers are all K-only):
- **Equal tensor parallelism** — the prefill and decode servers run the same `--tp`.
- **Single pipeline stage** — PP = 1 (the default).
- **mooncake or NIXL** transfer backend over RDMA / InfiniBand.
Launch the prefill server, then the decode server — the same recipe with `--disaggregation-mode decode` and no bootstrap port. Pick your hardware:
<Tabs>
<Tab title="Blackwell · MXFP8">
On Blackwell the MXFP8 recipe — fa4, page size 128, deep_gemm MoE, and the MSA fast path (§2.1) — is auto-selected, so each role adds only the `--disaggregation-*` flags. This is the validated **2 × 4×B200** setup (TP4 prefill on node A, TP4 decode on node B); point `--disaggregation-ib-device` at your RDMA NIC(s).
```bash Prefill server (node A)
sglang serve \
--model-path MiniMaxAI/MiniMax-M3-MXFP8 \
--trust-remote-code \
--reasoning-parser auto \
--tool-call-parser auto \
--tp 4 \
--disaggregation-mode prefill \
--disaggregation-transfer-backend nixl \
--disaggregation-ib-device mlx5_0 \
--host 0.0.0.0 --port 30000 \
--disaggregation-bootstrap-port 8998
```
```bash Decode server (node B)
sglang serve \
--model-path MiniMaxAI/MiniMax-M3-MXFP8 \
--trust-remote-code \
--reasoning-parser auto \
--tool-call-parser auto \
--tp 4 \
--disaggregation-mode decode \
--disaggregation-transfer-backend nixl \
--disaggregation-ib-device mlx5_0 \
--host 0.0.0.0 --port 30001
```
</Tab>
<Tab title="Hopper · bf16">
On Hopper (H200) M3 runs the bf16 build (§2.4) with Triton MoE and the built-in Triton sparse path, pinned to `--page-size 128` so both roles share the page layout the sparse-index transfer relies on. This is the validated **2 × 8×H200** setup (TP8 each).
```bash Prefill server (node A)
sglang serve \
--model-path MiniMaxAI/MiniMax-M3 \
--trust-remote-code \
--reasoning-parser auto \
--tool-call-parser auto \
--tp 8 \
--attention-backend triton \
--moe-runner-backend triton \
--page-size 128 \
--disaggregation-mode prefill \
--disaggregation-transfer-backend mooncake \
--disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \
--host 0.0.0.0 --port 30000 \
--disaggregation-bootstrap-port 8998
```
```bash Decode server (node B)
sglang serve \
--model-path MiniMaxAI/MiniMax-M3 \
--trust-remote-code \
--reasoning-parser auto \
--tool-call-parser auto \
--tp 8 \
--attention-backend triton \
--moe-runner-backend triton \
--page-size 128 \
--disaggregation-mode decode \
--disaggregation-transfer-backend mooncake \
--disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \
--host 0.0.0.0 --port 30001
```
</Tab>
</Tabs>
Then start the PD router, pointing it at the prefill bootstrap (URL plus its `--disaggregation-bootstrap-port`) and the decode endpoint:
```bash PD router
python3 -m sglang_router.launch_router \
--pd-disaggregation \
--prefill http://<prefill-host>:30000 8998 \
--decode http://<decode-host>:30001 \
--policy round_robin \
--host 0.0.0.0 --port 8000
```
Clients hit the router exactly like a single server — it splits each request across the two stages transparently:
<Accordion title="PD Client Example (Python)">
```python Example
from openai import OpenAI
client = OpenAI(base_url="http://<router-host>:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="MiniMaxAI/MiniMax-M3-MXFP8",
messages=[{"role": "user", "content": "What is 2 + 2?"}],
max_tokens=64,
)
print(response.choices[0].message.content)
```
**Output Example:**
```text Output
2 + 2 = 4
```
</Accordion>
**Validation.** PD disaggregation preserves output quality — the K-only sparse index transfers arrive intact and disaggregated output matches non-disaggregated serving. GSM8K is scored with the single sgl-eval harness used by the benchmark card above (full 1319-question split, chat with `--thinking`); see that card for per-platform single-node accuracy.
- **2 × 4×B200** (TP4+TP4, MXFP8, NIXL over InfiniBand) — output matches single-node serving. The 2-node PD serving benchmark (512-token input, 256-token output, 16 concurrent — a different workload from the card's single-node `random` isl=2048 / osl=256 / conc=64 row, so the throughput figures are not directly comparable) measured mean TTFT 1.1 s and TPOT 16.6 ms (≈ 60 tok/s per stream, ≈ 2.3k tok/s aggregate).
- **2 × 8×H200** (TP8+TP8, bf16, mooncake) — output matches single-node serving.