minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[Docs] Rename docs_new/ to docs/ (#32123)

Browse Source 
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
zijiexia
2026-08-03 16:51:00 -07:00
committed by GitHub
co-authored by Claude Opus 4.8
parent c949e91f18
commit b819d2fb5b
491 changed files with 122 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/Moonshotai/Kimi-K2.5.mdx
+1246
View File
File diff suppressed because it is too large Load Diff
docs/cookbook/autoregressive/Moonshotai/Kimi-K2.6.mdx
+1426
View File
File diff suppressed because it is too large Load Diff
docs/cookbook/autoregressive/Moonshotai/Kimi-K2.7-Code.mdx
+556
View File
@@ -0,0 +1,556 @@
---
title: Kimi-K2.7-Code
description: "Deploy Kimi-K2.7-Code with SGLang for coding-focused agentic workflows, thinking output, tool calling, and multimodal input."
metatags:
description: "Deploy Kimi-K2.7-Code native multimodal agentic model with SGLang - reasoning, tool calling, and multimodal capabilities."
---
## 1. Model Introduction
[Kimi-K2.7-Code](https://huggingface.co/moonshotai/Kimi-K2.7-Code) is a coding-focused agentic model by Moonshot AI, built on top of Kimi-K2.6. It improves real-world long-horizon coding task completion while reducing thinking-token usage by approximately 30% compared with Kimi-K2.6.
**Key Features:**
- **Coding-Focused Agentic Model**: Optimized for end-to-end coding workflows and complex software engineering tasks.
- **Token Efficiency**: Reduces thinking-token usage by approximately 30% versus Kimi-K2.6.
- **K2.6-Compatible Deployment**: Shares the same architecture as Kimi-K2.5/Kimi-K2.6, so the SGLang deployment method can be reused with the new model ID.
- **Native Multimodality**: Shares Kimi-K2.6's native multimodal architecture with a MoonViT vision encoder (400M parameters) and supports image and video (experimental) input.
**Benchmarks:**
<table>
<thead>
<tr>
<th>Benchmark</th>
<th>Kimi-K2.6</th>
<th>Kimi-K2.7-Code</th>
</tr>
</thead>
<tbody>
<tr>
<td>Kimi Code Bench v2</td>
<td>50.9</td>
<td>62.0</td>
</tr>
<tr>
<td>Program Bench</td>
<td>48.3</td>
<td>53.6</td>
</tr>
<tr>
<td>MLS Bench Lite</td>
<td>26.7</td>
<td>35.1</td>
</tr>
<tr>
<td>Kimi Claw 24/7 Bench</td>
<td>42.9</td>
<td>46.9</td>
</tr>
<tr>
<td>MCP Atlas</td>
<td>69.4</td>
<td>76.0</td>
</tr>
<tr>
<td>MCP Mark Verified</td>
<td>72.8</td>
<td>81.1</td>
</tr>
</tbody>
</table>
**Recommended Generation Parameters:**
- Thinking Mode: `temperature=1.0`, `top_p=0.95`
- Kimi-K2.7-Code forces thinking and preserve-thinking behavior; instant mode is not supported.
**Available Models:**
- **INT4 (native checkpoint)**: [moonshotai/Kimi-K2.7-Code](https://huggingface.co/moonshotai/Kimi-K2.7-Code)
**License:** Modified MIT for the native checkpoint.
For details, see the [official model card](https://huggingface.co/moonshotai/Kimi-K2.7-Code).
## 2. SGLang Installation
Refer to the [official SGLang installation guide](/docs/get-started/install).
## 3. Model Deployment
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, deployment strategy, and capabilities.
import { KimiK27CodeDeployment } from '/src/snippets/autoregressive/kimi-k27-code-deployment.jsx'
<KimiK27CodeDeployment />
### 3.2 Configuration Tips
- **Memory**: Requires GPUs with ≥140GB each. The native INT4 checkpoint supports H200 (8×, TP=8), B300 (8×, TP=8), GB300 (4×, TP=4), MI300X/MI325X (4×, TP=4), and MI350X/MI355X (4×, TP=4). Use `--context-length 128000` to conserve memory.
- **Context Length**: The model supports a 256K context length. Use a shorter `--context-length` when you need to reserve memory for larger batches.
- **Transformers Version**: The model card requires `transformers>=4.57.1,<5.0.0`.
- **AMD GPU TP Constraint**: On AMD GPUs, TP must be ≤ 4 (not 8). Kimi-K2.7-Code has 64 attention heads; the AITER MLA kernel requires `heads_per_gpu % 16 == 0`. With TP=4, each GPU gets 16 heads (valid). With TP=8, each GPU gets 8 heads (invalid).
- **AMD Docker Image**: Use `lmsysorg/sglang:v0.5.9-rocm700-mi35x` for MI350X/MI355X and `lmsysorg/sglang:v0.5.9-rocm700-mi30x` for MI300X/MI325X.
- **DP Attention**: Enable with `--dp <N> --enable-dp-attention` for production throughput. A common choice is to set `--dp` equal to `--tp`, but this is not required.
- **Reasoning Parser**: Add `--reasoning-parser kimi_k2` to separate thinking and content in model outputs.
- **Tool Call Parser**: Add `--tool-call-parser kimi_k2` for structured tool calls.
- **AMD FP8 KV Cache**: On AMD platforms the generator adds `--kv-cache-dtype fp8_e4m3` by default and sets `--mem-fraction-static 0.8` to fit the INT4 weights plus KV cache. FP8 KV cache trades a small amount of accuracy for memory; omit the flag if you observe accuracy regressions on your workload.
## 4. Model Invocation
### 4.1 Basic Usage
See [Basic API Usage](/docs/basic_usage/send_request).
### 4.2 Advanced Usage
#### 4.2.1 Multimodal (Vision + Text) Input
Kimi-K2.7-Code supports native multimodal input with images:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="moonshotai/Kimi-K2.7-Code",
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
}
},
{
"type": "text",
"text": "What is in this image? Describe it in detail."
}
]
}
]
)
print(response.choices[0].message.content)
```
**Output Example:**
```text Output
This image shows a **paper receipt from Auntie Anne's**, the pretzel chain restaurant. Here's a detailed breakdown:
## Header
- At the top left is the Auntie Anne's logo (a pretzel with a halo)
- The store name "**Auntie Anne's**" is printed prominently at the top
- Some text below the store name appears blurred/redacted (likely store location, address, or transaction details)
## Purchase Details
- **Item**: CINNAMON SUGAR
- **Quantity & Price**: 1 × 17,000
- **Item Total**: 17,000
## Financial Summary
- **SUB TOTAL**: 17,000
- **GRAND TOTAL**: 17,000
- **CASH IDR**: 20,000 (customer paid 20,000 Indonesian Rupiah)
- **CHANGE DUE**: 3,000
## Physical Description
- The receipt is printed on white thermal paper
- Some information in the middle section and toward the bottom is intentionally blurred/obscured
- The paper appears slightly curved/wrinkled and is placed on a dark brown surface (likely a table or counter)
The transaction is in **Indonesian Rupiah (IDR)**, indicating this purchase was made at an Auntie Anne's location in Indonesia. The customer bought one Cinnamon Sugar pretzel for 17,000 IDR and received 3,000 IDR in change after paying with 20,000 IDR cash.
```
#### 4.2.2 Reasoning Output
Kimi-K2.7-Code forces thinking mode and preserve-thinking behavior.
**Thinking Mode (default)** — reasoning content is automatically separated:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="moonshotai/Kimi-K2.7-Code",
messages=[
{"role": "user", "content": "Which one is bigger, 9.11 or 9.9? Think carefully."}
]
)
print("====== Reasoning Content (Thinking Mode) ======")
print(response.choices[0].message.reasoning_content)
print("====== Response (Thinking Mode) ======")
print(response.choices[0].message.content)
```
**Output Example:**
```text Output
====== Reasoning Content (Thinking Mode) ======
The user is asking which number is bigger: 9.11 or 9.9. This seems straightforward, but there's a viral internet debate about this due to decimal confusion.
Let me think carefully:
- 9.11 means 9 + 11/100 = 9.11
- 9.9 means 9 + 9/10 = 9.90
So 9.9 = 9.90, and 9.90 > 9.11 because 0.90 > 0.11.
The confusion often comes from people thinking of software versioning (where 9.11 comes after 9.9) or comparing the numbers after the decimal as whole numbers (11 vs 9, thinking 11 > 9).
So mathematically, 9.9 is clearly bigger. 9.9 - 9.11 = 0.79.
I should explain this clearly and address the common misconception.
====== Response (Thinking Mode) ======
Mathematically, **9.9 is bigger**.
Here's why:
**9.9 = 9.90**
When comparing decimals, you need to look at the same place values:
- 9.11 = 9 ones, 1 tenth, and 1 hundredth
- 9.9 = 9 ones, 9 tenths, and 0 hundredths (9.90)
Since **0.90 > 0.11**, it follows that **9.9 > 9.11**.
The difference is:
9.9 - 9.11 = 0.79
**Why people get confused:** Many mistakenly treat the decimals like whole numbers (thinking "11 is bigger than 9") or confuse this with software version numbering (where version 9.11 comes after version 9.9). But in standard mathematics, 9.9 is definitively larger.
```
#### 4.2.3 Preserve Thinking
Kimi-K2.7-Code keeps reasoning content across multi-turn interactions. This behavior is enabled by default and cannot be disabled.
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
messages = [
{
"role": "user",
"content": "Tell me three random numbers."
},
{
"role": "assistant",
"reasoning_content": "I'll start by listing five numbers: 473, 921, 235, 215, 222, and I'll tell you the first three.",
"content": "473, 921, 235"
},
{
"role": "user",
"content": "What are the other two numbers you have in mind?"
}
]
response = client.chat.completions.create(
model="moonshotai/Kimi-K2.7-Code",
messages=messages,
stream=False,
max_tokens=4096,
)
print(response.choices[0].message.content)
```
Some OpenAI-compatible deployments use `reasoning` instead of `reasoning_content` in assistant messages. Use the field your serving stack exposes.
#### 4.2.4 Tool Calling
Kimi-K2.7-Code supports tool calling capabilities for agentic tasks:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
response = client.chat.completions.create(
model="moonshotai/Kimi-K2.7-Code",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
stream=True
)
# Process streaming response
tool_calls_accumulator = {}
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if hasattr(delta, 'tool_calls') and delta.tool_calls:
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {'name': None, 'arguments': ''}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]['name'] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
if delta.content:
print(delta.content, end="", flush=True)
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
```
**Output Example:**
```text Output
Tool Call: get_weather
Arguments: {"location": "Beijing"}
```
**Handling Tool Call Results:**
```python Example
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": "The weather in Beijing is 22°C and sunny."
}
]
final_response = client.chat.completions.create(
model="moonshotai/Kimi-K2.7-Code",
messages=messages
)
print(final_response.choices[0].message.content)
```
**Output Example:**
```text Output
The weather in Beijing is currently **22°C and sunny**. ☀️
It's a nice, warm day there—great for being outdoors!
```
#### 4.2.5 Multimodal + Tool Calling (Agentic Vision)
Combine vision understanding with tool calling for advanced agentic tasks:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
tools = [
{
"type": "function",
"function": {
"name": "search_product",
"description": "Search for a product by name or description",
"parameters": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "The product name or description to search for"
}
},
"required": ["query"]
}
}
}
]
response = client.chat.completions.create(
model="moonshotai/Kimi-K2.7-Code",
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
}
},
{
"type": "text",
"text": "Can you identify this product and search for similar items?"
}
]
}
],
tools=tools
)
msg = response.choices[0].message
# Print reasoning process
if msg.reasoning_content:
print("=== Reasoning ===")
print(msg.reasoning_content)
# Print response content
if msg.content:
print("=== Content ===")
print(msg.content)
# Print tool calls
if msg.tool_calls:
print("=== Tool Calls ===")
for tc in msg.tool_calls:
print(f" Function: {tc.function.name}")
print(f" Arguments: {tc.function.arguments}")
```
**Output Example:**
```text Output
=== Reasoning ===
The user wants me to identify the product from the receipt and search for similar items. Looking at the receipt, it's from Auntie Anne's and the item purchased is "CINNAMON SUGAR" for 17,000 IDR. This is likely a Cinnamon Sugar Pretzel from Auntie Anne's, which is a popular pretzel chain.
I should search for this product using the search_product function. The query should be something like "Auntie Anne's Cinnamon Sugar Pretzel" or just "Cinnamon Sugar Pretzel" to find similar items.
=== Content ===
Based on the receipt, the product is a **Cinnamon Sugar Pretzel** from **Auntie Anne's** (a popular pretzel bakery chain). The receipt shows it was purchased for 17,000 Indonesian Rupiah (IDR).
Let me search for this product and similar items for you.
=== Tool Calls ===
Function: search_product
Arguments: {"query":"Auntie Anne's Cinnamon Sugar Pretzel"}
```
#### 4.2.6 Deployment Command Example
Deploy Kimi-K2.7-Code with the following command (H200/B300, reasoning and tool parsing enabled):
```shell Command
sglang serve \
--model-path moonshotai/Kimi-K2.7-Code \
--tp 8 \
--reasoning-parser kimi_k2 \
--tool-call-parser kimi_k2 \
--trust-remote-code \
--host 0.0.0.0 \
--port 30000
```
For GB300, use `--tp 4`.
## 5. Benchmark
The following results are from the official Kimi-K2.7-Code model card. They were evaluated with thinking mode enabled through Kimi Code CLI at `temperature=1.0`, `top_p=0.95`, and a 262,144-token context length unless otherwise stated.
<table>
<thead>
<tr>
<th>Category</th>
<th>Benchmark</th>
<th>Kimi-K2.6</th>
<th>Kimi-K2.7-Code</th>
</tr>
</thead>
<tbody>
<tr>
<td>Coding</td>
<td>Kimi Code Bench v2</td>
<td>50.9</td>
<td>62.0</td>
</tr>
<tr>
<td>Coding</td>
<td>Program Bench</td>
<td>48.3</td>
<td>53.6</td>
</tr>
<tr>
<td>Coding</td>
<td>MLS Bench Lite</td>
<td>26.7</td>
<td>35.1</td>
</tr>
<tr>
<td>Agentic</td>
<td>Kimi Claw 24/7 Bench</td>
<td>42.9</td>
<td>46.9</td>
</tr>
<tr>
<td>Agentic</td>
<td>MCP Atlas</td>
<td>69.4</td>
<td>76.0</td>
</tr>
<tr>
<td>Agentic</td>
<td>MCP Mark Verified</td>
<td>72.8</td>
<td>81.1</td>
</tr>
</tbody>
</table>
docs/cookbook/autoregressive/Moonshotai/Kimi-K2.mdx
+520
View File
@@ -0,0 +1,520 @@
---
title: Kimi-K2
metatags:
description: "Deploy Kimi-K2 MoE model with SGLang - 1T total parameters, 32B active, step-by-step reasoning and tool calling capabilities."
---
import { KimiK2Deployment } from '/src/snippets/autoregressive/kimi-k2-deployment.jsx';
## 1. Model Introduction
[Kimi-K2](https://moonshotai.github.io/Kimi-K2/) is a state-of-the-art MoE language model by Moonshot AI with 32B activated parameters and 1T total parameters.
**Model Variants:**
- **[Kimi-K2-Instruct](https://huggingface.co/moonshotai/Kimi-K2-Instruct)**: Post-trained model optimized for general-purpose chat and agentic tasks. Compatible with vLLM, SGLang, KTransformers, and TensorRT-LLM.
- **[Kimi-K2-Thinking](https://huggingface.co/moonshotai/Kimi-K2-Thinking)**: Advanced thinking model with step-by-step reasoning and tool calling. Native INT4 quantization with 256k context window. Ideal for complex reasoning and multi-step tool use.
- **ROCm Support**: Compatible with AMD MI300X GPUs via SGLang (verified).
For details, see [official documentation](https://github.com/MoonshotAI/Kimi-K2) and [technical report](https://www.arxiv.org/abs/2507.20534).
## 2. SGLang Installation
Refer to the [official SGLang installation guide](../../../docs/get-started/install).
## 3. Model Deployment
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and capabilities.
<KimiK2Deployment />
### 3.2 Configuration Tips
- **Memory**: Requires 8 GPUs with ≥140GB each (H200/B200). Use `--context-length 128000` to conserve memory.
- **Expert Parallelism (EP)**: Use `--ep` for better MoE throughput. See [EP docs](../../../docs/advanced_features/expert_parallelism).
- **Data Parallel (DP)**: Enable with `--dp 4 --enable-dp-attention` for production throughput.
- **KV Cache**: Use `--kv-cache-dtype fp8_e4m3` to reduce memory by 50% (CUDA 11.8+).
- **Reasoning Parser**: Add `--reasoning-parser kimi_k2` for Kimi-K2-Thinking to separate thinking and content.
- **Tool Call Parser**: Add `--tool-call-parser kimi_k2` for structured tool calls.
- **AMD GPU**: Set `SGLANG_ROCM_FUSED_DECODE_MLA=0` before launching AMD GPU.
## 4. Model Invocation
### 4.1 Basic Usage
See [Basic API Usage](../../../docs/get-started/quickstart).
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
Enable reasoning parser for Kimi-K2-Thinking:
```shell Command
python -m sglang.launch_server \
--model moonshotai/Kimi-K2-Thinking \
--reasoning-parser kimi_k2 \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
**Example:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="moonshotai/Kimi-K2-Thinking",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.6,
max_tokens=2048,
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
The user asks: "What is 15% of 240?" This is a straightforward percentage calculation problem. I need to solve it step by step.
Step 1: Understand what "percent" means.
- "Percent" means "per hundred". So 15% means 15 per 100, or 15/100, or 0.15.
Step 2: Convert the percentage to a decimal.
- 15% = 15 / 100 = 0.15
Step 3: Multiply the decimal by the number.
- 0.15 * 240
Step 4: Perform the multiplication.
- 0.15 * 240 = (15/100) * 240
- = 15 * 240 / 100
- = 3600 / 100
- = 36
Alternatively, I can calculate it directly:
- 0.15 * 240
- 15 * 240 = 3600
- 3600 / 100 = 36
Or, break it down:
- 10% of 240 = 24
- 5% of 240 = half of 10% = 12
- 15% of 240 = 10% + 5% = 24 + 12 = 36
I should present the solution clearly with steps. The most standard method is converting to decimal and multiplying.
Let me structure the answer:
1. Convert the percentage to a decimal.
2. Multiply the decimal by the number.
3. Show the calculation.
4. State the final answer.
This is simple and easy to follow.
=============== Content =================
Here is the step-by-step solution:
**Step 1: Convert the percentage to a decimal**
15% means 15 per 100, which is 15 ÷ 100 = **0.15**
**Step 2: Multiply the decimal by the number**
0.15 × 240
**Step 3: Calculate the result**
0.15 × 240 = **36**
**Answer:** 15% of 240 is **36**.
```
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
#### 4.2.2 Tool Calling
Kimi-K2-Instruct and Kimi-K2-Thinking support tool calling capabilities. Enable the tool call parser during deployment:
**Deployment Command:**
```shell Command
python -m sglang.launch_server \
--model moonshotai/Kimi-K2-Instruct \
--tool-call-parser kimi_k2 \
--tp 8 \
--trust-remote-code \
--host 0.0.0.0 \
--port 8000
```
**Python Example (with Thinking Process):**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="moonshotai/Kimi-K2-Thinking",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Accumulate tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================\n", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {
'name': None,
'arguments': ''
}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]['name'] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
# Print content
if delta.content:
print(delta.content, end="", flush=True)
# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"🔧 Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
print()
```
**Output Example:**
```text Output
=============== Thinking =================
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. Beijing is a major city in China, so I should be able to get weather data for it. The location parameter is required, but the unit parameter is optional. Since the user didn't specify a temperature unit, I can just provide the location and let the function use its default. I'll check the weather in Beijing for you.
=============== Content =================
🔧 Tool Call: get_weather
Arguments: {"location":"Beijing"}
```
**Note:**
- The reasoning parser shows how the model decides to use a tool
- Tool calls are clearly marked with the function name and arguments
- You can then execute the function and send the result back to continue the conversation
**Handling Tool Call Results:**
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="moonshotai/Kimi-K2-Thinking",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: NVIDIA B200 GPU (8x)
- Model: Kimi-K2-Instruct
- sglang version: 0.5.6.post1
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.
#### 5.1.1 Latency-Sensitive Benchmark
- Model Deployment Command:
```shell Command
python3 -m sglang.launch_server \
--model-path moonshotai/Kimi-K2-Instruct \
--tp 8 \
--dp 4 \
--enable-dp-attention \
--trust-remote-code \
--host 0.0.0.0 \
--port 8000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 8000 \
--model moonshotai/Kimi-K2-Instruct\
--num-prompts 10 \
--max-concurrency 1
```
- **Test Results**:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 44.93
Total input tokens: 1951
Total input text tokens: 1951
Total input vision tokens: 0
Total generated tokens: 2755
Total generated tokens (retokenized): 2748
Request throughput (req/s): 0.22
Input token throughput (tok/s): 43.42
Output token throughput (tok/s): 61.32
Peak output token throughput (tok/s): 64.00
Peak concurrent requests: 3
Total token throughput (tok/s): 104.74
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 4489.56
Median E2E Latency (ms): 4994.53
---------------Time to First Token----------------
Mean TTFT (ms): 141.22
Median TTFT (ms): 158.28
P99 TTFT (ms): 166.90
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 18.40
Median TPOT (ms): 15.63
P99 TPOT (ms): 39.88
---------------Inter-Token Latency----------------
Mean ITL (ms): 15.78
Median ITL (ms): 15.76
P95 ITL (ms): 16.36
P99 ITL (ms): 16.59
Max ITL (ms): 19.94
==================================================
```
#### 5.1.2 Throughput-Sensitive Benchmark
- Model Deployment Command:
```shell Command
python3 -m sglang.launch_server \
--model-path moonshotai/Kimi-K2-Instruct \
--tp 8 \
--dp 4 \
--ep 4 \
--enable-dp-attention \
--trust-remote-code \
--host 0.0.0.0 \
--port 8000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 8000 \
--model moonshotai/Kimi-K2-Instruct\
--num-prompts 1000 \
--max-concurrency 100
```
- **Test Results**:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 174.11
Total input tokens: 296642
Total input text tokens: 296642
Total input vision tokens: 0
Total generated tokens: 193831
Total generated tokens (retokenized): 168687
Request throughput (req/s): 5.74
Input token throughput (tok/s): 1703.73
Output token throughput (tok/s): 1113.25
Peak output token throughput (tok/s): 2383.00
Peak concurrent requests: 112
Total token throughput (tok/s): 2816.97
Concurrency: 89.60
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 15601.09
Median E2E Latency (ms): 10780.52
---------------Time to First Token----------------
Mean TTFT (ms): 457.42
Median TTFT (ms): 221.62
P99 TTFT (ms): 2475.32
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 97.23
Median TPOT (ms): 85.61
P99 TPOT (ms): 435.95
---------------Inter-Token Latency----------------
Mean ITL (ms): 78.61
Median ITL (ms): 43.66
P95 ITL (ms): 169.53
P99 ITL (ms): 260.91
Max ITL (ms): 1703.21
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 GSM8K Benchmark
- Server Command
```shell Command
python3 -m sglang.launch_server \
--model-path moonshotai/Kimi-K2-Instruct \
--tp 8 \
--dp 4 \
--trust-remote-code \
--host 0.0.0.0 \
--port 8000
```
- Benchmark Command
```shell Command
python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000
```
- **Result**:
```text Output
Accuracy: 0.960
Invalid: 0.000
Latency: 15.956 s
Output throughput: 1231.699 token/s
```
docs/cookbook/autoregressive/Moonshotai/Kimi-K3.mdx
+416
View File
@@ -0,0 +1,416 @@
---
title: Kimi-K3
description: "Deploy Moonshot AI's Kimi-K3 with SGLang — a 2.8T-parameter hybrid Mixture-of-Experts vision-language model (Kimi Delta Attention + MLA, 16/896 active experts) with NVIDIA and AMD recipes."
tag: NEW
---
## Deployment
<a id="install" />
<Accordion title="Install SGLang">
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install).
<Tabs>
<Tab title="Docker">
```bash Command
docker pull lmsysorg/sglang:kimi-k3 # CUDA13
docker pull lmsysorg/sglang:kimi-k3-cu12 # CUDA12
docker pull lmsysorg/sglang-rocm:rocm720-mi35x-k3-20260727 # ROCM
```
These tags publish with the public K3 launch; until then, build from the Dockerfiles linked below.
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
</Tab>
</Tabs>
If you do not want to use a Docker image, reproduce the dependency installation steps from the [CUDA 13 Dockerfile](https://github.com/sgl-project/sglang/blob/kimi-k3/docker/kimi_k3/kimi_k3_cu13.Dockerfile) or [CUDA 12 Dockerfile](https://github.com/sgl-project/sglang/blob/kimi-k3/docker/kimi_k3/kimi_k3_cu12.Dockerfile).
</Accordion>
Pick your hardware, then the deployment shape and operating point. Node count follows the hardware recipe (B200 2×8, GB200 4×4, H100 4×8, B300 1×8, H200 2×8 — 4×8 on Unified High-Throughput, GB300 2×4, MI350X/MI355X 1×8), so it is not a separate choice.
**PD Mode** — `Unified` serves prefill and decode together. `Prefill` / `Decode` split them into dedicated pools (see [PD disaggregation](#3-4-pd-disaggregation)); `Prefill` ships two strategies, both chunked at 16k. On the 8-GPU platforms (B300 1×8, GB300 2×4), `Default` is TP8 and `Long-Context` is `--pp-size 8 --tp-size 1`. On the 16-GPU platforms (B200 2×8, GB200 4×4), both are `--pp-size 16 --tp-size 1` and differ only in `--mem-fraction-static` (0.85 vs 0.90) — deep PP is the throughput shape there, not just the long-context one (see [Deep PP](#deep-pp-for-prefill)).
**Strategy** — the operating point within that shape:
- **Low-Latency** — no DCP, so the MLA KV stays TP-replicated. For chat. B200 splits its two nodes into PP2 × TP8; every other platform is flat TP.
- **Balanced** — the accuracy-preserving default: PP2 × DCPEP8 on B200 (the two pipeline stages and DCP8 split KV and KDA state), TP16/DCP16 on GB200, TP8/DCP8 on B300/GB300, TP8 ROCm/AITER on MI35x.
- **High-Throughput** — the large-scale lane: pick a **Cluster Size** and **Large-Scale Preset** in the Playground ([details](#large-scale-presets)). The cell itself is Balanced, except on H100 (plus `extra_buffer_lazy`) and H200 (widens to 4×8 TP32/EP32 at `--mem-fraction-static 0.90`).
`Long-Context` appears only under the `Prefill` PD mode; for long-context unified serving on B200, start from High-Throughput and raise `--context-length`.
**Spec Decode** — layers onto the strategy without changing it, on every platform except B200. DSPARK proposes 7 draft tokens per step (tune in the Playground) and requires `pp_size == 1`, so on B200 it also drops the pipeline and re-lays the same 16 GPUs flat: PP2 × TP8 → TP16, PP2 × DCPEP8 → DCPEP16. DFLASH has no published draft checkpoint. The win is largest on short interactive traffic and fades as the prompt grows.
<Note>
`--mamba-full-memory-ratio` is the one sizing flag, computed live: set your average request length in the [Mamba ratio calculator](#mamba-ratio-calculator); everything else follows the panels, and the result is pinned into the command.
</Note>
import { Deployment } from "/src/snippets/_deployment.jsx";
import { config } from "/src/snippets/configs/moonshotai/kimi-k3.jsx";
import { benchmarks } from "/src/snippets/configs/moonshotai/kimi-k3-benchmarks.jsx";
import { KimiK3MambaRatioCalculator } from "/src/snippets/_kimi_k3_mamba_ratio_calculator.jsx";
<Deployment config={config} benchmarks={benchmarks} />
### Mamba ratio calculator
<KimiK3MambaRatioCalculator />
<Accordion title="How --mamba-full-memory-ratio is calculated">
`--mamba-full-memory-ratio` is the ratio between the KDA state pool and the MLA KV pool. Every parameter below except `L` is read live from the Deploy panel and Playground selection; the balanced value is the per-request cost ratio:
```text
ratio = (S + D) x state_bytes / (L x (mla_kv_bytes / DCP + draft_kv_bytes))
```
- `S` — KDA state slots per request: `extra_buffer=5`, `extra_buffer_lazy=4`, `no_buffer=3`, disabled radix cache `=1`. `SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK` frees one slot on the extra-buffer strategies; with the overlap scheduler off (or `pp > 1`, which disables it) the track buffer costs one slot instead of two.
- `D` — verify intermediate states under speculative decoding: `0` when disabled, otherwise DSPARK block size + 1 (`8` at the default 7). ReplaySSM (`--enable-linear-replayssm-spec`) folds them into a per-slot ring, returning `D` to `0`.
- `state_bytes` — one state slot's bytes, from K3's fixed geometry, the attention-TP width, and the SSM dtype.
- `mla_kv_bytes` — one token's MLA latent KV bytes (KV-dtype dependent); DCP shards it across its ranks. The DSPARK draft model's KV (~1.4 KB per token) is replicated on every rank, so it enters flat — negligible without DCP, the same order as the sharded MLA share under DCP8.
- `L` — average total request length in tokens: input + output.
</Accordion>
<a id="playground" style={{ scrollMarginTop: "96px" }} />
## Advanced Features Playground
The Playground is where you experiment with **SGLang features beyond the deployment matrix**. The Deploy panel above emits the recipes the SGLang team is converging on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.
import { Playground } from "/src/snippets/_playground.jsx";
<Playground config={config} />
## 1. Model Introduction
**Kimi-K3** is Moonshot AI's flagship hybrid MoE vision-language model: **2.8 trillion parameters**, **16 of 896 experts** active per token, roughly **2.5× the scaling efficiency of Kimi-K2**. The backbone interleaves **Kimi Delta Attention (KDA)** with MLA across 93 layers (plus Attention Residuals and Stable LatentMoE); serving supports image input and a **1M-token** window with prefix caching. Weights ship in **MXFP4**: the FlashInfer MXFP4 (trtllm-gen SiTU) runner serves them on Blackwell, Marlin (W4A16) elsewhere, MegaMoE for short-context batch throughput.
K3 **always runs with thinking enabled**, with reasoning depth controlled by `reasoning_effort` (`low` / `high` / `max`; default `max`).
<Note>
Kimi-K3 is Moonshot AI's first open-source model in the trillion-plus class; **full model weights
are scheduled to release by July 27, 2026**. The recipes on this page were validated on the public
[`sgl-project/sglang` `kimi-k3` branch](https://github.com/sgl-project/sglang/tree/kimi-k3) — the HuggingFace
repository (`moonshotai/Kimi-K3`) and a public `lmsysorg/sglang` image with K3 support will be
available at launch.
Every cell in the Deploy panel above is currently marked **Final Verification In Progress**: the
recipe runs, but its serving round on the final weights and current code is still open. Re-measure
throughput and accuracy before you rely on any of them.
</Note>
**Recommended generation:** `temperature=1.0`, `top_p=0.95`, `presence_penalty=0`, `frequency_penalty=0` (fixed by the model; informational — do not hardcode in sample code).
**Resources:** [HuggingFace](https://huggingface.co/moonshotai/Kimi-K3) · [Kimi-K3 Quickstart](https://platform.kimi.ai/docs/guide/kimi-k3-quickstart).
## 2. Configuration Tips
**Memory: two pools, one flag.** K3 splits static memory into a worst-case-reserved **KDA state pool** (it sets the concurrency ceiling) and a paged **MLA KV pool**, divided by `--mamba-full-memory-ratio`. The command panel pins that flag to the [calculator](#mamba-ratio-calculator)'s output — set your average request length there; every other calculator input follows the panels. After boot, read back `max_total_num_tokens` (the KV side) and the admitted-request cap (the state side).
Capacity levers, all in the Playground. Each trades precision or cache behavior for capacity — re-verify accuracy on your workload:
| Lever | Effect |
|---|---|
| `--mamba-radix-cache-strategy extra_buffer_lazy` | 4 state slots per request instead of 5 |
| `--mamba-ssm-dtype bfloat16` | ~halves state bytes; with spec on, KDA verification falls back from the fused kernel to Triton |
| `--kv-cache-dtype fp8_e4m3` | halves KV bytes per token; under PD both roles must match at connect |
| `--mem-fraction-static` 0.90–0.92 | cheapest first win when the boot log shows a large idle `avail mem` |
| `SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1` | frees one more slot per request (experimental, under validation) |
Speculation: DSPARK holds block size + 1 (= 8) intermediate states per request — the calculator folds this in — and an unset `--max-running-requests` resets to 48 under spec (the command panel reminds you; set it explicitly to raise).
**MoE runner.** Leave `--moe-runner-backend` unset on Blackwell and it resolves to FlashInfer MXFP4 (W4A8, prebuilt trtllm-gen SiTU kernels) when the cubin pool is installed, Marlin (W4A16) otherwise; H100/H200 pin Marlin. The B200 Balanced and High-Throughput cells pin `flashinfer_mxfp4` explicitly because that is the shape they were brought up on — on an install without the pool, drop the flag to fall back to Marlin. The published Docker images already provision the **SiTU cubin pool**; to install it independently, run the same flow as the Dockerfile:
```bash
wget https://github.com/sgl-project/whl/releases/download/trtllm_gen_moe_cubin_20260617/trtllm_gen_moe_cubin_pool_20260617_v0613rc1.zip
sudo mkdir -p /opt/trtllm_gen_moe_cubin_pool
sudo unzip -q trtllm_gen_moe_cubin_pool_20260617_v0613rc1.zip -d /opt/trtllm_gen_moe_cubin_pool
export SGLANG_TRTLLM_GEN_MOE_CUBIN_POOL=/opt/trtllm_gen_moe_cubin_pool/trtllm_gen_moe_cubin_pool_20260617_v0613rc1
```
Remaining kernel sources JIT once from the public `flashinfer` wheel (a few minutes, cached).
**Attention backend.** Leave all three attention knobs unset on Blackwell: K3 resolves prefill, decode, and — under DSPARK — verification as a set (`trtllm_mla` across the board; `cutedsl_mla` takes decode and verification under DCP). On the non-DCP recipes, setting any one of the three cancels the auto-resolution for the others. The B200 Balanced and High-Throughput cells pin `--decode-attention-backend cutedsl_mla`, which is what auto-resolution picks for those DCP recipes anyway — it is written out because it is the shape they were brought up on, not because it changes the resolution. H100/H200 pin `flashmla` for decode.
**Context length.** `--context-length` bounds the longest accepted request plus some context-scaled buffers; it does not size the KV pool. For long context the lever that adds capacity is `fp8_e4m3` KV.
**DSPARK.** Adds `--speculative-algorithm DSPARK` plus the draft checkpoint on top of the showing strategy. Leave `--speculative-draft-attention-backend` unset. No serving round on the final draft checkpoint has landed — measure against the same recipe running NOSPEC before adopting.
**Per-platform notes:**
| Platform | Topology | Notes |
|---|---|---|
| B300 1×8 | TP8 (+DCP8) | accuracy-first defaults on Low-Latency and Balanced |
| GB300 2×4 | TP8/DCP8 | MNNVL transport and cuMem auto-detected |
| B200 2×8 | PP2 × TP8 on Low-Latency, PP2 × DCPEP8 on Balanced and High-Throughput. DSPARK re-lays the same 16 GPUs as TP16 / TP16+DCP16+EP16. PD prefill is TP1 × PP16 | Unified serves all three operating points; `Long-Context` is a `Prefill`-only strategy |
| GB200 4×4 | TP16/DCP16 | MNNVL auto-detected |
| H200 2×8 (4×8 on Unified High-Throughput) | TP16/EP16 + symm-mem, Marlin + FlashMLA; High-Throughput widens to TP32/EP32 over 4 nodes at mem-frac 0.90 with `extra_buffer_lazy` | same block on every node; export the cross-node NIC (`GLOO_SOCKET_IFNAME` / `NCCL_SOCKET_IFNAME`, `SGLANG_HOST_IP`); keep `NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1` |
| H100 4×8 | TP32/EP32, Marlin + FlashMLA | SM90a build of the K3 image; pin NCCL/Gloo to the same NIC on all nodes; least post-weight headroom (80 GB) |
| MI350X/MI355X 1×8 | TP8 ROCm/AITER | AITER A8W4 FlyDSL MoE, Triton attention, graph bs up to 256; DSPARK supported |
**DCP notes** — the DCP cells are Balanced and High-Throughput on every Blackwell platform, in both the `Unified` and `Decode` roles:
- DCP is the only axis that shards the TP-replicated MLA KV; Low-Latency skips it.
- Leave `--dcp-comm-backend` unset (fabric-resolved: `fi_a2a` on GB200/GB300, `a2a` on B200/B300).
- No `--enable-symm-mem` under DCP (force-disabled for decode-graph correctness).
- Explicit `tokenspeed_mla` force-rewrites `--kv-cache-dtype` to fp8; the default `cutedsl_mla` serves either dtype.
- Calculator ratios run well above 1 here (`r > 1` is legal): `bfloat16` state buys admission, `fp8` KV buys context.
- Don't use EP with an a2a backend: a2a buffers reclaim the KV that DCP buys. Compose only to measure. a2a backend is set when `SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1` or `--moe-a2a-backend` is set.
No cell has a serving round in this exact shape — treat them as starting points to verify.
## 3. Advanced Usage
### 3.1 Reasoning
K3 always thinks; the `kimi_k3` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) separates that thinking from the final answer — thinking lands in `message.reasoning_content`, the answer in `message.content`. Control the reasoning depth with `reasoning_effort` (`low` / `high` / `max`; default `max`).
<Accordion title="Reasoning Example (Python)">
```python Example
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
model="moonshotai/Kimi-K3",
messages=[{"role": "user", "content": "What is 15% of 240?"}],
reasoning_effort="high", # "low" | "high" | "max" (default max)
)
msg = resp.choices[0].message
print("Reasoning:", getattr(msg, "reasoning_content", None))
print("Answer:", msg.content)
```
</Accordion>
<Accordion title="Example Output">
```text Output
Pending update...
```
</Accordion>
### 3.2 Tool Calling
Enable the `kimi_k3` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. Because K3 is a thinking model, the follow-up turn may put text in `reasoning_content` as well as `content` — print both.
<Accordion title="Tool Calling Example (Python)">
```python Example
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
tools = [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a city",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
}]
resp = client.chat.completions.create(
model="moonshotai/Kimi-K3",
messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
tools=tools,
)
msg = resp.choices[0].message
print("Reasoning:", getattr(msg, "reasoning_content", None))
print("Tool calls:", msg.tool_calls)
```
</Accordion>
<Accordion title="Example Output">
```text Output
Pending update...
```
</Accordion>
### 3.3 HiCache (Hierarchical KV Caching)
K3's hybrid HiCache tiers the paged MLA KV **and** the KDA/mamba state across L1 (GPU) / L2 (host) / L3 (Mooncake) — enable it from the **HiCache** card in the [Playground above](#playground) for long multi-turn workloads.
- On the DCP recipes (Blackwell Balanced / High-Throughput, in both the `Unified` and `Decode` roles), the host tiers are not fully DCP-aware yet: **L3 always, and L1+L2 with Spec Decode on, drop the DCP flags** (the command hints call it out — per-request KV capacity shrinks accordingly). L1+L2 with Spec Decode off keeps DCP. Only DCP goes: the MLA KV reverts to TP-replicated, but the cell's other parallelism stays, so B300/GB300/GB200 land on plain TP while B200 Unified keeps its `--pp-size 2` / `--ep-size`.
- Low-Latency and the Hopper recipes take all tiers unchanged.
<a id="pd-disaggregation" />
### 3.4 PD Disaggregation
PD splits prefill and decode into separate server groups; because K3 is hybrid, the transfer moves **both** the paged MLA KV and the KDA recurrent state.
- **Transfer**: the cells emit **NiXL** (RDMA); Mooncake stays selectable in the Playground.
- **Ports**: prefill `30000`, decode `30100` (derived ZMQ/dist ranges must not collide on a shared host). The positional `8998` after `--prefill` must match `--disaggregation-bootstrap-port`, or only the decode worker registers.
- **Decode state pool**: chunk cache — one slot per request; `--mamba-radix-cache-strategy` is inert. Keep `--disaggregation-decode-extra-slots` pinned: unpinned it defaults to twice the batch below 32 requests and **zero** above.
#### Deep PP for prefill
Deep PP is `--tp-size 1` with one pipeline stage per GPU — `--pp-size 8` on B300/GB300, `--pp-size 16` on B200/GB200. Pipeline P2P overlaps the next microbatch's compute, unlike TP/EP collectives, and each stage owns whole layers (a clean slice of KV and state). `--tp-size 1` is also what buys context: above TP1 the MLA KV is replicated across the TP ranks, so TP2 × PP8 holds roughly half the tokens of TP1 × PP16 for the same memory.
- Use one stage per GPU; a shallow split still pays the in-stage all-reduce and can lose to flat TP.
- Pays only with several requests in flight. On the 8-GPU platforms that is why `Default` stays TP8; on the 16-GPU platforms deep PP wins at the Default operating point too, so both strategies use it — measured on GB200 at ISL 8192 / concurrency 32, PP16 × TP1 reached 4550 prefill tok/s/GPU vs 3596 (PP8 × TP2), 2407 (TEP16), and 1652 (TP16). Below concurrency ~8 the pipeline cannot fill and TEP16 leads instead (1947 vs 1227) — use `--tp-size 16 --ep-size 16` there.
- DSPARK off (`pp_size == 1` required) — on B200/GB200 that applies to `Default` as well.
- Fan one prefill role out to several decode roles; budget for in-transfer KV on the decode side.
<Accordion title="Router">
```bash Command
python3 -m sglang_router.launch_router \
--pd-disaggregation \
--prefill http://<prefill-host>:30000 8998 \
--decode http://<decode-host>:30100 \
--host 0.0.0.0 --port 8000 \
--disable-circuit-breaker \
--health-check-interval-secs 999999
```
</Accordion>
Clients then send requests to the router (`:8000`) instead of an individual role server.
### 3.5 VLM Serving Profiles
The open-source K3 serving contract currently supports **image input only** — its
processor rejects video and audio input.
#### Recommended high-speed VLM
The command panel now opens on the **B300 · Unified · Balanced**
recipe below. It makes the VLM-specific performance choices explicit:
```bash Command
sglang serve \
--trust-remote-code \
--model-path moonshotai/Kimi-K3 \
--tp-size 8 \
--dcp-size 8 \
--mem-fraction-static 0.85 \
--mm-feature-transport cuda_ipc \
--mm-processor-worker-num 2 \
--mm-io-worker-num 16 \
--reasoning-parser kimi_k3 \
--tool-call-parser kimi_k3 \
--host 0.0.0.0 \
--port 30000
```
- `--mm-feature-transport cuda_ipc` — single-node only: skips the CPU round trip, bounded pool (per-tensor CPU fallback when full), reserves up to `SGLANG_MM_FEATURE_CACHE_MB` on the base GPU. Multi-node recipes use CPU transport.
- 2 processor / 16 I/O workers are the measured defaults; more adds contention.
- Leave `--mm-attention-backend` unset — auto-selected, with a correctness fallback.
- Don't add `--mm-enable-dp-encoder`; K3 already shards images across TP ranks.
#### VLM compatibility
| Feature | K3 behavior |
|---|---|
| PD | Supported. Image processing and ViT run on prefill; the PD transfer then moves both paged MLA KV and KDA recurrent state as described in [PD disaggregation](#pd-disaggregation). |
| EPD | Supported on the public `kimi-k3` branch. Use an `--encoder-only` vision role and a `--language-only` prefill role; add the normal decode role for full EPD. See the [EPD guide](../../../docs/advanced_features/epd_disaggregation). |
| MM encoder DP | Built in. K3 shards complete images across TP ranks, so leave `--mm-enable-dp-encoder` unset in unified, PD-prefill, and encoder-only roles. |
| CUDA IPC | Compatible with the local processor-to-scheduler path on a single-node unified or PD-prefill role. It does not replace `--encoder-transfer-backend` for EPD or the PD KV/KDA transfer, and its bounded pool consumes HBM. |
| ViT BCG | Compatible with unified and encoder-only roles, but recommended only for repeated encoder shapes after measuring the HBM trade-off below. |
#### Should ViT BCG be enabled?
Keep ViT BCG **off** for general serving; enable `SGLANG_VIT_ENABLE_CUDA_GRAPH=1` only for ViT-only / EPD encoder workloads with recurring image shapes and spare HBM.
- The win is confined to the encoder — no reliable end-to-end TTFT/TPOT gain in full-model serving.
- Each captured graph retains HBM (graph + per-entry metadata); measure on your own shapes.
- The default cache captures after two hits and falls back to eager above 6,144 tokens; do not enlarge it without measuring.
#### Low-HBM VLM
Use this profile when keeping HBM headroom matters more than peak concurrency.
It removes the 1 GiB CUDA IPC pool, keeps ViT BCG disabled, halves the context
window, caps concurrency, and lowers the static-memory target:
```bash Command
SGLANG_VIT_ENABLE_CUDA_GRAPH=0 \
sglang serve \
--trust-remote-code \
--model-path moonshotai/Kimi-K3 \
--tp-size 8 \
--context-length 65536 \
--enable-symm-mem \
--mem-fraction-static 0.82 \
--mm-feature-transport cpu \
--mm-processor-worker-num 2 \
--mm-io-worker-num 16 \
--reasoning-parser kimi_k3 \
--tool-call-parser kimi_k3 \
--host 0.0.0.0 \
--port 30000
```
`--mem-fraction-static 0.82` is a conservative B300 starting point, not a portable minimum: raise it toward `0.85` if startup reports insufficient memory; if HBM must go back to other workloads, reduce context/concurrency first. The precision levers (`fp8_e4m3` KV, `bfloat16` SSM state) save far more but stay accuracy-gated.
<a id="large-scale-presets" />
### 3.6 Large-Scale Serving Presets (16–64 GPUs, Blackwell)
**The KDA state pool is the concurrency ceiling** — DP, EP, and DCP do not shard it; only attention-TP width, SSM dtype, and cache strategy change the per-GPU bill. The MLA KV is cheap to shrink (fp8) or deduplicate (DCP).
Two presets come out of this, at `N = 8k` GPUs:
| Preset | What it trades | Pick it for |
|---|---|---|
| **Peak Throughput** — `dp = k`, attention-TP 8 | State shards 8-way. The per-step KDA all-reduce stays within one 8-GPU B200/B300 node, or spans two 4-GPU GB200/GB300 nodes over MNNVL. `--kv-cache-dtype fp8_e4m3` is load-bearing — bf16 KV does not fit 128 requests per replica. | Maximum sustained TPS — the default large-scale shape. |
| **Peak Capacity (+DCP8)** — `dp = k` + `--dcp-size 8` | Deduplicates the attention-TP group's MLA KV: concurrency ceiling +72% at the same engine throughput, ~1.8× ITL. | Context ≥ ~16K, or per-replica concurrency past 128. |
- **Radix cache** is independent of the preset: for prefix-free traffic (offline batch, evals) switch it off (Playground's **Prefix Cache** card) — one state slot per request instead of 4–5.
- The fully data-parallel extreme (`--dp-size` = GPU count, attention-TP 1) — the shape behind the 64-GPU sweep's ~3K tok/s per GPU — is not a preset: 288 GB GPUs only, radix forced off, no head-to-head against the preset shape.
The Peak Throughput preset at 32 GPUs on B200/B300 (4 nodes × 8; every node runs the same command with its own `--node-rank`). On GB200/GB300 the same 32-GPU shape uses 8 nodes × 4, and the Playground emits `--nnodes 8`:
```bash Command
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=20480 \
sglang serve \
--trust-remote-code \
--model-path moonshotai/Kimi-K3 \
--tp-size 32 --ep-size 32 \
--enable-dp-attention --dp-size 4 --enable-dp-lm-head \
--nnodes 4 --node-rank <rank> --dist-init-addr <node0-ip>:20000 \
--moe-a2a-backend megamoe --moe-runner-backend deep_gemm \
--kv-cache-dtype fp8_e4m3 \
--mamba-ssm-dtype bfloat16 \
--mamba-radix-cache-strategy extra_buffer_lazy \
--mem-fraction-static 0.92 \
--reasoning-parser kimi_k3 --tool-call-parser kimi_k3 \
--host 0.0.0.0 --port 30000
```
Scale by holding the per-replica shape fixed and moving only the replica count; pool sizing rides the calculator-driven `--mamba-full-memory-ratio`, which folds in DP, DCP, precision, and speculation:
| GPUs | B200/B300 nodes | GB200/GB300 nodes | `--tp-size` / `--ep-size` | `--dp-size` |
|---|---|---|---|---|
| 16 | 2×8 | 4×4 | 16 | 2 |
| 32 | 4×8 | 8×4 | 32 | 4 |
| 64 | 8×8 | 16×4 | 64 | 8 |
For Peak Capacity, add `--dcp-size 8` and re-derive the pool split with the [Mamba ratio calculator](#mamba-ratio-calculator).
Both presets are one click away in the [Playground above](#playground): pick a **Cluster Size** and a **Large-Scale Preset** and the full command composes onto whichever cell is showing.
Decisions the preset already makes:
- **MegaMoE on `deep_gemm`** — the fastest a2a backend; needs the SiTU cubin pool ([§2](#2-configuration-tips)).
- **SP-MoE and shared-expert overlap** engage automatically under EP a2a; the K3 all-reduce fusion does not.
- **Spec Decode follows the Deploy knob.** Acceptance thins at large batch; spec × EP × DP-attention is validated only at 8-GPU EP8 × DP2 (full GSM8K) — experimental at these scales.
<Note>
No preset has a full serving round on final weights; the constants derive from measured single- and dual-node rounds plus a 64-GPU sweep. Validate throughput and accuracy on your workload before committing a fleet.
</Note>
docs/cookbook/autoregressive/Moonshotai/Kimi-Linear.mdx
+297
View File
@@ -0,0 +1,297 @@
---
title: Kimi-Linear
metatags:
description: "Deploy Kimi-Linear with SGLang - community contribution guide for Moonshot AI's Kimi-Linear model deployment."
---
import { KimiLinearDeployment } from '/src/snippets/autoregressive/kimi-linear-deployment.jsx';
## AMD GPU Support
## 1. Model Introduction
Kimi Linear is a hybrid linear attention architecture that outperforms traditional full attention methods across various contexts, including short, long, and reinforcement learning (RL) scaling regimes. At its core is Kimi Delta Attention (KDA)—a refined version of Gated DeltaNet that introduces a more efficient gating mechanism to optimize the use of finite-state RNN memory.
This generation delivers comprehensive upgrades across the board:
Kimi Delta Attention (KDA): A linear attention mechanism that refines the gated delta rule with finegrained gating.
Hybrid Architecture: A 3:1 KDA-to-global MLA ratio reduces memory usage while maintaining or surpassing the quality of full attention.
Superior Performance: Outperforms full attention in a variety of tasks, including long-context and RL-style benchmarks on 1.4T token training runs with fair comparisons.
High Throughput: Achieves up to 6× faster decoding and significantly reduces time per output token (TPOT).
For more details, please refer to the [official Kimi Linear GitHub Repository]: https://github.com/MoonshotAI/Kimi-Linear
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
## 3. Model Deployment
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
<KimiLinearDeployment />
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
### 4.2 Advanced Usage
#### 4.2.1 Launch the docker
```shell Command
docker pull lmsysorg/sglang:v0.5.7-rocm700-mi30x
```
```shell Command
docker run -d -it --ipc=host --network=host --privileged \
--cap-add=CAP_SYS_ADMIN \
--device=/dev/kfd --device=/dev/dri --device=/dev/mem \
--group-add video --cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
-v /:/work \
-e SHELL=/bin/bash \
--name Kimi-linear \
lmsysorg/sglang:v0.5.7-rocm700-mi30x \
/bin/bash
```
#### 4.2.2 pre-installation steps inside the docker
```shell Command
pip install sentencepiece tiktoken
```
#### 4.2.3 Launch the server
```shell Command
export SGLANG_ROCM_FUSED_DECODE_MLA=0
SGLANG_ROCM_FUSED_DECODE_MLA=0 python3 -m sglang.launch_server \
--model-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
--tokenizer-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
--tp 4 \
--trust-remote-code
```
## 5. Benchmark
### 5.1 Speed Benchmark
Test Environment:
Hardware: AMD MI300X GPU
Model: Kimi-Linear-48B-A3B-Instruct
Tensor Parallelism: 4
sglang version: 0.5.7
- **Model Deployment**
```bash Command
SGLANG_ROCM_FUSED_DECODE_MLA=0 python3 -m sglang.launch_server \
--model-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
--tokenizer-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
--tp 4 \
--trust-remote-code
```
### 5.1.1 Low Concurrency (Latency-Optimized)
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model moonshotai/Kimi-Linear-48B-A3B-Instruct \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 23.86
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4001
Request throughput (req/s): 0.42
Input token throughput (tok/s): 255.70
Output token throughput (tok/s): 176.86
Peak output token throughput (tok/s): 190.00
Peak concurrent requests: 2
Total token throughput (tok/s): 432.56
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 2383.93
Median E2E Latency (ms): 1911.63
---------------Time to First Token----------------
Mean TTFT (ms): 141.33
Median TTFT (ms): 126.27
P99 TTFT (ms): 294.76
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 5.32
Median TPOT (ms): 5.33
P99 TPOT (ms): 5.36
---------------Inter-Token Latency----------------
Mean ITL (ms): 5.33
Median ITL (ms): 5.32
P95 ITL (ms): 5.44
P99 ITL (ms): 5.58
Max ITL (ms): 11.46
==================================================
```
### 5.1.2 Medium Concurrency (Balanced)
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model moonshotai/Kimi-Linear-48B-A3B-Instruct \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 31.38
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 40805
Total generated tokens (retokenized): 39667
Request throughput (req/s): 2.55
Input token throughput (tok/s): 1264.13
Output token throughput (tok/s): 1300.37
Peak output token throughput (tok/s): 1801.00
Peak concurrent requests: 21
Total token throughput (tok/s): 2564.50
Concurrency: 14.13
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 5543.18
Median E2E Latency (ms): 5755.31
---------------Time to First Token----------------
Mean TTFT (ms): 175.25
Median TTFT (ms): 137.87
P99 TTFT (ms): 292.92
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 10.75
Median TPOT (ms): 10.87
P99 TPOT (ms): 16.74
---------------Inter-Token Latency----------------
Mean ITL (ms): 10.54
Median ITL (ms): 7.95
P95 ITL (ms): 13.68
P99 ITL (ms): 116.80
Max ITL (ms): 299.89
==================================================
```
### 5.1.3 High Concurrency (Throughput-Optimized)
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model moonshotai/Kimi-Linear-48B-A3B-Instruct \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 79.71
Total input tokens: 249831
Total input text tokens: 249831
Total input vision tokens: 0
Total generated tokens: 252662
Total generated tokens (retokenized): 228448
Request throughput (req/s): 6.27
Input token throughput (tok/s): 3134.20
Output token throughput (tok/s): 3169.72
Peak output token throughput (tok/s): 6109.00
Peak concurrent requests: 110
Total token throughput (tok/s): 6303.92
Concurrency: 94.80
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 15113.92
Median E2E Latency (ms): 13851.52
---------------Time to First Token----------------
Mean TTFT (ms): 564.46
Median TTFT (ms): 226.04
P99 TTFT (ms): 2683.14
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 29.63
Median TPOT (ms): 31.28
P99 TPOT (ms): 38.84
---------------Inter-Token Latency----------------
Mean ITL (ms): 28.85
Median ITL (ms): 16.29
P95 ITL (ms): 123.42
P99 ITL (ms): 157.80
Max ITL (ms): 2481.11
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 GSM8K Benchmark
- Server Command
```shell Command
SGLANG_ROCM_FUSED_DECODE_MLA=0 python3 -m sglang.launch_server \
--model-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
--tokenizer-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
--tp 4 \
--trust-remote-code
```
- Benchmark Command
```shell Command
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
```
- **Result**:
```text Output
Accuracy: 0.705
Invalid: 0.000
Latency: 11.855 s
Output throughput: 3224.982 token/s
```