minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

Add new Mintlify documentation site (docs_new/) (#23001)

Browse Source 
Co-authored-by: AdityaVKochar <adityavardhankochar@gmail.com>
Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com>
Co-authored-by: adhyan-jain <adhyanjain2006@gmail.com>
Co-authored-by: Adhyan Jain <71976554+adhyan-jain@users.noreply.github.com>
Co-authored-by: Maitri-shah29 <maitrirajivshah@gmail.com>
Co-authored-by: Adarsh Shirawalmath <114558126+adarshxs@users.noreply.github.com>
Co-authored-by: Maitri Shah <shah29maitri@gmail.com>
Co-authored-by: Aditya Vardhan Kochar <80113212+AdityaVKochar@users.noreply.github.com>
Co-authored-by: Rishit Shivam <164783543+pokymono@users.noreply.github.com>
Co-authored-by: Rishitshivam <164783543+Rishitshivam@users.noreply.github.com>
Co-authored-by: IshhanKheria <ishhankheria06@gmail.com>
Co-authored-by: Ishita Joshi <ishitata.joshi@gmail.com>
Co-authored-by: Richard Chen <104477092+Richardczl98@users.noreply.github.com>
Co-authored-by: longGGGGGG <553746008@qq.com>
Co-authored-by: Richard <richardchen@radixark.ai>
Co-authored-by: Nakul Sinha <nakul.new4socials@gmail.com>
Co-authored-by: Divyam Agrawal <ludicrouslytrue@gmail.com>
Co-authored-by: Richardczl98 <Zhenlinc@stanford.edu>
Co-authored-by: Krishang Zinzuwadia <krishangzinzuwadia@gmail.com>
Co-authored-by: nimeshas <nimesha.s106@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jignas Paturu <86356085+JignasP@users.noreply.github.com>
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
This commit is contained in:
Mingyi
2026-04-20 15:10:22 -07:00
committed by GitHub
co-authored by AdityaVKochar mintlify[bot] adhyan-jain Adhyan Jain Maitri-shah29 Adarsh Shirawalmath Maitri Shah Aditya Vardhan Kochar Rishit Shivam Rishitshivam IshhanKheria Ishita Joshi Richard Chen longGGGGGG Richard Nakul Sinha Divyam Agrawal Richardczl98 Krishang Zinzuwadia nimeshas Claude Opus 4.6 github-actions[bot] Jignas Paturu zijiexia
parent 575fdc2c4c
commit a3291b5654
330 changed files with 100371 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-Math-V2.mdx
+522
View File
@@ -0,0 +1,522 @@
---
title: DeepSeek-Math-V2
metatags:
description: "Deploy DeepSeek-Math-V2 with SGLang - advanced mathematical reasoning model with gold-level IMO/CMO performance and theorem-proving capabilities."
---
import { DeepSeekMathV2Deployment } from '/src/snippets/autoregressive/deepseek-math-v2-deployment.jsx';
## 1. Model Introduction
[DeepSeek-Math-V2](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2) is DeepSeek's advanced mathematical reasoning model with strong theorem-proving capabilities. The model demonstrates exceptional performance on mathematical competitions, achieving gold-level scores on IMO 2025 and CMO 2024, and a near-perfect 118/120 on Putnam 2024 with scaled test-time compute.
**Key Features:**
- **Strong Theorem-Proving**: Gold-level performance on IMO 2025 and CMO 2024
- **Self-Verifiable Reasoning**: Implements self-verifiable mathematical reasoning for improved accuracy
- **Competition-Level Math**: Near-perfect score (118/120) on Putnam 2024
- **Large MoE Model**: ~671B total parameters, requires high-memory GPUs (B200 183GB or B300 275GB)
**Available Models:**
- **BF16 (Full Weights)**: [deepseek-ai/DeepSeek-Math-V2](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2) - Full precision weights
**License:**
To use DeepSeek-Math-V2, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2/blob/main/LICENSE) for details.
## 2. SGLang Installation
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy.
<DeepSeekMathV2Deployment />
### 3.2 Configuration Tips
**Hardware Requirements:**
- **B200 (183GB)**: BF16 tp=8
- **B300 (275GB)**: BF16 tp=8
**DP Attention:**
- Enable DP attention for high-throughput scenarios
- The `--dp` value commonly matches the `--tp` value
- Trade-off: Higher throughput at the cost of slightly increased latency
## 4. Model Invocation
### 4.1 Deployment Command
Deploy the model using the command generated above. Example for B200:
```shell Command
sglang serve --model-path deepseek-ai/DeepSeek-Math-V2 \
--tp 8 \
--ep 8 \
--reasoning-parser deepseek-r1 \
--host 0.0.0.0 \
--port 30000
```
### 4.2 Mathematical Reasoning
DeepSeek-Math-V2 excels at mathematical problem-solving with step-by-step reasoning.
**Streaming with Thinking Process:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Mathematical reasoning problem
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-Math-V2",
messages=[
{"role": "user", "content": "Prove that for any positive integer n, the sum 1 + 2 + 3 + ... + n = n(n+1)/2"}
],
max_tokens=4096,
stream=True
)
# Process the stream
thinking_started = False
has_thinking = False
has_answer = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
We need to prove that for any positive integer n, the sum 1 + 2 + 3 + ... + n = n(n+1)/2.
This is a classic formula for the sum of the first n natural numbers. We can prove by induction.
Base case: n=1, LHS = 1, RHS = 1*(1+1)/2 = 1*2/2 = 1. Holds.
Inductive step: Assume true for n = k, i.e., 1 + 2 + ... + k = k(k+1)/2. Then for n = k+1, sum = 1 + 2 + ... + k + (k+1) = [k(k+1)/2] + (k+1) = (k(k+1) + 2(k+1))/2 = (k+1)(k+2)/2 = (k+1
)((k+1)+1)/2. So holds for k+1. By induction, holds for all positive integers n.
...
=============== Content =================
We can prove the well-known formula for the sum of the first \(n\) positive integers in several ways. Two of the most elementary are presented below.
---
### 1. Proof by mathematical induction
**Base case (\(n=1\))**:
\[
1 = \frac{1\cdot(1+1)}{2}= \frac{1\cdot2}{2}=1,
\]
so the formula holds for \(n=1\).
**Inductive hypothesis:**
Assume that for some positive integer \(k\) the formula is true, i.e.
\[
1+2+\dots+k = \frac{k(k+1)}{2}.
\]
**Inductive step (\(k \to k+1\))**:
Consider the sum up to \(k+1\):
\[
\begin{aligned}
1+2+\dots+k+(k+1) &= \bigl(1+2+\dots+k\bigr) + (k+1) \\[4pt]
&= \frac{k(k+1)}{2} + (k+1) \qquad\text{(by the induction hypothesis)}\\[4pt]
&= (k+1)\left(\frac{k}{2}+1\right)\\[4pt]
&= (k+1)\frac{k+2}{2}\\[4pt]
&= \frac{(k+1)(k+2)}{2}\\[4pt]
&= \frac{(k+1)\bigl((k+1)+1\bigr)}{2}.
\end{aligned}
\]
Thus the formula also holds for \(n=k+1\).
By the principle of mathematical induction,
\[
1+2+3+\dots+n = \frac{n(n+1)}{2}
\]
for every positive integer \(n\).
---
### 2. Proof by pairing (Gauss’s trick)
Let
\[
S = 1 + 2 + 3 + \dots + n.
\]
Write the same sum in reverse order:
\[
S = n + (n-1) + (n-2) + \dots + 1.
\]
Add the two equalities term‑by‑term:
\[
\begin{aligned}
2S &= (1+n) + \bigl(2+(n-1)\bigr) + \bigl(3+(n-2)\bigr) + \dots + (n+1)\\
&= \underbrace{(n+1)+(n+1)+\dots+(n+1)}_{n\ \text{times}}\\
&= n\,(n+1).
\end{aligned}
\]
Therefore
\[
S = \frac{n(n+1)}{2}.
\]
Both proofs are rigorous and show that the formula holds for all positive integers \(n\).
```
### 4.3 Competition-Level Problems
**Example: IMO-style Problem:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# IMO-style problem
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-Math-V2",
messages=[
{"role": "user", "content": "Let a, b, c be positive real numbers such that abc = 1. Prove that (a-1+1/b)(b-1+1/c)(c-1+1/a) <= 1."}
],
max_tokens=8192,
stream=True
)
# Process the stream
thinking_started = False
has_thinking = False
has_answer = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
if delta.content:
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
We need to prove that for positive real numbers a,b,c with abc = 1, we have:
\[
(a - 1 + \frac{1}{b})(b - 1 + \frac{1}{c})(c - 1 + \frac{1}{a}) \le 1.
\]
We can rewrite the expressions: Since abc=1, we have 1/b = ac, 1/c = ab, 1/a = bc. Wait careful: abc=1 => 1/b = ac? Actually 1/b = ac? Let's check: abc=1 => ac = 1/b? Multiply both sides by something: abc=1 => (ac) b = 1 => ac = 1/b. Yes, because (ac) * b = 1 => ac = 1/b. Similarly, ab = 1/c, bc = 1/a. So we can rewrite:
...
=============== Content =================
We are given positive real numbers \(a,b,c\) with \(abc=1\). We must prove
\[
\Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr)\le 1 .
\]
---
### 1. A convenient substitution
Because \(abc=1\), we can write
\[
a=\frac{x}{y},\qquad b=\frac{y}{z},\qquad c=\frac{z}{x}
\]
with positive numbers \(x,y,z\).
(For instance, take \(x=1,\;y=\frac1a,\;z=\frac1{ab}\); then indeed \(a=\frac{x}{y},\;b=\frac{y}{z}\) and, using \(abc=1\), we obtain \(c=\frac{z}{x}=\frac1{ab}=c\).)
---
### 2. Rewriting the factors
\[
\begin{aligned}
a-1+\frac1b &=\frac{x}{y}-1+\frac{z}{y}= \frac{x+z-y}{y},\\[2mm]
b-1+\frac1c &=\frac{y}{z}-1+\frac{x}{z}= \frac{x+y-z}{z},\\[2mm]
c-1+\frac1a &=\frac{z}{x}-1+\frac{y}{x}= \frac{y+z-x}{x}.
\end{aligned}
\]
Hence the product becomes
\[
P=\Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr)
=\frac{(x+z-y)(x+y-z)(y+z-x)}{xyz}.
\]
---
### 3. Reducing to a known inequality
We have to show \(P\le1\), i.e.
\[
(x+z-y)(x+y-z)(y+z-x)\le xyz .
\tag{1}
\]
Set
\[
p=x+y+z,\qquad q=xy+yz+zx,\qquad r=xyz .
\]
Notice that
\[
x+z-y=p-2y,\quad x+y-z=p-2z,\quad y+z-x=p-2x .
\]
Therefore
\[
\begin{aligned}
(x+z-y)(x+y-z)(y+z-x)
&=(p-2x)(p-2y)(p-2z)\\
&=p^{3}-2p^{2}(x+y+z)+4p(xy+yz+zx)-8xyz\\
&=-p^{3}+4pq-8r .
\end{aligned}
\]
Inequality (1) is thus equivalent to
\[
-p^{3}+4pq-8r\le r\quad\Longleftrightarrow\quad 4pq-p^{3}\le 9r .
\tag{2}
\]
---
### 4. Applying Schur’s inequality
Schur’s inequality of third degree states that for any non‑negative \(x,y,z\)
\[
p^{3}+9r\ge 4pq .
\]
Rearranged, this is exactly \(4pq-p^{3}\le 9r\), which is (2).
Since our \(x,y,z\) are positive, Schur’s inequality applies and (2) holds.
Consequently (1) is true, and we obtain \(P\le1\).
---
### 5. Equality case
Equality in Schur’s inequality for positive numbers occurs only when \(x=y=z\).
Then \(a=b=c=1\), and indeed the product equals \(1\).
---
Thus for all positive \(a,b,c\) with \(abc=1\),
\[
\Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr)\le 1 .
\]
∎
```
## 5. Benchmark
### 5.1 Accuracy Benchmark
#### 5.1.1 GSM8K Benchmark
**Benchmark Command:**
```shell Command
python3 benchmark/gsm8k/bench_sglang.py --num-questions 200 --port 30000
```
**Test Results:**
```text Output
Accuracy: 0.975
Invalid: 0.000
Latency: 34.358 s
Output throughput: 540.162 token/s
```
### 5.2 Speed Benchmark
**Test Environment:**
- Hardware: NVIDIA B200 GPU (8x, 183GB each)
- Model: DeepSeek-Math-V2
- Tensor Parallelism: 8
- SGLang Version: 0.5.8
#### 5.2.1 Latency Benchmark
**Benchmark Command:**
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model deepseek-ai/DeepSeek-Math-V2 \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 10 \
--max-concurrency 1
```
**Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 53.34
Total input tokens: 1972
Total input text tokens: 1972
Total generated tokens: 2784
Total generated tokens (retokenized): 2778
Request throughput (req/s): 0.19
Input token throughput (tok/s): 36.97
Output token throughput (tok/s): 52.19
Peak output token throughput (tok/s): 56.00
Peak concurrent requests: 3
Total token throughput (tok/s): 89.16
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 5330.72
Median E2E Latency (ms): 5879.28
P90 E2E Latency (ms): 8320.33
P99 E2E Latency (ms): 9921.29
---------------Time to First Token----------------
Mean TTFT (ms): 183.38
Median TTFT (ms): 177.92
P99 TTFT (ms): 217.64
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 17.96
Median TPOT (ms): 18.39
P99 TPOT (ms): 19.03
---------------Inter-Token Latency----------------
Mean ITL (ms): 18.57
Median ITL (ms): 18.63
P95 ITL (ms): 19.26
P99 ITL (ms): 19.48
Max ITL (ms): 24.93
==================================================
```
#### 5.2.2 Throughput Benchmark
**Benchmark Command:**
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model deepseek-ai/DeepSeek-Math-V2 \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 1000 \
--max-concurrency 100
```
**Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 217.36
Total input tokens: 301701
Total input text tokens: 301701
Total generated tokens: 188375
Total generated tokens (retokenized): 187456
Request throughput (req/s): 4.60
Input token throughput (tok/s): 1388.05
Output token throughput (tok/s): 866.67
Peak output token throughput (tok/s): 2589.00
Peak concurrent requests: 109
Total token throughput (tok/s): 2254.72
Concurrency: 89.81
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 19521.73
Median E2E Latency (ms): 12076.76
P90 E2E Latency (ms): 47248.87
P99 E2E Latency (ms): 86862.79
---------------Time to First Token----------------
Mean TTFT (ms): 790.40
Median TTFT (ms): 456.81
P99 TTFT (ms): 4223.33
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 106.52
Median TPOT (ms): 107.24
P99 TPOT (ms): 238.33
---------------Inter-Token Latency----------------
Mean ITL (ms): 100.29
Median ITL (ms): 38.34
P95 ITL (ms): 237.00
P99 ITL (ms): 347.49
Max ITL (ms): 3642.56
==================================================
```
docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-OCR-2.mdx
+250
View File
@@ -0,0 +1,250 @@
---
title: DeepSeek-OCR-2
metatags:
description: "Deploy DeepSeek-OCR-2 with SGLang - high-accuracy text extraction from images and documents for OCR tasks."
---
import { DeepSeekOCR2Deployment } from '/src/snippets/autoregressive/deepseek-ocr-v2-deployment.jsx';
## 1. Model Introduction
[DeepSeek-OCR-2](https://github.com/deepseek-ai/DeepSeek-OCR-2) is DeepSeek's next-generation OCR (Optical Character Recognition) model, building on DeepSeek-OCR with improved accuracy and broader document understanding capabilities. The model is optimized for high-accuracy text extraction from images across a wide variety of document types and formats.
**Key Features:**
- **Semantic-Aware Visual Encoding (DeepEncoder V2)**: DeepSeek-OCR-2 introduces DeepEncoder V2, which models document reading order in a more human-like, semantic-driven manner rather than relying on fixed raster scanning. This significantly improves logical reading flow in complex layouts (e.g., multi-column documents).
- **Stronger Layout and Structural Understanding**: DeepSeek-OCR-2 demonstrates improved performance on structured documents such as tables, forms, and dense multi-column pages. It reduces reading-order errors and improves overall document parsing robustness compared to the original version.
- **Improved Accuracy While Maintaining Token Efficiency**: The original DeepSeek-OCR emphasized aggressive visual token compression. OCR-2 maintains high token efficiency while delivering higher benchmark performance, particularly on document-level understanding tasks.
- **Better Generalization Across Complex Document Tasks**: DeepSeek-OCR-2 performs more consistently across multilingual documents, structured data extraction, and visually complex content, making it more suitable for real-world document intelligence scenarios beyond plain text OCR.
**Available Models:**
- **Base Model**: [deepseek-ai/DeepSeek-OCR-2](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2) - Recommended for OCR tasks
**License:**
To use DeepSeek-OCR-2, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2/blob/main/LICENSE.txt) for details.
For more details, please refer to the [official DeepSeek-OCR-2 repository](https://github.com/deepseek-ai/DeepSeek-OCR-2).
## 2. SGLang Installation
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy. SGLang supports serving DeepSeek-OCR-2 on NVIDIA H200 and B200, and AMD MI300X, MI355X, and MI325X GPUs.
<DeepSeekOCR2Deployment />
**Note**: DeepSeek-OCR-2 has ~3.58B parameters and easily fits on a single modern GPU. For low-latency serving, no model parallelism is needed. For high-throughput requirements, consider using data parallelism with the SGLang Model Gateway — see [DP, DPA and SGLang DP Router](../../../docs/advanced_features/sgl_model_gateway) for more details.
### 3.2 Configuration Tips
For more detailed configuration tips, please refer to [DeepSeek V3/V3.1/R1 Usage](../../../docs/basic_usage/deepseek_v3).
## 4. Model Invocation
### 4.1 Basic Usage
**OpenAI-compatible request example**
```python Example
import requests
url = "http://localhost:30000/v1/chat/completions"
data = {
"model": "deepseek-ai/DeepSeek-OCR-2",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "<image>\n<|grounding|>Convert the document to markdown."},
{"type": "image_url", "image_url": {"url": "https://example.com/your_image.jpg"}},
],
}
],
"max_tokens": 512,
}
response = requests.post(url, json=data)
print(response.text)
```
**Reference**
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Recommended Prompts
The following prompts are recommended by the [official model card](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2#main-prompts).
**Structured document conversion** — extracts text while preserving layout:
```text Example
<image>
<|grounding|>Convert the document to markdown.
```
**Free-form OCR** — extracts without layouts:
```text Example
<image>
Free OCR.
```
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: NVIDIA H200 GPU (1x)
- Model: DeepSeek-OCR-2
- Tensor Parallelism: 1
- sglang version: 0.0.0.dev1+g93fca0bbc
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses. For more details on how to perform evaluation, see [Evaluating New Models with SGLang](../../../docs/developer_guide/evaluating_new_models).
#### 5.1.1 Latency-Sensitive Benchmark
- Model Deployment Command:
```shell Command
sglang serve \
--model-path deepseek-ai/DeepSeek-OCR-2 \
--enable-multimodal \
--host 0.0.0.0 \
--port 30000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 0.0.0.0 \
--port 30000 \
--model deepseek-ai/DeepSeek-OCR-2 \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 10 \
--max-concurrency 1
```
- **Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 3.54
Total input tokens: 1972
Total input text tokens: 1972
Total generated tokens: 2784
Total generated tokens (retokenized): 2710
Request throughput (req/s): 2.83
Input token throughput (tok/s): 557.53
Output token throughput (tok/s): 787.10
Peak output token throughput (tok/s): 818.00
Peak concurrent requests: 5
Total token throughput (tok/s): 1344.63
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 352.69
Median E2E Latency (ms): 392.34
P90 E2E Latency (ms): 540.64
P99 E2E Latency (ms): 639.01
---------------Time to First Token----------------
Mean TTFT (ms): 18.08
Median TTFT (ms): 16.57
P99 TTFT (ms): 25.67
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 1.18
Median TPOT (ms): 1.21
P99 TPOT (ms): 1.22
---------------Inter-Token Latency----------------
Mean ITL (ms): 1.21
Median ITL (ms): 1.21
P95 ITL (ms): 1.28
P99 ITL (ms): 1.44
Max ITL (ms): 4.32
==================================================
```
#### 5.1.2 Throughput-Sensitive Benchmark
- Model Deployment Command:
```shell Command
sglang serve \
--model-path deepseek-ai/DeepSeek-OCR-2 \
--enable-multimodal \
--tp 1 \
--ep 1 \
--dp 1 \
--enable-dp-attention \
--host 0.0.0.0 \
--port 30000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 0.0.0.0 \
--port 30000 \
--model deepseek-ai/DeepSeek-OCR-2 \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 1000 \
--max-concurrency 100
```
- **Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 14.79
Total input tokens: 301698
Total input text tokens: 301698
Total generated tokens: 188375
Total generated tokens (retokenized): 185236
Request throughput (req/s): 67.63
Input token throughput (tok/s): 20402.54
Output token throughput (tok/s): 12738.99
Peak output token throughput (tok/s): 17508.00
Peak concurrent requests: 187
Total token throughput (tok/s): 33141.53
Concurrency: 86.87
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 1284.50
Median E2E Latency (ms): 866.07
P90 E2E Latency (ms): 3027.32
P99 E2E Latency (ms): 5490.63
---------------Time to First Token----------------
Mean TTFT (ms): 86.08
Median TTFT (ms): 50.09
P99 TTFT (ms): 613.92
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 7.79
Median TPOT (ms): 6.54
P99 TPOT (ms): 50.10
---------------Inter-Token Latency----------------
Mean ITL (ms): 6.42
Median ITL (ms): 4.64
P95 ITL (ms): 23.65
P99 ITL (ms): 39.62
Max ITL (ms): 452.65
==================================================
```
docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-OCR.mdx
+204
View File
@@ -0,0 +1,204 @@
---
title: DeepSeek-OCR
metatags:
description: "Deploy DeepSeek-OCR with SGLang - high-accuracy text extraction from images and documents for OCR tasks."
---
## 1. Model Introduction
[DeepSeek-OCR](https://github.com/deepseek-ai/DeepSeek-OCR) is DeepSeek's advanced OCR (Optical Character Recognition) model designed for high-accuracy text extraction from images. The model is optimized for various document processing and image-to-text conversion tasks.
**Key Features:**
- **Advanced OCR**: High-accuracy text recognition from images and documents
- **Multi-Modality**: Supports various image formats and document types
**Available Models:**
- **Base Model**: [deepseek-ai/DeepSeek-OCR](https://huggingface.co/deepseek-ai/DeepSeek-OCR) - Recommended for OCR tasks
**License:**
To use DeepSeek-OCR, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-OCR/blob/main/LICENSE) for details.
For more details, please refer to the [official DeepSeek-OCR repository](https://github.com/deepseek-ai/DeepSeek-OCR).
## 2. SGLang Installation
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy.
import { DeepSeekOCRDeployment } from "/src/snippets/autoregressive/deepseek-ocr-deployment.jsx";
<DeepSeekOCRDeployment />
### 3.2 Configuration Tips
For more detailed configuration tips, please refer to [DeepSeek V3/V3.1/R1 Usage](../../../docs/basic_usage/deepseek_v3).
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: AMD MI300X GPU (1x)
- Model: DeepSeek-OCR
- Tensor Parallelism: 1
- sglang version: 0.5.7
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses.
#### 5.1.1 Latency-Sensitive Benchmark
- Model Deployment Command:
```shell Command
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-OCR \
--tp 1 \
--dtype float16 \
--host 0.0.0.0 \
--port 8000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 8000 \
--model deepseek-ai/DeepSeek-OCR \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 10 \
--max-concurrency 1
```
- **Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 4.45
Total input tokens: 1972
Total input text tokens: 1972
Total input vision tokens: 0
Total generated tokens: 2784
Total generated tokens (retokenized): 2770
Request throughput (req/s): 2.25
Input token throughput (tok/s): 442.89
Output token throughput (tok/s): 625.26
Peak output token throughput (tok/s): 635.00
Peak concurrent requests: 4
Total token throughput (tok/s): 1068.16
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 443.32
Median E2E Latency (ms): 493.29
---------------Time to First Token----------------
Mean TTFT (ms): 21.59
Median TTFT (ms): 20.89
P99 TTFT (ms): 24.81
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 1.47
Median TPOT (ms): 1.52
P99 TPOT (ms): 1.53
---------------Inter-Token Latency----------------
Mean ITL (ms): 1.52
Median ITL (ms): 1.51
P95 ITL (ms): 1.76
P99 ITL (ms): 1.93
Max ITL (ms): 8.28
==================================================
```
#### 5.1.2 Throughput-Sensitive Benchmark
- Model Deployment Command:
```shell Command
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-OCR \
--tp 1 \
--ep 1 \
--dp 1 \
--enable-dp-attention \
--dtype float16 \
--host 0.0.0.0 \
--port 8000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 8000 \
--model deepseek-ai/DeepSeek-OCR \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 1000 \
--max-concurrency 100
```
- **Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 16.24
Total input tokens: 301698
Total input text tokens: 301698
Total input vision tokens: 0
Total generated tokens: 188375
Total generated tokens (retokenized): 186927
Request throughput (req/s): 61.59
Input token throughput (tok/s): 18582.90
Output token throughput (tok/s): 11602.84
Peak output token throughput (tok/s): 15479.00
Peak concurrent requests: 179
Total token throughput (tok/s): 30185.75
Concurrency: 85.53
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 1388.60
Median E2E Latency (ms): 901.43
---------------Time to First Token----------------
Mean TTFT (ms): 73.36
Median TTFT (ms): 50.21
P99 TTFT (ms): 349.53
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 7.42
Median TPOT (ms): 7.31
P99 TPOT (ms): 27.99
---------------Inter-Token Latency----------------
Mean ITL (ms): 7.04
Median ITL (ms): 4.62
P95 ITL (ms): 21.11
P99 ITL (ms): 36.92
Max ITL (ms): 172.15
==================================================
```
docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-R1.mdx
+910
View File
@@ -0,0 +1,910 @@
---
title: DeepSeek-R1
metatags:
description: "Deploy DeepSeek-R1 reasoning model with SGLang - advanced step-by-step reasoning with FP8/FP4 quantization for NVIDIA and AMD GPUs."
---
import { DeepSeekR1BasicDeployment } from '/src/snippets/autoregressive/deepseek-r1-basic-deployment.jsx';
import { DeepSeekR1AdvancedDeployment } from '/src/snippets/autoregressive/deepseek-r1-advanced-deployment.jsx';
## 1. Model Introduction
[DeepSeek-R1](https://github.com/deepseek-ai/DeepSeek-R1) is DeepSeek's advanced reasoning model that combines powerful language understanding with step-by-step reasoning capabilities. The model is available in multiple quantization formats optimized for different hardware platforms.
**Key Features:**
- **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving
- **Multiple Quantizations**: FP8 and FP4 variants for different performance/memory trade-offs
- **Hardware Optimization**: Specifically tuned for NVIDIA B200 (Blackwell) and H200 (Hopper) GPUs, and AMD MI300X, MI325X and MI355X GPUs
- **High Performance**: Optimized for both throughput and latency scenarios
**Available Models:**
- **FP8 (8-bit quantized)**: [deepseek-ai/DeepSeek-R1-0528](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528) - Recommended for H200 and MI300X
- **FP4 (4-bit quantized)**: [nvidia/DeepSeek-R1-0528-FP4-v2](https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2) - Recommended for B200 and MI355X
**License:**
To use DeepSeek-R1, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528/blob/main/LICENSE) for details.
For more details, please refer to the [official DeepSeek-R1 repository](https://github.com/deepseek-ai/DeepSeek-R1).
## 2. SGLang Installation
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate a basic deployment command for your hardware platform, quantization method, and deployment strategy.
<DeepSeekR1BasicDeployment />
### 3.2 Optimal Configurations
Pareto-optimal configurations for B200, H200, MI300X, MI325X, and MI355X hardware.
<DeepSeekR1AdvancedDeployment />
### 3.3 Configuration Tips
For more detailed configuration tips and advanced tuning, please refer to [DeepSeek V3/V3.1/R1 Usage](../../../docs/basic_usage/deepseek_v3).
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
DeepSeek-R1 supports advanced reasoning capabilities with built-in thinking process. Enable the reasoning parser during deployment to separate the thinking and content sections:
```shell Command
python -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-R1-0528 \
--reasoning-parser deepseek-r1 \
--tp 8
```
**Streaming with Thinking Process:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-R1-0528",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
To solve this problem, I need to calculate 15% of 240.
Step 1: Convert 15% to decimal: 15% = 0.15
Step 2: Multiply 240 by 0.15
Step 3: 240 × 0.15 = 36
=============== Content =================
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
```
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
#### 4.2.2 Tool Calling
DeepSeek-R1 supports tool calling capabilities. Enable the tool call parser:
```shell Command
python -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-R1-0528 \
--reasoning-parser deepseek-r1 \
--tool-call-parser deepseekv3 \
--chat-template examples/chat_template/tool_chat_template_deepseekr1.jinja \
--tp 8
```
**Python Example (with Thinking Process):**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-R1-0528",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
if tool_call.function:
print(f"🔧 Tool Call: {tool_call.function.name}")
print(f" Arguments: {tool_call.function.arguments}")
# Print content
if delta.content:
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
I should call the function with location="Beijing".
=============== Content =================
🔧 Tool Call: get_weather
Arguments:
🔧 Tool Call: None
Arguments: {"location": "Beijing"}
```
**Note:**
- The reasoning parser shows how the model decides to use a tool
- Tool calls are clearly marked with the function name and arguments
- You can then execute the function and send the result back to continue the conversation
**Handling Tool Call Results:**
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-R1-0528",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```
## 5. Benchmark
This section uses **industry-standard configurations** for comparable benchmark results.
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: B200 GPU (8x)
- Model: DeepSeek-R1-0528
- Tensor Parallelism: 8
- SGLang Version: 0.5.6.post1
**Benchmark Methodology:**
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
#### 5.1.1 Standard Test Scenarios
Three core scenarios reflect real-world usage patterns:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
</tr>
</tbody>
</table>
#### 5.1.2 Concurrency Levels
Test each scenario at different concurrency levels to capture the throughput vs. latency trade-off:
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
#### 5.1.3 Number of Prompts
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
---
#### 5.1.4 Benchmark Commands
**Scenario 1: Chat (1K/1K) - Most Important**
- **Model Deployment**
```bash Command
python -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-R1-0528 \
--tp 8
```
- Low Concurrency (Latency-Optimized)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-R1-0528 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 40.00
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 4210
Total generated tokens (retokenized): 4205
Request throughput (req/s): 0.25
Input token throughput (tok/s): 152.52
Output token throughput (tok/s): 105.24
Peak output token throughput (tok/s): 110.00
Peak concurrent requests: 2
Total token throughput (tok/s): 257.76
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 3998.40
Median E2E Latency (ms): 3207.53
---------------Time to First Token----------------
Mean TTFT (ms): 153.00
Median TTFT (ms): 140.76
P99 TTFT (ms): 214.66
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 9.16
Median TPOT (ms): 9.15
P99 TPOT (ms): 9.21
---------------Inter-Token Latency----------------
Mean ITL (ms): 9.16
Median ITL (ms): 9.15
P95 ITL (ms): 9.47
P99 ITL (ms): 9.63
Max ITL (ms): 15.45
==================================================
```
- Medium Concurrency (Balanced)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-R1-0528 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 51.21
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 40725
Total generated tokens (retokenized): 40458
Request throughput (req/s): 1.56
Input token throughput (tok/s): 774.66
Output token throughput (tok/s): 795.30
Peak output token throughput (tok/s): 1088.00
Peak concurrent requests: 21
Total token throughput (tok/s): 1569.96
Concurrency: 13.93
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 8918.33
Median E2E Latency (ms): 9466.16
---------------Time to First Token----------------
Mean TTFT (ms): 273.51
Median TTFT (ms): 131.71
P99 TTFT (ms): 839.57
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 17.56
Median TPOT (ms): 17.46
P99 TPOT (ms): 28.68
---------------Inter-Token Latency----------------
Mean ITL (ms): 17.02
Median ITL (ms): 14.70
P95 ITL (ms): 16.41
P99 ITL (ms): 112.38
Max ITL (ms): 461.90
==================================================
```
- High Concurrency (Throughput-Optimized)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-R1-0528 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 110.46
Total input tokens: 249831
Total input text tokens: 249831
Total input vision tokens: 0
Total generated tokens: 252162
Total generated tokens (retokenized): 251441
Request throughput (req/s): 4.53
Input token throughput (tok/s): 2261.80
Output token throughput (tok/s): 2282.90
Peak output token throughput (tok/s): 3900.00
Peak concurrent requests: 109
Total token throughput (tok/s): 4544.71
Concurrency: 92.26
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 20380.71
Median E2E Latency (ms): 19391.65
---------------Time to First Token----------------
Mean TTFT (ms): 563.14
Median TTFT (ms): 147.62
P99 TTFT (ms): 2632.11
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 40.11
Median TPOT (ms): 41.98
P99 TPOT (ms): 50.10
---------------Inter-Token Latency----------------
Mean ITL (ms): 39.37
Median ITL (ms): 26.36
P95 ITL (ms): 98.16
P99 ITL (ms): 150.08
Max ITL (ms): 2052.85
==================================================
```
**Scenario 2: Reasoning (1K/8K)**
- Low Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-R1-0528 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 411.34
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 44452
Total generated tokens (retokenized): 44390
Request throughput (req/s): 0.02
Input token throughput (tok/s): 14.83
Output token throughput (tok/s): 108.07
Peak output token throughput (tok/s): 110.00
Peak concurrent requests: 2
Total token throughput (tok/s): 122.90
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 41132.04
Median E2E Latency (ms): 44288.71
---------------Time to First Token----------------
Mean TTFT (ms): 125.76
Median TTFT (ms): 126.19
P99 TTFT (ms): 137.69
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 9.21
Median TPOT (ms): 9.20
P99 TPOT (ms): 9.27
---------------Inter-Token Latency----------------
Mean ITL (ms): 9.23
Median ITL (ms): 9.22
P95 ITL (ms): 9.64
P99 ITL (ms): 9.86
Max ITL (ms): 15.18
==================================================
```
- Medium Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-R1-0528 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 348.93
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 318226
Total generated tokens (retokenized): 317630
Request throughput (req/s): 0.23
Input token throughput (tok/s): 113.69
Output token throughput (tok/s): 912.02
Peak output token throughput (tok/s): 1088.00
Peak concurrent requests: 19
Total token throughput (tok/s): 1025.70
Concurrency: 14.07
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 61360.70
Median E2E Latency (ms): 62071.20
---------------Time to First Token----------------
Mean TTFT (ms): 176.02
Median TTFT (ms): 153.75
P99 TTFT (ms): 268.44
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 15.42
Median TPOT (ms): 15.59
P99 TPOT (ms): 16.07
---------------Inter-Token Latency----------------
Mean ITL (ms): 15.39
Median ITL (ms): 15.17
P95 ITL (ms): 16.62
P99 ITL (ms): 18.13
Max ITL (ms): 226.59
==================================================
```
- High Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-R1-0528 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 320 \
--max-concurrency 64 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 589.31
Total input tokens: 158939
Total input text tokens: 158939
Total input vision tokens: 0
Total generated tokens: 1300705
Total generated tokens (retokenized): 1297658
Request throughput (req/s): 0.54
Input token throughput (tok/s): 269.70
Output token throughput (tok/s): 2207.16
Peak output token throughput (tok/s): 2944.00
Peak concurrent requests: 68
Total token throughput (tok/s): 2476.86
Concurrency: 57.03
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 105032.36
Median E2E Latency (ms): 108229.09
---------------Time to First Token----------------
Mean TTFT (ms): 223.91
Median TTFT (ms): 158.15
P99 TTFT (ms): 474.86
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 25.94
Median TPOT (ms): 26.72
P99 TPOT (ms): 27.99
---------------Inter-Token Latency----------------
Mean ITL (ms): 25.79
Median ITL (ms): 25.37
P95 ITL (ms): 26.70
P99 ITL (ms): 105.49
Max ITL (ms): 237.91
==================================================
```
**Scenario 3: Summarization (8K/1K)**
- Low Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-R1-0528 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 40.65
Total input tokens: 41941
Total input text tokens: 41941
Total input vision tokens: 0
Total generated tokens: 4210
Total generated tokens (retokenized): 4195
Request throughput (req/s): 0.25
Input token throughput (tok/s): 1031.65
Output token throughput (tok/s): 103.56
Peak output token throughput (tok/s): 110.00
Peak concurrent requests: 2
Total token throughput (tok/s): 1135.20
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 4063.62
Median E2E Latency (ms): 3296.13
---------------Time to First Token----------------
Mean TTFT (ms): 165.91
Median TTFT (ms): 154.96
P99 TTFT (ms): 240.92
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 9.26
Median TPOT (ms): 9.27
P99 TPOT (ms): 9.42
---------------Inter-Token Latency----------------
Mean ITL (ms): 9.28
Median ITL (ms): 9.28
P95 ITL (ms): 9.66
P99 ITL (ms): 9.83
Max ITL (ms): 14.06
==================================================
```
- Medium Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-R1-0528 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 56.71
Total input tokens: 300020
Total input text tokens: 300020
Total input vision tokens: 0
Total generated tokens: 41589
Total generated tokens (retokenized): 41490
Request throughput (req/s): 1.41
Input token throughput (tok/s): 5290.75
Output token throughput (tok/s): 733.41
Peak output token throughput (tok/s): 1024.00
Peak concurrent requests: 20
Total token throughput (tok/s): 6024.16
Concurrency: 14.25
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 10098.99
Median E2E Latency (ms): 10623.46
---------------Time to First Token----------------
Mean TTFT (ms): 486.80
Median TTFT (ms): 189.59
P99 TTFT (ms): 2138.73
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 19.06
Median TPOT (ms): 19.23
P99 TPOT (ms): 30.69
---------------Inter-Token Latency----------------
Mean ITL (ms): 18.53
Median ITL (ms): 15.63
P95 ITL (ms): 16.64
P99 ITL (ms): 109.71
Max ITL (ms): 1471.36
==================================================
```
- High Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-R1-0528 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 320 \
--max-concurrency 64 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 115.55
Total input tokens: 1273893
Total input text tokens: 1273893
Total input vision tokens: 0
Total generated tokens: 169680
Total generated tokens (retokenized): 169275
Request throughput (req/s): 2.77
Input token throughput (tok/s): 11024.93
Output token throughput (tok/s): 1468.50
Peak output token throughput (tok/s): 2254.00
Peak concurrent requests: 70
Total token throughput (tok/s): 12493.43
Concurrency: 59.45
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 21465.98
Median E2E Latency (ms): 20686.26
---------------Time to First Token----------------
Mean TTFT (ms): 913.93
Median TTFT (ms): 224.92
P99 TTFT (ms): 6257.83
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 39.93
Median TPOT (ms): 40.99
P99 TPOT (ms): 60.91
---------------Inter-Token Latency----------------
Mean ITL (ms): 38.83
Median ITL (ms): 26.29
P95 ITL (ms): 113.81
P99 ITL (ms): 176.94
Max ITL (ms): 5521.53
==================================================
```
#### 5.1.5 Understanding the Results
**Key Metrics:**
- **Request Throughput (req/s)**: Number of requests processed per second
- **Output Token Throughput (tok/s)**: Total tokens generated per second
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
**Why These Configurations Matter:**
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
- **Variable Concurrency**: Captures the Pareto frontier - the optimal trade-off between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
**Interpreting Results:**
- Compare your results against baseline numbers for your hardware
- Higher throughput at same latency = better performance
- Lower TTFT = more responsive user experience
- Lower TPOT = faster generation speed
### 5.2 Accuracy Benchmark
Document model accuracy on standard benchmarks:
#### 5.2.1 GSM8K Benchmark
- Benchmark Command
```bash Command
python3 benchmark/gsm8k/bench_sglang.py \
--num-shots 8 \
--num-questions 1316 \
--parallel 1316
```
**Test Results:**
```text Output
Accuracy: 0.959
Invalid: 0.000
Latency: 29.185 s
Output throughput: 4854.672 token/s
```
docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V3.mdx
+520
View File
@@ -0,0 +1,520 @@
---
title: "DeepSeek-V3"
metatags:
description: "Deploy DeepSeek-V3 MoE model with SGLang - efficient architecture with strong reasoning, coding, and tool-augmented capabilities."
---
## 1. Model Introduction
[DeepSeek V3](https://huggingface.co/deepseek-ai/DeepSeek-V3) is a large-scale Mixture-of-Experts (MoE) language model developed by DeepSeek, designed to deliver strong general-purpose reasoning, coding, and tool-augmented capabilities with high training and inference efficiency. As the latest generation in the DeepSeek model family, DeepSeek V3 introduces systematic architectural and training innovations that significantly improve performance across reasoning, mathematics, coding, and long-context understanding, while maintaining a competitive compute cost.
Key highlights include:
- **Efficient MoE architecture**: DeepSeek V3 adopts a fine-grained Mixture-of-Experts design with a large number of experts and sparse activation, enabling high model capacity while keeping inference and training costs manageable.
- **Advanced reasoning and coding**: The model demonstrates strong performance on mathematical reasoning, logical inference, and real-world coding benchmarks, benefiting from improved data curation and training strategies.
- **Long-context capability**: DeepSeek V3 supports extended context lengths, allowing it to handle long documents, complex multi-step reasoning, and agent-style workflows more effectively.
- **Tool use and function calling**: The model is trained to support structured outputs and tool invocation, enabling seamless integration with external tools and agent frameworks during inference.
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
import { DeepSeekV3Deployment } from "/src/snippets/autoregressive/deepseek-v3-deployment.jsx";
<DeepSeekV3Deployment />
### 3.2 Configuration Tips
For more detailed configuration tips, please refer to [DeepSeek-V3 Usage](../../../docs/basic_usage/deepseek_v3).
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [Basic API Usage](../../../docs/get-started/quickstart)
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
DeepSeek-V3 supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
```shell Command
python -m sglang.launch_server \
--model deepseek-ai/DeepSeek-V3 \
--reasoning-parser deepseek-v3 \
--tp 8
```
**Streaming with Thinking Process:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
extra_body = {"chat_template_kwargs": {"thinking": True}},
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
To determine 15% of a number, follow these steps:
**Step 1: Understand the Problem**
You need to find 15% of a given number. Let's assume the number is 240 for this example.
**Step 2: Convert the Percentage to a Decimal**
To work with percentages in calculations, convert the percentage to its decimal form. To do this, divide the percentage by 100.
\[ 15\% = \frac{15}{100} = 0.15 \]
**Step 3: Multiply the Decimal by the Number**
Now, multiply the decimal form of the percentage by the number you want to find the percentage of.
\[ 0.15 \times 240 \]
**Step 4: Perform the Multiplication**
Calculate the product:
\[ 0.15 \times 240 = 36 \]
**Step 5: Conclusion**
Therefore, 15% of 240 is:
\boxed{36}
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
```
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
#### 4.2.2 Tool Calling
DeepSeek-V3 supports tool calling capabilities. Enable the tool call parser:
**Deployment Command:**
```shell Command
python -m sglang.launch_server \
--model deepseek-ai/DeepSeek-V3 \
--tool-call-parser deepseekv3 \
--reasoning-parser deepseek-v3 \
--chat-template ./examples/chat_template/tool_chat_template_deepseekv3.jinja \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
**Python Example (with Thinking Process):**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
extra_body = {"chat_template_kwargs": {"thinking": True}},
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Accumulate tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================\n", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {
'name': None,
'arguments': ''
}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]['name'] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
# Print content
if delta.content:
print(delta.content, end="", flush=True)
# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"🔧 Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
print()
```
**Output Example:**
```text Output
=============== Thinking =================
<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather
```json
{"location": "Beijing", "unit": "celsius"}
```<｜tool▁call▁end｜><｜tool▁calls▁end｜>
```
**Note:**
- The reasoning parser shows how the model decides to use a tool
- Tool calls are clearly marked with the function name and arguments
- You can then execute the function and send the result back to continue the conversation
**Handling Tool Call Results:**
Please attach the code blocks below to the previous Python script.
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: AMD MI300X GPU (8x)
- Model: DeepSeek-V3
- Tensor Parallelism: 8
- sglang version: 0.5.7
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses.
#### 5.1.1 Latency-Sensitive Benchmark
- Model Deployment Command:
```shell Command
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3 \
--tp 8 \
--dp 8 \
--enable-dp-attention \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--host 0.0.0.0 \
--port 8000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 8000 \
--model deepseek-ai/DeepSeek-V3 \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 10 \
--max-concurrency 1
```
- **Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 81.27
Total input tokens: 1972
Total input text tokens: 1972
Total input vision tokens: 0
Total generated tokens: 2784
Total generated tokens (retokenized): 2774
Request throughput (req/s): 0.12
Input token throughput (tok/s): 24.27
Output token throughput (tok/s): 34.26
Peak output token throughput (tok/s): 65.00
Peak concurrent requests: 2
Total token throughput (tok/s): 58.52
Concurrency: 1.00
Accept length: 2.61
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 8123.17
Median E2E Latency (ms): 7982.65
---------------Time to First Token----------------
Mean TTFT (ms): 1080.76
Median TTFT (ms): 1248.82
P99 TTFT (ms): 1896.37
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 25.04
Median TPOT (ms): 24.76
P99 TPOT (ms): 32.09
---------------Inter-Token Latency----------------
Mean ITL (ms): 25.41
Median ITL (ms): 20.14
P95 ITL (ms): 60.28
P99 ITL (ms): 60.99
Max ITL (ms): 61.49
==================================================
```
#### 5.1.2 Throughput-Sensitive Benchmark
- Model Deployment Command:
```shell Command
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3 \
--tp 8 \
--ep 8 \
--dp 8 \
--enable-dp-attention \
--host 0.0.0.0 \
--port 8000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 8000 \
--model deepseek-ai/DeepSeek-V3 \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 1000 \
--max-concurrency 100
```
- **Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 406.16
Total input tokens: 301701
Total input text tokens: 301701
Total input vision tokens: 0
Total generated tokens: 188375
Total generated tokens (retokenized): 187542
Request throughput (req/s): 2.46
Input token throughput (tok/s): 742.81
Output token throughput (tok/s): 463.80
Peak output token throughput (tok/s): 1299.00
Peak concurrent requests: 109
Total token throughput (tok/s): 1206.61
Concurrency: 87.53
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 35552.98
Median E2E Latency (ms): 21466.07
---------------Time to First Token----------------
Mean TTFT (ms): 1521.51
Median TTFT (ms): 476.80
P99 TTFT (ms): 8329.50
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 214.73
Median TPOT (ms): 152.00
P99 TPOT (ms): 1155.85
---------------Inter-Token Latency----------------
Mean ITL (ms): 182.10
Median ITL (ms): 79.18
P95 ITL (ms): 398.60
P99 ITL (ms): 1488.96
Max ITL (ms): 43465.60
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 GSM8K Benchmark
- **Benchmark Command:**
```shell Command
python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000
```
- **Test Results**:
- DeepSeek-V3
```text Output
Accuracy: 0.960
Invalid: 0.000
Latency: 32.450 s
Output throughput: 614.211 token/s
```
#### 5.2.2 MMLU Benchmark
- **Benchmark Command:**
```shell Command
cd sglang
bash benchmark/mmlu/download_data.sh
python3 benchmark/mmlu/bench_sglang.py --nsub 10 --port 8000
```
- **Test Results**:
- DeepSeek-V3
```text Output
subject: abstract_algebra, #q:100, acc: 0.800
subject: anatomy, #q:135, acc: 0.874
subject: astronomy, #q:152, acc: 0.928
subject: business_ethics, #q:100, acc: 0.880
subject: clinical_knowledge, #q:265, acc: 0.928
subject: college_biology, #q:144, acc: 0.965
subject: college_chemistry, #q:100, acc: 0.670
subject: college_computer_science, #q:100, acc: 0.840
subject: college_mathematics, #q:100, acc: 0.800
subject: college_medicine, #q:173, acc: 0.861
Total latency: 58.339
Average accuracy: 0.871
```
docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V3_1.mdx
+941
View File
@@ -0,0 +1,941 @@
---
title: DeepSeek-V3.1
metatags:
description: "Deploy DeepSeek-V3.1 MoE model with SGLang - hybrid reasoning, improved tool calling, and agentic behavior for complex multi-step tasks."
---
## 1. Model Introduction
[DeepSeek V3.1](https://huggingface.co/deepseek-ai/DeepSeek-V3.1) is an advanced Mixture-of-Experts (MoE) large language model developed by DeepSeek, representing a major capability and usability upgrade over DeepSeek V3. As a refined iteration in the DeepSeek V3 family, DeepSeek V3.1 introduces a hybrid reasoning paradigm that supports both fast non-thinking responses and explicit multi-step reasoning, alongside significantly improved tool calling and agentic behavior. The model demonstrates strong performance across reasoning, mathematics, coding, long-context understanding, and real-world agent workflows, benefiting from continued training, alignment optimization, and inference-time refinements. DeepSeek V3.1 is designed to serve as a robust general-purpose foundation model, well suited for conversational AI, structured tool invocation, search-augmented generation, and complex multi-step tasks, while maintaining high efficiency through its sparse MoE architecture.
**[DeepSeek-V3.1-Terminus](https://huggingface.co/deepseek-ai/DeepSeek-V3.1-Terminus)** is an experimental version designed for general conversations and long-context processing. It features hybrid thinking capabilities, allowing you to toggle between "Think" mode for deliberate reasoning and "Non-Think" mode for faster responses. Recommended for general conversations, long-context processing, and experimental use cases.
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
import { DeepSeekV31Deployment } from "/src/snippets/autoregressive/deepseek-v31-deployment.jsx";
<DeepSeekV31Deployment />
### 3.2 Configuration Tips
For more detailed configuration tips, please refer to [DeepSeek V3/V3.1/R1 Usage](../../../docs/basic_usage/deepseek_v3).
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [Basic API Usage](../../../docs/get-started/quickstart)
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
DeepSeek-V3.1 supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
```shell Command
python -m sglang.launch_server \
--model deepseek-ai/DeepSeek-V3.1-Terminus \
--reasoning-parser deepseek-v3 \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
**Streaming with Thinking Process:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3.1-Terminus",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
extra_body = {"chat_template_kwargs": {"thinking": True}},
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
First, the problem is asking for 15% of 240. Percent means per hundred, so 15% is the same as 15 out of 100, or 15/100.
To find a percentage of a number, I can multiply the number by the percentage expressed as a decimal. So, I need to convert 15% to a decimal. To do that, I divide 15 by 100, which gives me 0.15.
Now, I multiply 0.15 by 240. So, the calculation is 0.15 × 240.
I can compute this step by step. First, I know that 15% of 100 is 15, but since 240 is larger, I need to adjust. Alternatively, I can think of 10% of 240, which is easy because 10% is just 240 divided by 10, which is 24. Then, 5% is half of 10%, so half of 24 is 12. Therefore, 15% is 10% plus 5%, so 24 plus 12, which equals 36.
I should also do the multiplication to confirm. 0.15 × 240. I can break it down: 0.15 × 200 = 30, and 0.15 × 40 = 6, so 30 + 6 = 36. Same answer.
So, 15% of 240 is 36.
The problem says "step by step," so I should present it clearly.
=============== Content =================
To find 15% of 240, follow these steps:
1. Understand that "percent" means "per hundred," so 15% is equivalent to \( \frac{15}{100} \).
2. Convert 15% to a decimal by dividing by 100: \( 15\% = \frac{15}{100} = 0.15 \).
3. Multiply the decimal by 240: \( 0.15 \times 240 \).
4. Perform the multiplication:
- \( 0.15 \times 200 = 30 \)
- \( 0.15 \times 40 = 6 \)
- Add the results: \( 30 + 6 = 36 \).
Alternatively, you can find 15% by breaking it into parts:
- 10% of 240 is \( \frac{10}{100} \times 240 = 0.10 \times 240 = 24 \).
- 5% of 240 is half of 10%, so \( \frac{24}{2} = 12 \).
- Add 10% and 5%: \( 24 + 12 = 36 \).
Thus, 15% of 240 is 36.
```
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
#### 4.2.2 Tool Calling
DeepSeek-V3.1 and DeepSeek-V3.1-Terminus support tool calling capabilities. Enable the tool call parser:
**Note:** DeepSeek-V3.1-Speciale does **NOT** support tool calling. It is designed exclusively for deep reasoning tasks.
**Deployment Command:**
```shell Command
python -m sglang.launch_server \
--model deepseek-ai/DeepSeek-V3.1-Terminus \
--tool-call-parser deepseekv31 \
--reasoning-parser deepseek-v3 \
--chat-template ./examples/chat_template/tool_chat_template_deepseekv31.jinja \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
For DeepSeek-V3.1, use `--tool-call-parser deepseekv31` as well.
**Python Example (with Thinking Process):**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3.1-Terminus",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
extra_body = {"chat_template_kwargs": {"thinking": True}},
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Accumulate tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================\n", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {
'name': None,
'arguments': ''
}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]['name'] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
# Print content
if delta.content:
print(delta.content, end="", flush=True)
# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"🔧 Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
print()
```
**Output Example:**
```text Output
=============== Thinking =================
Hmm, the user is asking for the weather in Beijing. This is a straightforward request that matches exactly what the weather tool can provide.
I need to call the get_weather function with Beijing as the location parameter. The user didn't specify a temperature unit, so I'll default to Celsius since that's commonly used in most parts of the world.
The tool call format needs to be precise - just the city name and unit selection. Once I get the weather data back, I'll present it clearly to the user.I'll check the weather in Beijing for you.
=============== Content =================
🔧 Tool Call: get_weather
Arguments: {"location": "Beijing", "unit": "celsius"}
```
**Note:**
- The reasoning parser shows how the model decides to use a tool
- Tool calls are clearly marked with the function name and arguments
- You can then execute the function and send the result back to continue the conversation
**Handling Tool Call Results:**
Please attach the code blocks below to the previous Python script.
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3.1-Terminus",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "Currently, it is **22°C and sunny** in Beijing."
```
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: AMD MI300X GPU (8x)
- Model: DeepSeek-V3.1-Terminus
- Tensor Parallelism: 8
- sglang version: 0.5.7
**Benchmark Methodology:**
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
#### 5.1.1 Standard Test Scenarios
Three core scenarios reflect real-world usage patterns:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
</tr>
</tbody>
</table>
#### 5.1.2 Concurrency Levels
Test each scenario at different concurrency levels to capture the throughput vs. latency trade-off:
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
#### 5.1.3 Number of Prompts
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
---
#### 5.1.4 Benchmark Commands
**Scenario 1: Chat (1K/1K) - Most Important**
- **Model Deployment**
```bash Command
python -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3.1 \
--tp 8
```
- Low Concurrency (Latency-Optimized)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 106.24
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4201
Request throughput (req/s): 0.09
Input token throughput (tok/s): 57.43
Output token throughput (tok/s): 39.72
Peak output token throughput (tok/s): 43.00
Peak concurrent requests: 2
Total token throughput (tok/s): 97.15
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 10620.29
Median E2E Latency (ms): 8868.09
---------------Time to First Token----------------
Mean TTFT (ms): 557.85
Median TTFT (ms): 213.58
P99 TTFT (ms): 1625.28
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 23.84
Median TPOT (ms): 23.90
P99 TPOT (ms): 24.03
---------------Inter-Token Latency----------------
Mean ITL (ms): 23.90
Median ITL (ms): 23.92
P95 ITL (ms): 24.15
P99 ITL (ms): 24.25
Max ITL (ms): 25.44
==================================================
```
- Medium Concurrency (Balanced)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 107.71
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 40805
Total generated tokens (retokenized): 40625
Request throughput (req/s): 0.74
Input token throughput (tok/s): 368.28
Output token throughput (tok/s): 378.84
Peak output token throughput (tok/s): 508.00
Peak concurrent requests: 19
Total token throughput (tok/s): 747.12
Concurrency: 13.72
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 18473.65
Median E2E Latency (ms): 19558.42
---------------Time to First Token----------------
Mean TTFT (ms): 607.91
Median TTFT (ms): 191.32
P99 TTFT (ms): 2135.13
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 35.50
Median TPOT (ms): 35.99
P99 TPOT (ms): 43.62
---------------Inter-Token Latency----------------
Mean ITL (ms): 35.10
Median ITL (ms): 32.18
P95 ITL (ms): 33.03
P99 ITL (ms): 159.99
Max ITL (ms): 453.99
==================================================
```
- High Concurrency (Throughput-Optimized)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 207.65
Total input tokens: 249831
Total input text tokens: 249831
Total input vision tokens: 0
Total generated tokens: 252662
Total generated tokens (retokenized): 251238
Request throughput (req/s): 2.41
Input token throughput (tok/s): 1203.15
Output token throughput (tok/s): 1216.79
Peak output token throughput (tok/s): 2100.00
Peak concurrent requests: 106
Total token throughput (tok/s): 2419.94
Concurrency: 91.02
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 37800.20
Median E2E Latency (ms): 35921.56
---------------Time to First Token----------------
Mean TTFT (ms): 835.15
Median TTFT (ms): 236.88
P99 TTFT (ms): 2868.52
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 73.33
Median TPOT (ms): 76.35
P99 TPOT (ms): 97.63
---------------Inter-Token Latency----------------
Mean ITL (ms): 73.30
Median ITL (ms): 50.82
P95 ITL (ms): 180.67
P99 ITL (ms): 186.83
Max ITL (ms): 1661.39
==================================================
```
**Scenario 2: Reasoning (1K/8K)**
- Low Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 1097.29
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 44462
Total generated tokens (retokenized): 44313
Request throughput (req/s): 0.01
Input token throughput (tok/s): 5.56
Output token throughput (tok/s): 40.52
Peak output token throughput (tok/s): 43.00
Peak concurrent requests: 2
Total token throughput (tok/s): 46.08
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 109725.52
Median E2E Latency (ms): 117748.67
---------------Time to First Token----------------
Mean TTFT (ms): 156.67
Median TTFT (ms): 156.19
P99 TTFT (ms): 159.87
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 24.41
Median TPOT (ms): 24.51
P99 TPOT (ms): 24.96
---------------Inter-Token Latency----------------
Mean ITL (ms): 24.65
Median ITL (ms): 24.58
P95 ITL (ms): 25.68
P99 ITL (ms): 25.93
Max ITL (ms): 29.80
==================================================
```
- Medium Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 775.02
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 318306
Total generated tokens (retokenized): 317426
Request throughput (req/s): 0.10
Input token throughput (tok/s): 51.18
Output token throughput (tok/s): 410.70
Peak output token throughput (tok/s): 512.00
Peak concurrent requests: 18
Total token throughput (tok/s): 461.89
Concurrency: 13.86
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 134236.65
Median E2E Latency (ms): 135181.28
---------------Time to First Token----------------
Mean TTFT (ms): 214.35
Median TTFT (ms): 194.12
P99 TTFT (ms): 300.27
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 33.72
Median TPOT (ms): 34.00
P99 TPOT (ms): 34.75
---------------Inter-Token Latency----------------
Mean ITL (ms): 33.69
Median ITL (ms): 33.71
P95 ITL (ms): 34.50
P99 ITL (ms): 34.92
Max ITL (ms): 164.76
==================================================
```
- High Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 320 \
--max-concurrency 64 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 1231.97
Total input tokens: 158939
Total input text tokens: 158939
Total input vision tokens: 0
Total generated tokens: 1301025
Total generated tokens (retokenized): 1296845
Request throughput (req/s): 0.26
Input token throughput (tok/s): 129.01
Output token throughput (tok/s): 1056.05
Peak output token throughput (tok/s): 1472.00
Peak concurrent requests: 67
Total token throughput (tok/s): 1185.07
Concurrency: 56.17
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 216256.25
Median E2E Latency (ms): 224192.84
---------------Time to First Token----------------
Mean TTFT (ms): 317.68
Median TTFT (ms): 235.28
P99 TTFT (ms): 649.39
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 53.30
Median TPOT (ms): 55.10
P99 TPOT (ms): 56.58
---------------Inter-Token Latency----------------
Mean ITL (ms): 53.13
Median ITL (ms): 52.95
P95 ITL (ms): 56.23
P99 ITL (ms): 181.04
Max ITL (ms): 208.61
==================================================
```
**Scenario 3: Summarization (8K/1K)**
- Low Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 114.47
Total input tokens: 41941
Total input text tokens: 41941
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4194
Request throughput (req/s): 0.09
Input token throughput (tok/s): 366.39
Output token throughput (tok/s): 36.87
Peak output token throughput (tok/s): 42.00
Peak concurrent requests: 2
Total token throughput (tok/s): 403.26
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 11442.86
Median E2E Latency (ms): 9508.87
---------------Time to First Token----------------
Mean TTFT (ms): 883.78
Median TTFT (ms): 481.38
P99 TTFT (ms): 2217.45
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 24.93
Median TPOT (ms): 25.05
P99 TPOT (ms): 26.11
---------------Inter-Token Latency----------------
Mean ITL (ms): 25.08
Median ITL (ms): 25.08
P95 ITL (ms): 26.18
P99 ITL (ms): 26.28
Max ITL (ms): 27.41
==================================================
```
- Medium Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 162.33
Total input tokens: 300020
Total input text tokens: 300020
Total input vision tokens: 0
Total generated tokens: 41669
Total generated tokens (retokenized): 41443
Request throughput (req/s): 0.49
Input token throughput (tok/s): 1848.27
Output token throughput (tok/s): 256.70
Peak output token throughput (tok/s): 467.00
Peak concurrent requests: 19
Total token throughput (tok/s): 2104.97
Concurrency: 14.52
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 29456.89
Median E2E Latency (ms): 27628.16
---------------Time to First Token----------------
Mean TTFT (ms): 1784.30
Median TTFT (ms): 1347.21
P99 TTFT (ms): 5384.54
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 53.65
Median TPOT (ms): 52.09
P99 TPOT (ms): 74.39
---------------Inter-Token Latency----------------
Mean ITL (ms): 53.23
Median ITL (ms): 34.52
P95 ITL (ms): 35.81
P99 ITL (ms): 513.25
Max ITL (ms): 2865.73
==================================================
```
- High Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 320 \
--max-concurrency 64 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 282.55
Total input tokens: 1273893
Total input text tokens: 1273893
Total input vision tokens: 0
Total generated tokens: 170000
Total generated tokens (retokenized): 169081
Request throughput (req/s): 1.13
Input token throughput (tok/s): 4508.6
Output token throughput (tok/s): 601.67
Peak output token throughput (tok/s): 1216
Peak concurrent requests: 68
Total token throughput (tok/s): 5110.27
Concurrency: 59.81
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 52810.32
Median E2E Latency (ms): 50981.81
---------------Time to First Token----------------
Mean TTFT (ms): 786.69
Median TTFT (ms): 499.38
P99 TTFT (ms): 2925.98
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 97.93
Median TPOT (ms): 103.45
P99 TPOT (ms): 157.84
---------------Inter-Token Latency----------------
Mean ITL (ms): 98.11
Median ITL (ms): 55.7
P95 ITL (ms): 240.71
P99 ITL (ms): 1114.36
==================================================
```
#### 5.1.5 Understanding the Results
**Key Metrics:**
- **Request Throughput (req/s)**: Number of requests processed per second
- **Output Token Throughput (tok/s)**: Total tokens generated per second
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
**Why These Configurations Matter:**
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
- **Variable Concurrency**: Captures the Pareto frontier - the optimal trade-off between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
**Interpreting Results:**
- Compare your results against baseline numbers for your hardware
- Higher throughput at same latency = better performance
- Lower TTFT = more responsive user experience
- Lower TPOT = faster generation speed
### 5.2 Accuracy Benchmark
Document model accuracy on standard benchmarks:
#### 5.2.1 GSM8K Benchmark
- Benchmark Command
```bash Command
python3 benchmark/gsm8k/bench_sglang.py \
--num-shots 8 \
--num-questions 1316 \
--parallel 1316
```
**Test Results:**
```text Output
Accuracy: 0.959
Invalid: 0.000
Latency: 29.185 s
Output throughput: 4854.672 token/s
```
docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2.mdx
+827
View File
@@ -0,0 +1,827 @@
---
title: DeepSeek-V3.2
metatags:
description: "Deploy DeepSeek-V3.2 with SGLang - featuring DeepSeek Sparse Attention for efficient long-context processing and deep reasoning capabilities."
---
## 1. Model Introduction
The DeepSeek-V3.2 series includes three model variants, each optimized for different use cases:
**[DeepSeek-V3.2-Exp](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp)** is an upgraded version of DeepSeek-V3.1-Terminus, introducing the DeepSeek Sparse Attention (DSA) mechanism through continued training. DSA is a fine-grained sparse attention mechanism powered by a lightning indexer, enabling DeepSeek-V3.2-Exp to achieve significant efficiency improvements in long-context scenarios. Recommended for general conversations, long-context processing, and efficient inference.
**[DeepSeek-V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2)** is the standard version suitable for general tasks and conversational scenarios. For local deployment, we recommend setting the sampling parameters to temperature = 1.0, top_p = 0.95. Recommended for standard conversations and general tasks.
**[DeepSeek-V3.2-Speciale](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Speciale)** is a special variant designed exclusively for deep reasoning tasks. This model is specifically optimized for scenarios requiring complex logical reasoning and deep thinking. However this model does not support tool calls (see below). For local deployment, we recommend setting the sampling parameters to temperature = 1.0, top_p = 0.95. Recommended for deep reasoning tasks, complex logical problems, and mathematical reasoning.
**[DeepSeek-V3.2-NVFP4](https://huggingface.co/nvidia/DeepSeek-V3.2-NVFP4)** is an NVIDIA-optimized NVFP4-quantized variant of DeepSeek-V3.2 for Blackwell devices. It uses ModelOpt FP4 quantization with a choice of MoE runner backends (`flashinfer_trtllm` (recommended), `flashinfer_cutlass`, or `flashinfer_cutedsl`), enabling efficient deployment with lower tensor parallelism (TP=4). It supports the same features as DeepSeek-V3.2 including tool calling, reasoning, and speculative decoding (MTP).
**[DeepSeek-V3.2-MXFP4](https://huggingface.co/amd/DeepSeek-V3.2-mxfp4)** is an OCP-MXFP4 optimized variant for DeepSeek-V3.2 for AMD MI300X/MI355X devices. It uses OCP MXFP4 quantization with a triton mxfp4 backend (the same backend for gptoss-120B), enabling efficient deployment with lower tensor parallelism (TP=8) in a single node. It includes the same features as DeepSeek-V3.2 including tool calling, reasoning, fp8-kv, CP, TP and speculative decoding MTP.
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities. SGLang supports serving DeepSeek V3.2 on NVIDIA H200, B200, and AMD MI300X/MI355X GPUs.
import { DeepSeekV32Deployment } from "/src/snippets/autoregressive/deepseek-v32-deployment.jsx";
<DeepSeekV32Deployment />
### 3.2 Configuration Tips
For more detailed configuration tips, please refer to [DeepSeek-V3.2 Usage](../../../docs/basic_usage/deepseek_v32).
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [Basic API Usage](../../../docs/basic_usage/send_request)
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
DeepSeek-V3.2 supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
```shell Command
sglang serve \
--model deepseek-ai/DeepSeek-V3.2-Exp \
--reasoning-parser deepseek-v3 \
--tp 8 \
--host 0.0.0.0 \
--port 30000
```
**Streaming with Thinking Process:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3.2-Exp",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
extra_body = {"chat_template_kwargs": {"thinking": True}},
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
To solve this problem, I need to calculate 15% of 240.
Step 1: Convert 15% to decimal: 15% = 0.15
Step 2: Multiply 240 by 0.15
Step 3: 240 × 0.15 = 36
=============== Content =================
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
```
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
#### 4.2.2 Tool Calling
DeepSeek-V3.2 and DeepSeek-V3.2-Exp support tool calling capabilities. But they use different parameters. Enable the tool call parser:
**Note:** DeepSeek-V3.2-Speciale does **NOT** support tool calling. It is designed exclusively for deep reasoning tasks.
**Deployment Command:**
For DeepSeek-V3.2-Exp:
```shell Command
sglang serve \
--model deepseek-ai/DeepSeek-V3.2-Exp \
--tool-call-parser deepseekv31 \
--reasoning-parser deepseek-v3 \
--chat-template ./examples/chat_template/tool_chat_template_deepseekv32.jinja \
--tp 8 \
--host 0.0.0.0 \
--port 30000
```
For DeepSeek-V3.2, use `--tool-call-parser deepseekv32` and remove `--chat-template`.
**Python Example (with Thinking Process):**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3.2-Exp",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
extra_body = {"chat_template_kwargs": {"thinking": True}},
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Accumulate tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================\n", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {
'name': None,
'arguments': ''
}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]['name'] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
# Print content
if delta.content:
print(delta.content, end="", flush=True)
# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
print()
```
**Output Example:**
```text Output
=============== Thinking =================
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
I should call the function with location="Beijing".
=============== Content =================
Tool Call: get_weather
Arguments: {"location": "Beijing", "unit": "celsius"}
```
**Note:**
- The reasoning parser shows how the model decides to use a tool
- Tool calls are clearly marked with the function name and arguments
- You can then execute the function and send the result back to continue the conversation
**Handling Tool Call Results:**
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3.2-Exp",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```
#### 4.2.3 Enabling PP, CP and TP with FP8 KV cache
We suggested `DP2` + `MTP` for local deployment of agentic workflow with DeepSeek V3.2 on Hopper platform:
```shell Command
export SGLANG_DEEPEP_LL_COMBINE_SEND_NUM_SMS=32
export SGLANG_SET_CPU_AFFINITY=1
# Test workload ISL/OSL=1k/1k, raw tap : 4948.16 toks/sec, MAX ITL 5970
# dp 2 : 5019.54 toks/sec, MAX ITL 7233
# dp 4 : 4942.82 toks/sec, MAX ITL 35654
# dp 2 + mtp : 6842.51 toks/sec, MAX ITL 3081
sglang_args=$(echo serve \
--model-path $MAPPED_MODEL_PATH \
--nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 16 \
--dp 2 --enable-dp-attention --page-size 64 \
--trust-remote-code --host "0.0.0.0" --port 30000 \
--log-requests \
--context-length 65536 --max-running-requests 128 \
--speculative-algorithm EAGLE \
--speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3 \
--allow-auto-truncate --enable-metrics \
--tool-call-parser deepseekv32 --reasoning-parser deepseek-v3 \
--served-model-name DeepSeek-V3.2-Opt-dp2-mtp
)
sglang_args=($sglang_args)
sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log
```
**CP + PP + EP + DP**
`CP` is currently enabled with `PP=2` on Hopper platform and we can reduce TP=16 to TP=8 from standalone deployment:
```shell Command
# verified on Hopper platform
sglang_args=$(echo serve \
--model-path $MAPPED_MODEL_PATH \
--nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 8 --pp-size 2 --dp 1 --enable-dp-attention \
--moe-a2a-backend deepep --ep-size 16 \
--page-size 128 \
--chunked-prefill-size 16384 \
--attention-backend nsa \
--nsa-prefill-backend flashmla_sparse \
--nsa-decode-backend flashmla_sparse \
--enable-nsa-prefill-context-parallel \
--nsa-prefill-cp-mode round-robin-split \
--cuda-graph-max-bs 128 \
--max-running-requests 128 \
--trust-remote-code --host "0.0.0.0" --port 30000 \
--log-requests \
--context-length 65536 \
--allow-auto-truncate --enable-metrics \
--tool-call-parser deepseekv32 --reasoning-parser deepseek-v3 \
--served-model-name DeepSeek-V3.2-nsa-pp-cp-ep-dp
)
sglang_args=($sglang_args)
sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log
```
**fp8 KV + CP + PP**
With FP8 KV, we can have less memory footprint. This can be combined with various parallel schemes:
```shell Command
# verified in Hopper platform
dp=1
dp_config=" \
--dp 1 --enable-dp-attention \
"
cp_config=" \
--enable-nsa-prefill-context-parallel \
"
if [ "$dp" -eq 1 ]; then
cp_config=" \
$cp_config \
--nsa-prefill-cp-mode round-robin-split \
"
else
cp_config=" \
$cp_config \
--nsa-prefill-cp-mode in-seq-split \
"
fi
# see discussion : https://github.com/sgl-project/sglang/pull/12065
sglang_args=$(echo serve \
--model-path $MAPPED_MODEL_PATH \
--nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 8 --pp-size 2 --pp-async-batch-depth 1 \
$dp_config \
--trust-remote-code --host "0.0.0.0" --port 30000 \
--log-requests \
--context-length 65536 --max-running-requests 128 \
$cp_config \
--kv-cache-dtype fp8_e4m3 \
--allow-auto-truncate --enable-metrics \
--tool-call-parser deepseekv32 --reasoning-parser deepseek-v3 \
--served-model-name DeepSeek-V3.2-Opt-fp8kv-pp2-cp4
)
sglang_args=($sglang_args)
sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log
```
## 5. Benchmark
### 5.1 Speed Benchmark on Blackwell
**Test Environment:**
- Hardware: NVIDIA B200 GPU (8x)
- Model: DeepSeek-V3.2-Exp
- Tensor Parallelism: 8
- sglang version: 0.5.6
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses.
#### 5.1.1 Latency-Sensitive Benchmark
- Model Deployment Command:
```shell Command
sglang serve \
--model-path deepseek-ai/DeepSeek-V3.2-Exp \
--tp 8 \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--host 0.0.0.0 \
--port 30000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model deepseek-ai/DeepSeek-V3.2-Exp \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 10 \
--max-concurrency 1
```
- **Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 29.11
Total input tokens: 1972
Total input text tokens: 1972
Total input vision tokens: 0
Total generated tokens: 2784
Total generated tokens (retokenized): 2777
Request throughput (req/s): 0.34
Input token throughput (tok/s): 67.73
Output token throughput (tok/s): 95.62
Peak output token throughput (tok/s): 157.00
Peak concurrent requests: 3
Total token throughput (tok/s): 163.36
Concurrency: 1.00
Accept length: 2.46
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 2909.74
Median E2E Latency (ms): 3088.27
P90 E2E Latency (ms): 4200.62
P99 E2E Latency (ms): 5588.52
---------------Time to First Token----------------
Mean TTFT (ms): 317.58
Median TTFT (ms): 191.31
P99 TTFT (ms): 740.79
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 9.09
Median TPOT (ms): 9.25
P99 TPOT (ms): 11.73
---------------Inter-Token Latency----------------
Mean ITL (ms): 9.35
Median ITL (ms): 7.64
P95 ITL (ms): 22.81
P99 ITL (ms): 23.33
Max ITL (ms): 31.45
==================================================
```
#### 5.1.2 Throughput-Sensitive Benchmark
- Model Deployment Command:
```shell Command
sglang serve \
--model-path deepseek-ai/DeepSeek-V3.2-Exp \
--tp 8 \
--ep 8 \
--dp 8 \
--enable-dp-attention \
--host 0.0.0.0 \
--port 30000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model deepseek-ai/DeepSeek-V3.2-Exp \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 1000 \
--max-concurrency 100
```
- **Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 219.09
Total input tokens: 301701
Total input text tokens: 301701
Total input vision tokens: 0
Total generated tokens: 188375
Total generated tokens (retokenized): 187443
Request throughput (req/s): 4.56
Input token throughput (tok/s): 1377.06
Output token throughput (tok/s): 859.80
Peak output token throughput (tok/s): 2465.00
Peak concurrent requests: 109
Total token throughput (tok/s): 2236.86
Concurrency: 88.05
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 19291.23
Median E2E Latency (ms): 11927.39
---------------Time to First Token----------------
Mean TTFT (ms): 530.36
Median TTFT (ms): 444.00
P99 TTFT (ms): 1504.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 106.16
Median TPOT (ms): 106.69
P99 TPOT (ms): 221.12
---------------Inter-Token Latency----------------
Mean ITL (ms): 100.46
Median ITL (ms): 41.73
P95 ITL (ms): 225.67
P99 ITL (ms): 392.37
Max ITL (ms): 975.03
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 GSM8K Benchmark
- **Benchmark Command:**
```shell Command
python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 30000
```
- **Test Results**:
- DeepSeek-V3.2-Exp
```
Accuracy: 0.980
Invalid: 0.000
Latency: 19.128 s
Output throughput: 965.919 token/s
```
#### 5.2.2 MMLU Benchmark
- **Benchmark Command:**
```shell Command
cd sglang
bash benchmark/mmlu/download_data.sh
python3 benchmark/mmlu/bench_sglang.py --nsub 10 --port 30000
```
- **Test Results**:
- DeepSeek-V3.2-Exp
```
subject: abstract_algebra, #q:100, acc: 0.780
subject: anatomy, #q:135, acc: 0.874
subject: astronomy, #q:152, acc: 0.961
subject: business_ethics, #q:100, acc: 0.860
subject: clinical_knowledge, #q:265, acc: 0.925
subject: college_biology, #q:144, acc: 0.972
subject: college_chemistry, #q:100, acc: 0.660
subject: college_computer_science, #q:100, acc: 0.880
subject: college_mathematics, #q:100, acc: 0.840
subject: college_medicine, #q:173, acc: 0.879
Total latency: 7.961
Average accuracy: 0.879
```
### 5.3 Speed Benchmark on Hopper
**Test Environment:**
- Hardware: NVIDIA H800 GPU (16x)
- Model: DeepSeek-V3.2
- Tensor Parallelism: 16
- sglang version: 0.5.9
#### 5.3.1 Latency-Sensitive Benchmark
- Model Deployment Command:
```shell Command
export SGLANG_DEEPEP_LL_COMBINE_SEND_NUM_SMS=32
export SGLANG_SET_CPU_AFFINITY=1
# Test workload ISL/OSL=1k/1k, raw tap : 4948.16 toks/sec, MAX ITL 5970
# dp 2 : 5019.54 toks/sec, MAX ITL 7233
# dp 4 : 4942.82 toks/sec, MAX ITL 35654
# dp 2 + mtp : 6842.51 toks/sec, MAX ITL 3081
sglang_args=$(echo serve \
--model-path $MAPPED_MODEL_PATH \
--nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 16 \
--dp 2 --enable-dp-attention --page-size 64 \
--trust-remote-code --host "0.0.0.0" --port 30000 \
--log-requests \
--context-length 65536 --max-running-requests 128 \
--speculative-algorithm EAGLE \
--speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3 \
--allow-auto-truncate --enable-metrics \
--tool-call-parser deepseekv32 --reasoning-parser deepseek-v3 \
--served-model-name DeepSeek-V3.2-Opt-dp2-mtp
)
sglang_args=($sglang_args)
sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host $MASTER_ADDR \
--port 30000 \
--model deepseek-ai/DeepSeek-V3.2 \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 10 \
--max-concurrency 1
```
- **Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: 64.0
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 48.96
Total input tokens: 6101
Total input text tokens: 6101
Total generated tokens: 4220
Total generated tokens (retokenized): 4217
Request throughput (req/s): 0.20
Input token throughput (tok/s): 124.62
Output token throughput (tok/s): 86.20
Peak output token throughput (tok/s): 113.00
Peak concurrent requests: 2
Total token throughput (tok/s): 210.81
Concurrency: 1.00
Accept length: 3.27
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 4893.12
Median E2E Latency (ms): 3742.47
P90 E2E Latency (ms): 8877.37
P99 E2E Latency (ms): 10769.85
---------------Time to First Token----------------
Mean TTFT (ms): 199.88
Median TTFT (ms): 176.15
P99 TTFT (ms): 272.49
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 10.99
Median TPOT (ms): 10.88
P99 TPOT (ms): 13.93
---------------Inter-Token Latency----------------
Mean ITL (ms): 11.15
Median ITL (ms): 8.86
P95 ITL (ms): 17.29
P99 ITL (ms): 33.71
Max ITL (ms): 36.84
==================================================
```
#### 5.3.2 Throughput-Sensitive Benchmark
We simply use the same deployment method and vary the throughput by maximizing concurrencies:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host $MASTER_ADDR \
--port 30000 \
--model deepseek-ai/DeepSeek-V3.2 \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 2048 \
--max-concurrency 1024 # see picture below why we use 1024 for concurrency, hence num prompts 2048
```
DeepSeek 3.2 can steadily support concurrency up to `1024` and when concurrency is greater than `128`, the TTFT increase sharply:
![DeepSeek V3.2 Concurrency ISL/OSL=1024/128](https://github.com/user-attachments/assets/d5c9c9fb-44f3-4793-a0fd-f8fa954546f5)
Performance record:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: 64.0
Max request concurrency: 1024
Successful requests: 2048
Benchmark duration (s): 408.09
Total input tokens: 1048992
Total input text tokens: 1048992
Total generated tokens: 1032734
Total generated tokens (retokenized): 1031817
Request throughput (req/s): 5.02
Input token throughput (tok/s): 2570.50
Output token throughput (tok/s): 2530.66
Peak output token throughput (tok/s): 5092.00
Peak concurrent requests: 1035
Total token throughput (tok/s): 5101.16
Concurrency: 763.41
Accept length: 3.26
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 152117.70
Median E2E Latency (ms): 181704.84
P90 E2E Latency (ms): 215924.77
P99 E2E Latency (ms): 231679.59
---------------Time to First Token----------------
Mean TTFT (ms): 127729.28
Median TTFT (ms): 170098.94
P99 TTFT (ms): 185705.73
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 49.18
Median TPOT (ms): 48.48
P99 TPOT (ms): 77.24
---------------Inter-Token Latency----------------
Mean ITL (ms): 48.46
Median ITL (ms): 52.11
P95 ITL (ms): 110.26
P99 ITL (ms): 200.63
Max ITL (ms): 2666.37
==================================================
```
By adding `--random-range-ratio 1`, we could get even higher statistical numbers:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: 64.0
Max request concurrency: 1024
Successful requests: 2048
Benchmark duration (s): 612.87
Total input tokens: 2097152
Total input text tokens: 2097152
Total generated tokens: 2097152
Total generated tokens (retokenized): 2096201
Request throughput (req/s): 3.34
Input token throughput (tok/s): 3421.84
Output token throughput (tok/s): 3421.84
Peak output token throughput (tok/s): 9077.00
Peak concurrent requests: 1039
Total token throughput (tok/s): 6843.68
Concurrency: 772.66
Accept length: 3.26
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 231222.27
Median E2E Latency (ms): 289846.24
P90 E2E Latency (ms): 314480.41
P99 E2E Latency (ms): 320392.27
---------------Time to First Token----------------
Mean TTFT (ms): 194081.02
Median TTFT (ms): 252945.22
P99 TTFT (ms): 279637.50
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 36.31
Median TPOT (ms): 36.73
P99 TPOT (ms): 46.33
---------------Inter-Token Latency----------------
Mean ITL (ms): 36.31
Median ITL (ms): 23.18
P95 ITL (ms): 96.79
P99 ITL (ms): 135.81
Max ITL (ms): 3121.00
==================================================
```
docs_new/cookbook/autoregressive/Ernie/Ernie4.5-VL.mdx
+28
View File
@@ -0,0 +1,28 @@
---
title: Ernie4.5-VL
metatags:
description: "Deploy Ernie4.5-VL vision-language model with SGLang - community contribution guide for Baidu's multimodal model."
---
## 📝 Community Contribution Welcome
This guide is currently under development. We welcome community contributions!
If you have experience deploying **Ernie4.5-VL** with SGLang, please help us complete this documentation.
## 🚀 How to Contribute
```shell Command
git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git
cd sglang-cookbook
git checkout -b add-ernie4-5-vl-guide
# Edit this file and submit a PR
```
## 📚 Reference
- [GLM-4.6V](../GLM/GLM-4.6V)
---
**Let's build this together!** 🌟
docs_new/cookbook/autoregressive/Ernie/Ernie4.5.mdx
+696
View File
@@ -0,0 +1,696 @@
---
title: Ernie4.5
metatags:
description: "Deploy Ernie4.5 with SGLang - community contribution guide for Baidu's Ernie 4.5 model deployment."
---
import { Ernie45Deployment } from '/src/snippets/autoregressive/ernie-45-deployment.jsx';
## 1. Model Introduction
The **ERNIE-4.5** series is a family of large language models developed by Baidu. ERNIE (Enhanced Representation through Knowledge Integration) 4.5 represents an advanced version of the ERNIE series, optimized for general-purpose tasks and conversational scenarios.
ERNIE-4.5 delivers advanced features as below:
- **Heterogeneous Modality Structure**: MoE architecture that supports parameter sharing across modalities while allowing dedicated parameters for each individual modality, enhancing multimodal understanding without compromising, and even improving, performance on text-related tasks.
- **Vision Encoder**: Dedicated adaptive-resolution ViT with 2D RoPE and image packing; for video, adaptive frame sampling and timestamp rendering, supporting both shared and modality-specific visual processing.
- **Adapter**: Shared modality-bridging module with spatial and temporal compression to align vision to text embedding space, enabling cross-modal understanding without compromising text representations.
- **Multimodal Position Embedding**: Unified 3D RoPE (temporal, height, width) for vision and 1D RoPE for text in a single embedding space, supporting parameter sharing while encoding modality-specific positions.
- **Hardware Optimization**: Specifically tuned for AMD MI300X, MI325X, and MI355X GPUs.
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
<Ernie45Deployment />
## 4. API Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
The following example demonstrates deployment using ERNIE-4.5-21B-A3B-PT.
```shell Command
python -m sglang.launch_server \
--model baidu/ERNIE-4.5-21B-A3B-PT \
--tp 1
```
**Basic Python Client Example:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="baidu/ERNIE-4.5-21B-A3B-PT",
messages=[
{"role": "user", "content": "What is artificial intelligence?"}
],
temperature=1.0,
top_p=0.95,
max_tokens=1024
)
print(response.choices[0].message.content)
```
**Output Example:**
```text Output
**Artificial Intelligence (AI)** is the simulation of human intelligence processes by machines, particularly computer systems. These processes include **learning** (acquiring information and rules for using the information), **reasoning** (using rules to reach approximate or definite conclusions), and **self-correction**. AI encompasses a wide range of techniques, algorithms, and methodologies designed to enable machines to perform tasks that typically require human intelligence.
### Key Characteristics of AI:
...
### In Summary:
AI represents a transformative force with the potential to revolutionize industries and enhance human capabilities. However, its development requires careful consideration of ethical, legal, and social implications to ensure that it benefits society as a whole. As AI continues to evolve, ongoing dialogue among stakeholders will be crucial to balancing innovation with responsibility.
```
**Streaming Example:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="baidu/ERNIE-4.5-21B-A3B-PT",
messages=[
{"role": "user", "content": "Explain quantum computing in simple terms."}
],
temperature=1.0,
top_p=0.95,
max_tokens=2048,
stream=True
)
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if delta.content:
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
Sure! Here’s a simple explanation of quantum computing:
### **Quantum Computing: Making Computers Super Fast (But Weird) Using Quantum Rules**
1. **Classic vs. Quantum Computers**
- **Normal computers** use **bits** (0s and 1s) to store and process information.
- **Quantum computers** use **qubits** (short for quantum bits). Unlike bits, qubits can be **0, 1, or both at the same time** (this is called **superposition**).
2. **Superposition: The Magic Behind Speed**
- A single qubit can represent **0 and 1 simultaneously**, like a coin spinning in the air.
- Many qubits working together (in something called **quantum parallelism**) can **check multiple possibilities at once**, making quantum computers much faster for certain problems.
3. **Entanglement: Making Qubits Link**
- When qubits are **entangled**, their states are linked—changing one instantly affects the other, no matter how far apart they are (this is called **spooky action at a distance** by Einstein).
- Entanglement allows quantum computers to process information in **very efficient ways**.
4. **What Quantum Computers Are Good At**
- **Cracking encryption** (like RSA).
- **Factoring large numbers** (used in encryption and cryptography).
- **Searching unsorted databases** (way faster than classical computers).
- **Simulating quantum systems** (like molecules for drug discovery).
- **Optimizing problems** (like logistics or finance).
5. **Challenges & Current State**
- Qubits are **fragile** and easily disturbed (called **decoherence**).
- Engineers are working to keep qubits stable long enough to do useful calculations.
- Today’s quantum computers are **small and experimental**, but the goal is to build powerful ones that outperform classical supercomputers.
### **Final Thought**
Quantum computing isn’t just a faster calculator—it’s a **new way of thinking about problems** using the weird laws of physics. While still new, it has the potential to revolutionize fields like medicine, AI, and cybersecurity.
Would you like an example of how a quantum computer might solve a problem? 😊
```
## 5. Benchmark
This section uses **industry-standard configurations** for comparable benchmark results.
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: AMD MI300X GPU (1x)
- Model: ERNIE-4.5-21B-A3B-PT
- Tensor Parallelism: 1
- SGLang Version: 0.5.7
**Benchmark Methodology:**
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
#### 5.1.1 Standard Scenario Benchmark
- Model Deployment Command:
```bash Command
python -m sglang.launch_server \
--model-path baidu/ERNIE-4.5-21B-A3B-PT \
--tp 1
```
##### 5.1.1.1 Low Concurrency (Latency-Optimized)
- Benchmark Command:
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model baidu/ERNIE-4.5-21B-A3B-PT \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 58.72
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4219
Request throughput (req/s): 0.17
Input token throughput (tok/s): 103.90
Output token throughput (tok/s): 71.87
Peak output token throughput (tok/s): 245.00
Peak concurrent requests: 2
Total token throughput (tok/s): 175.77
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 5869.86
Median E2E Latency (ms): 1870.80
---------------Time to First Token----------------
Mean TTFT (ms): 4152.58
Median TTFT (ms): 36.81
P99 TTFT (ms): 37498.23
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 4.07
Median TPOT (ms): 4.09
P99 TPOT (ms): 4.09
---------------Inter-Token Latency----------------
Mean ITL (ms): 4.08
Median ITL (ms): 4.08
P95 ITL (ms): 4.14
P99 ITL (ms): 4.20
Max ITL (ms): 4.67
==================================================
```
##### 5.1.1.2 Medium Concurrency (Balanced)
- Benchmark Command:
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model baidu/ERNIE-4.5-21B-A3B-PT \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 34.30
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 40805
Total generated tokens (retokenized): 40773
Request throughput (req/s): 2.33
Input token throughput (tok/s): 1156.62
Output token throughput (tok/s): 1189.77
Peak output token throughput (tok/s): 1392.00
Peak concurrent requests: 21
Total token throughput (tok/s): 2346.39
Concurrency: 14.14
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 6060.62
Median E2E Latency (ms): 6496.70
---------------Time to First Token----------------
Mean TTFT (ms): 78.90
Median TTFT (ms): 45.90
P99 TTFT (ms): 234.33
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 11.99
Median TPOT (ms): 12.16
P99 TPOT (ms): 14.81
---------------Inter-Token Latency----------------
Mean ITL (ms): 11.75
Median ITL (ms): 11.48
P95 ITL (ms): 12.24
P99 ITL (ms): 34.85
Max ITL (ms): 105.01
==================================================
```
##### 5.1.1.3 High Concurrency (Throughput-Optimized)
- Benchmark Command:
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model baidu/ERNIE-4.5-21B-A3B-PT \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 66.63
Total input tokens: 249831
Total input text tokens: 249831
Total input vision tokens: 0
Total generated tokens: 252662
Total generated tokens (retokenized): 252449
Request throughput (req/s): 7.50
Input token throughput (tok/s): 3749.79
Output token throughput (tok/s): 3792.28
Peak output token throughput (tok/s): 4902.00
Peak concurrent requests: 113
Total token throughput (tok/s): 7542.06
Concurrency: 90.33
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 12036.90
Median E2E Latency (ms): 11782.16
---------------Time to First Token----------------
Mean TTFT (ms): 104.86
Median TTFT (ms): 84.62
P99 TTFT (ms): 297.85
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 23.89
Median TPOT (ms): 24.62
P99 TPOT (ms): 26.91
---------------Inter-Token Latency----------------
Mean ITL (ms): 23.66
Median ITL (ms): 20.48
P95 ITL (ms): 45.57
P99 ITL (ms): 54.31
Max ITL (ms): 185.12
==================================================
```
#### 5.1.2 Reasoning Scenario Benchmark
##### 5.1.2.1 Low Concurrency
- Benchmark Command:
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model baidu/ERNIE-4.5-21B-A3B-PT \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 185.11
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 44462
Total generated tokens (retokenized): 44423
Request throughput (req/s): 0.05
Input token throughput (tok/s): 32.96
Output token throughput (tok/s): 240.19
Peak output token throughput (tok/s): 245.00
Peak concurrent requests: 2
Total token throughput (tok/s): 273.15
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 18508.84
Median E2E Latency (ms): 19866.81
---------------Time to First Token----------------
Mean TTFT (ms): 32.59
Median TTFT (ms): 32.14
P99 TTFT (ms): 38.58
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 4.13
Median TPOT (ms): 4.13
P99 TPOT (ms): 4.20
---------------Inter-Token Latency----------------
Mean ITL (ms): 4.16
Median ITL (ms): 4.12
P95 ITL (ms): 4.31
P99 ITL (ms): 4.36
Max ITL (ms): 7.28
==================================================
```
##### 5.1.2.2 Medium Concurrency
- Benchmark Command:
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model baidu/ERNIE-4.5-21B-A3B-PT \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 263.48
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 318306
Total generated tokens (retokenized): 317984
Request throughput (req/s): 0.30
Input token throughput (tok/s): 150.55
Output token throughput (tok/s): 1208.09
Peak output token throughput (tok/s): 1408.00
Peak concurrent requests: 19
Total token throughput (tok/s): 1358.64
Concurrency: 14.35
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 47249.55
Median E2E Latency (ms): 47828.67
---------------Time to First Token----------------
Mean TTFT (ms): 62.77
Median TTFT (ms): 57.10
P99 TTFT (ms): 93.70
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 11.92
Median TPOT (ms): 12.09
P99 TPOT (ms): 12.50
---------------Inter-Token Latency----------------
Mean ITL (ms): 11.86
Median ITL (ms): 12.04
P95 ITL (ms): 12.68
P99 ITL (ms): 13.61
Max ITL (ms): 39.94
==================================================
```
##### 5.1.2.3 High Concurrency
- Benchmark Command:
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model baidu/ERNIE-4.5-21B-A3B-PT \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 320 \
--max-concurrency 64 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 428.30
Total input tokens: 158939
Total input text tokens: 158939
Total input vision tokens: 0
Total generated tokens: 1301025
Total generated tokens (retokenized): 1299877
Request throughput (req/s): 0.75
Input token throughput (tok/s): 371.09
Output token throughput (tok/s): 3037.63
Peak output token throughput (tok/s): 3880.00
Peak concurrent requests: 69
Total token throughput (tok/s): 3408.73
Concurrency: 57.08
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 76392.58
Median E2E Latency (ms): 79698.73
---------------Time to First Token----------------
Mean TTFT (ms): 92.79
Median TTFT (ms): 78.71
P99 TTFT (ms): 168.89
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 18.81
Median TPOT (ms): 19.15
P99 TPOT (ms): 19.81
---------------Inter-Token Latency----------------
Mean ITL (ms): 18.77
Median ITL (ms): 18.77
P95 ITL (ms): 19.86
P99 ITL (ms): 42.08
Max ITL (ms): 74.36
==================================================
```
#### 5.1.3 Summarization Scenario Benchmark
##### 5.1.3.1 Low Concurrency
- Benchmark Command:
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model baidu/ERNIE-4.5-21B-A3B-PT \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 18.59
Total input tokens: 41941
Total input text tokens: 41941
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4216
Request throughput (req/s): 0.54
Input token throughput (tok/s): 2256.43
Output token throughput (tok/s): 227.04
Peak output token throughput (tok/s): 245.00
Peak concurrent requests: 2
Total token throughput (tok/s): 2483.46
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 1856.72
Median E2E Latency (ms): 1513.87
---------------Time to First Token----------------
Mean TTFT (ms): 86.66
Median TTFT (ms): 72.30
P99 TTFT (ms): 167.13
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 4.19
Median TPOT (ms): 4.22
P99 TPOT (ms): 4.30
---------------Inter-Token Latency----------------
Mean ITL (ms): 4.20
Median ITL (ms): 4.23
P95 ITL (ms): 4.34
P99 ITL (ms): 4.42
Max ITL (ms): 5.68
==================================================
```
##### 5.1.3.2 Medium Concurrency
- Benchmark Command:
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model baidu/ERNIE-4.5-21B-A3B-PT \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 40.25
Total input tokens: 300020
Total input text tokens: 300020
Total input vision tokens: 0
Total generated tokens: 41669
Total generated tokens (retokenized): 41646
Request throughput (req/s): 1.99
Input token throughput (tok/s): 7454.72
Output token throughput (tok/s): 1035.37
Peak output token throughput (tok/s): 1310.00
Peak concurrent requests: 20
Total token throughput (tok/s): 8490.09
Concurrency: 14.37
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 7229.56
Median E2E Latency (ms): 7578.95
---------------Time to First Token----------------
Mean TTFT (ms): 137.38
Median TTFT (ms): 122.59
P99 TTFT (ms): 485.34
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 14.04
Median TPOT (ms): 14.24
P99 TPOT (ms): 20.77
---------------Inter-Token Latency----------------
Mean ITL (ms): 13.64
Median ITL (ms): 12.36
P95 ITL (ms): 14.72
P99 ITL (ms): 57.39
Max ITL (ms): 411.31
==================================================
```
##### 5.1.3.3 High Concurrency
- Benchmark Command:
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model baidu/ERNIE-4.5-21B-A3B-PT \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 320 \
--max-concurrency 64 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 78.33
Total input tokens: 1273893
Total input text tokens: 1273893
Total input vision tokens: 0
Total generated tokens: 170000
Total generated tokens (retokenized): 169888
Request throughput (req/s): 4.09
Input token throughput (tok/s): 16262.33
Output token throughput (tok/s): 2170.20
Peak output token throughput (tok/s): 3005.00
Peak concurrent requests: 73
Total token throughput (tok/s): 18432.53
Concurrency: 58.79
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 14392.52
Median E2E Latency (ms): 14460.70
---------------Time to First Token----------------
Mean TTFT (ms): 184.82
Median TTFT (ms): 155.24
P99 TTFT (ms): 379.82
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 26.97
Median TPOT (ms): 28.31
P99 TPOT (ms): 33.61
---------------Inter-Token Latency----------------
Mean ITL (ms): 26.79
Median ITL (ms): 20.55
P95 ITL (ms): 47.55
P99 ITL (ms): 145.64
Max ITL (ms): 287.62
==================================================
```
### 5.2 Accuracy Benchmark
Document model accuracy on standard benchmarks:
#### 5.2.1 GSM8K Benchmark
- Benchmark Command:
```bash Command
python3 benchmark/gsm8k/bench_sglang.py \
--num-shots 8 \
--num-questions 1316 \
--parallel 1316
```
- Test Results:
- ERNIE-4.5-21B-A3B-PT
```
Accuracy: 0.865
Invalid: 0.000
Latency: 21.669 s
Output throughput: 10359.790 token/s
```
docs_new/cookbook/autoregressive/FlashLabs/Chroma1.0.mdx
+193
View File
@@ -0,0 +1,193 @@
---
title: Chroma-1.0
metatags:
description: "Deploy Chroma-1.0 end-to-end speech conversation model with SGLang - real-time speech generation, voice cloning, and speech reasoning."
tag: NEW
---
## 1. Model Introduction
[Chroma-1.0](https://github.com/FlashLabs-AI-Corp/FlashLabs-Chroma) is an open-source end-to-end speech conversation model developed by FlashLabs, focusing on the following core capabilities:
- **Real-time Speech Generation**: Supports low-latency speech synthesis, suitable for real-time conversational scenarios.
- **Customized Voice Cloning**: Capable of cloning and replicating specific speaker voice characteristics.
- **End-to-End Architecture**: Provides a complete processing workflow from speech to speech.
- **Speech Reasoning**: Equipped with reasoning capabilities to understand and process speech content.
## 2. Architecture Overview
**Chroma-1.0** utilizes a hybrid serving architecture rather than a direct SGLang deployment. This design choice is driven by:
1. **Complex Model Architecture**: The end-to-end speech processing pipeline involves specialized components that go beyond standard text generation loops.
2. **KV Cache & State Management**: The model requires custom handling of KV caches that differs from standard implementations.
3. **Batching Limitations**: The current implementation supports a batch size of 1, meaning SGLang's advanced continuous batching capabilities are not yet fully applicable.
Therefore, you will start the **FlashLabs Server**, which manages the overall workflow and selectively leverages SGLang for specific inference components where supported.
- **Outer Layer**: FlashLabs Server (Handles Audio I/O, State, and Model Logic)
- **Inner Engine**: SGLang Instance (Utilized for specific acceleration where applicable)
## 3. Installation & Setup
We recommend following these steps to set up the environment and prepare the model.
### Step 1: Get the Docker Image
Pull the official pre-built image from Docker Hub to ensure all dependencies are correctly configured.
```bash Command
docker pull flashlabs/chroma:latest
```
### Step 2: Download Model Weights
Download the **Chroma-4B** weights from Hugging Face. You can choose one of the following methods:
**Method 1: Using Python (Recommended)**
```bash Command
huggingface-cli download FlashLabs/Chroma-4B --local-dir Chroma-4B
```
**Method 2: Using Git Clone**
Make sure you have Git LFS installed before cloning.
```bash Command
# Install Git LFS first
git lfs install
# Clone the repository
git clone https://huggingface.co/FlashLabs/Chroma-4B Chroma-4B
```
### Step 3: Download Chroma Codes (SGLang version)
```bash Command
git clone https://github.com/FlashLabs-AI-Corp/Chroma-SGLang.git
cd Chroma-SGLang
```
### Step 4: Run the Server
```bash Command
docker run -d \
--gpus all \
-p 8000:8000 \
-w /app/Chroma-SGLang \
-v "your_Chroma-SGLang_path":/app/Chroma-SGLang \
-v "your_chroma_path":/model \
-e CHROMA_MODEL_PATH=/model \
-e DP_SIZE="1" \
flashlabs/chroma:latest \
/opt/conda/bin/python -m uvicorn api_server:app \
--host 0.0.0.0 \
--port 8000 \
--workers 1
```
or run simply the following one line command
```bash Command
docker-compose up -d
```
## 5. Client Usage Example
Once the server is running, you can interact with it using HTTP requests.
### Python Client
```python Example
import requests
import base64
url = "http://localhost:8000/v1/chat/completions"
headers = {"Content-Type": "application/json"}
payload = {
"model": "chroma",
"messages": [
{
"role": "system",
"content": "You are Chroma, a voice agent developed by FlashLabs."
},
{
"role": "user",
"content": [
{"type": "audio", "audio": "assets/question_audio.wav"}
]
}
],
"max_tokens": 1000,
"return_audio": True
}
response = requests.post(url, json=payload, headers=headers)
result = response.json()
if result.get("audio"):
audio_data = base64.b64decode(result["audio"])
with open("output.wav", "wb") as f:
f.write(audio_data)
print("Audio saved to output.wav")
```
### OpenAI SDK Compatible Example
```python Example
from openai import OpenAI
client = OpenAI(
api_key="dummy",
base_url="http://localhost:8000/v1"
)
response = client.chat.completions.create(
model="chroma",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{
"role": "user",
"content": [
{"type": "audio", "audio": "assets/question_audio.wav"}
]
}
],
extra_body={
"prompt_text": "I have not... I'm so exhausted, I haven't slept in a very long time. It could be because... Well, I used our... Uh, I'm, I just use... This is what I use every day. I use our cleanser every day, I use serum in the morning and then the moistu- daily moisturizer. That's what I use every morning.",
"prompt_audio": "assets/ref_audio.wav",
"return_audio": True
}
)
print(response)
```
### CLI (cURL)
```bash Command
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "chroma",
"messages": [
{
"role": "system",
"content": "You are Chroma, a voice agent developed by FlashLabs."
},
{
"role": "user",
"content": [
{
"type": "audio",
"audio": "assets/question_audio.wav"
}
]
}
],
"max_tokens": 1000,
"return_audio": true
}' | jq -r '.audio' | base64 -d > output.wav
```
docs_new/cookbook/autoregressive/GLM/GLM-4.5.mdx
+490
View File
@@ -0,0 +1,490 @@
---
title: GLM-4.5
metatags:
description: "Deploy GLM-4.5 with SGLang on AMD GPUs - advanced reasoning, function calling, BF16/FP8 quantization options."
---
## 1. Model Introduction
[GLM-4.5](https://huggingface.co/zai-org/GLM-4.5) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding.
**Key Features:**
- **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving
- **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
- **Hardware Optimization**: Specifically tuned for AMD MI300X/MI325X/MI355X GPUs
- **High Performance**: Optimized for both throughput and latency scenarios
**Available Models:**
- **BF16 (Full precision)**: [zai-org/GLM-4.5](https://huggingface.co/zai-org/GLM-4.5) - Recommended for MI300X/MI325X/MI355X
- **FP8 (8-bit quantized)**: [zai-org/GLM-4.5-FP8](https://huggingface.co/zai-org/GLM-4.5-FP8) - Recommended for MI300X/MI325X/MI355X
**License:**
Please refer to the [official GLM-4.5 model card](https://huggingface.co/zai-org/GLM-4.5) for license details.
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.
import { GLM45Deployment } from "/src/snippets/autoregressive/glm-45-deployment.jsx";
<GLM45Deployment />
### 3.2 Configuration Tips
For more detailed configuration tips, please refer to [GLM-4.5/GLM-4.6 Usage](../../../docs/basic_usage/glm45).
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
GLM-4.5 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:
```shell Command
python -m sglang.launch_server \
--model zai-org/GLM-4.5 \
--reasoning-parser glm45 \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
**Streaming with Thinking Process:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="zai-org/GLM-4.5",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
To solve this problem, I need to calculate 15% of 240.
Step 1: Convert 15% to decimal: 15% = 0.15
Step 2: Multiply 240 by 0.15
Step 3: 240 × 0.15 = 36
=============== Content =================
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
```
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
#### 4.2.2 Tool Calling
GLM-4.5 supports tool calling capabilities. Enable the tool call parser:
```shell Command
python -m sglang.launch_server \
--model zai-org/GLM-4.5 \
--reasoning-parser glm45 \
--tool-call-parser glm45 \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
**Python Example (with Thinking Process):**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="zai-org/GLM-4.5",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
if tool_call.function:
print(f"Tool Call: {tool_call.function.name}")
print(f" Arguments: {tool_call.function.arguments}")
# Print content
if delta.content:
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
I should call the function with location="Beijing".
=============== Content =================
Tool Call: get_weather
Arguments: {"location": "Beijing", "unit": "celsius"}
```
## 5. Benchmark
This section uses **industry-standard configurations** for comparable benchmark results.
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: AMD MI300X (8x), AMD MI325X (8x), AMD MI355X (8x)
- Model: GLM-4.5
- Tensor Parallelism: 8
- SGLang Version: 0.5.6.post1
**Benchmark Methodology:**
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
#### 5.1.1 Standard Test Scenarios
Three core scenarios reflect real-world usage patterns:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
</tr>
</tbody>
</table>
#### 5.1.2 Concurrency Levels
Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
#### 5.1.3 Number of Prompts
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
---
#### 5.1.4 Benchmark Commands
**Scenario 1: Chat (1K/1K) - Most Important**
- **Model Deployment**
```bash Command
python -m sglang.launch_server \
--model zai-org/GLM-4.5 \
--tp 8
```
- Low Concurrency (Latency-Optimized)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.5 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
- Medium Concurrency (Balanced)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.5 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
- High Concurrency (Throughput-Optimized)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.5 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100 \
--request-rate inf
```
**Scenario 2: Reasoning (1K/8K)**
- Low Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.5 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
- Medium Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.5 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
- High Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.5 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 320 \
--max-concurrency 64 \
--request-rate inf
```
**Scenario 3: Summarization (8K/1K)**
- Low Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.5 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
- Medium Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.5 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
- High Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.5 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 320 \
--max-concurrency 64 \
--request-rate inf
```
#### 5.1.5 Understanding the Results
**Key Metrics:**
- **Request Throughput (req/s)**: Number of requests processed per second
- **Output Token Throughput (tok/s)**: Total tokens generated per second
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
**Why These Configurations Matter:**
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
- **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
**Interpreting Results:**
- Compare your results against baseline numbers for your hardware
- Higher throughput at same latency = better performance
- Lower TTFT = more responsive user experience
- Lower TPOT = faster generation speed
### 5.2 Accuracy Benchmark
Document model accuracy on standard benchmarks:
#### 5.2.1 GSM8K Benchmark
- Benchmark Command
```bash Command
python -m sglang.test.few_shot_gsm8k \
--num-questions 200 \
--port 30000
```
docs_new/cookbook/autoregressive/GLM/GLM-4.5V.mdx
+533
View File
@@ -0,0 +1,533 @@
---
title: GLM-4.5V
metatags:
description: "Deploy GLM-4.5V vision-language model with SGLang - SOTA multimodal performance, 64K context, image reasoning and video understanding."
---
## 1. Model Introduction
[GLM-4.5V](https://huggingface.co/zai-org/GLM-4.5V) is a state-of-the-art multimodal vision-language model from ZhipuAI, built on the next-generation flagship text foundation model GLM-4.5-Air (106B parameters, 12B active). It achieves SOTA performance among models of the same scale across 42 public vision-language benchmarks. Through efficient hybrid training, GLM-4.5V focuses on real-world usability and enables full-spectrum vision reasoning across diverse visual content types.
**Hardware Support:** NVIDIA B200/H100/H200, AMD MI300X/MI325X/MI355X
GLM-4.5V introduces several key features:
- **Image Reasoning & Grounding** Scene understanding, complex multi-image analysis, and spatial recognition with precise visual element localization. Supports bounding box predictions with normalized coordinates (0-1000) for accurate object detection.
- **Video Understanding** Long video segmentation and event recognition, supporting comprehensive temporal analysis across extended video sequences.
- **GUI Agent Tasks** Screen reading, icon recognition, and desktop operation assistance for agent-based applications. Enables natural interaction with graphical user interfaces.
- **Complex Chart & Long Document Parsing** Research report analysis and information extraction from documents with text, charts, tables, and figures. Processes up to 64K tokens of multimodal context.
- **Thinking Mode Switch** Allows users to balance between quick responses and deep reasoning. Users can enable/disable Chain-of-Thought reasoning based on task requirements for improved accuracy and interpretability.
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
The GLM-4.5V offers models in various sizes and architectures, optimized for different hardware platforms. The recommended launch configurations vary by hardware and model size.
**Interactive Command Generator**: Use the interactive configuration generator below to customize your deployment settings. Select your hardware platform, model size, quantization method, and other options to generate the appropriate launch command.
import { GLM45VDeployment } from "/src/snippets/autoregressive/glm-45v-deployment.jsx";
<GLM45VDeployment />
### 3.2 Configuration Tips
- **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size * number of images in current running requests.)
- **TP=8 Configuration**: When using Tensor Parallelism (TP) of 8, the vision attention's 12 heads cannot be evenly divided. You can resolve this by adding `--mm-enable-dp-encoder`.
- **Fast Model Loading**: For large models (like the 106B version), you can speed up model loading by using `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'`.
- For more detailed configuration tips, please refer to [GLM-4.5V/GLM-4.6V Usage](../../../docs/basic_usage/glmv).
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
### 4.2 Advanced Usage
#### 4.2.1 Multi-Modal Inputs
GLM-4.5V supports both image and video inputs. Here's a basic example with image input:
```python Example
import time
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:30000/v1",
timeout=3600
)
messages = [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
}
},
{
"type": "text",
"text": "Describe this image in detail."
}
]
}
]
start = time.time()
response = client.chat.completions.create(
model="zai-org/GLM-4.5V",
messages=messages,
max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```
**Example Output:**
```text Output
Response costs: 3.37s
Generated text: Auntie Anne's
CINNAMON SUGAR
1 x 17,000 17,000
SUB TOTAL 17,000
GRAND TOTAL 17,000
CASH IDR 20,000
CHANGE DUE 3,000
```
**Multi-Image Input Example:**
GLM-4.5V can process multiple images in a single request for comparison or analysis:
```python Example
import time
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:30000/v1",
timeout=3600
)
messages = [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
}
},
{
"type": "image_url",
"image_url": {
"url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
}
},
{
"type": "text",
"text": "Compare these two images and describe the differences in 100 words or less. Focus on the key visual elements, colors, textures, and any notable contrasts between the two scenes. Be specific about what you see in each image."
}
]
}
]
start = time.time()
response = client.chat.completions.create(
model="zai-org/GLM-4.5V",
messages=messages,
max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```
**Example Output:**
```text Output
Response costs: 3.86s
Generated text: The first image shows a close - up of a few red taxis on a street with storefronts in the background. The taxis are in a line, and the scene has an urban, busy feel with visible shop displays. The second image is an aerial view of a large taxi parking area with numerous red and green taxis, some with hoods open. The scene is more open, with a parking lot layout, and includes elements like a bridge and grassy areas. Key differences: number of taxis (few vs many), perspective (close - up vs aerial), color variety (mostly red vs red and green), and setting (street with shops vs parking lot).
```
**Video Input Example:**
GLM-4.5V supports video understanding by processing video URLs:
```python Example
import time
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:30000/v1",
timeout=3600
)
messages = [
{
"role": "user",
"content": [
{
"type": "video_url",
"video_url": {
"url": "https://videos.pexels.com/video-files/4114797/4114797-uhd_3840_2160_25fps.mp4"
}
},
{
"type": "text",
"text": "Describe what happens in this video."
}
]
}
]
start = time.time()
response = client.chat.completions.create(
model="zai-org/GLM-4.5V",
messages=messages,
max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```
**Note:**
- For video processing, ensure you have sufficient context length configured (up to 64K tokens)
- Video processing may require more memory; adjust `--mem-fraction-static` accordingly
- You can also provide local file paths using `file://` protocol
**Example Output:**
```text Output
Response costs: 3.89s
Generated text: A person wearing blue gloves is using a microscope. They are adjusting the focus knob with one hand while holding a pipette with the other, suggesting they are preparing or examining a sample on the slide beneath the objective lens. The microscope's 40x objective lens is positioned over the slide, indicating a high-magnification observation. The person carefully manipulates the slide and the microscope controls, likely to achieve a clear view of the specimen.
```
#### 4.2.2 Thinking Mode
GLM-4.5V supports thinking mode for enhanced reasoning. Enable thinking mode during deployment:
```shell Command
python -m sglang.launch_server \
--model-path zai-org/GLM-4.5V \
--reasoning-parser glm45 \
--tp 4 \
--host 0.0.0.0 \
--port 30000
```
**Streaming with Thinking Process:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="zai-org/GLM-4.5V",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
**Disable Thinking Mode:**
To disable thinking mode for a specific request:
```python Example
response = client.chat.completions.create(
model="zai-org/GLM-4.5V",
messages=[{"role": "user", "content": "What is the capital of France?"}],
extra_body={"chat_template_kwargs": {"enable_thinking": False}}
)
```
#### 4.2.3 Tool Calling
GLM-4.5V supports tool calling capabilities. Enable the tool call parser:
```shell Command
python -m sglang.launch_server \
--model-path zai-org/GLM-4.5V \
--reasoning-parser glm45 \
--tool-call-parser glm45 \
--tp 4 \
--host 0.0.0.0 \
--port 30000
```
**Python Example (with Thinking Process):**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="zai-org/GLM-4.5V",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Accumulate tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================\n", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {
'name': None,
'arguments': ''
}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]['name'] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
# Print content
if delta.content:
print(delta.content, end="", flush=True)
# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"🔧 Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
print()
```
**Output Example:**
```text Output
=============== Thinking =================
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
I should call the function with location="Beijing".
=============== Content =================
🔧 Tool Call: get_weather
Arguments: {"location": "Beijing", "unit": "celsius"}
```
**Note:**
- The reasoning parser shows how the model decides to use a tool
- Tool calls are clearly marked with the function name and arguments
- You can then execute the function and send the result back to continue the conversation
**Handling Tool Call Results:**
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="zai-org/GLM-4.5V",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```
## 5. Benchmark
### 5.1 Accuracy Benchmark
Document model accuracy on standard benchmarks:
#### 5.1.1 MMMU Benchmark
- Benchmark Command
```bash Command
python3 benchmark/mmmu/bench_sglang.py --response-answer-regex "<\|begin_of_box\|>(.*)<\|end_of_box\|>" --port 30000 --concurrency 64
```
- Test Result
```text Output
Benchmark time: 616.6163094160147
answers saved to: ./answer_sglang.json
Evaluating...
answers saved to: ./answer_sglang.json
{'Accounting': {'acc': 0.867, 'num': 30},
'Agriculture': {'acc': 0.567, 'num': 30},
'Architecture_and_Engineering': {'acc': 0.667, 'num': 30},
'Art': {'acc': 0.667, 'num': 30},
'Art_Theory': {'acc': 0.9, 'num': 30},
'Basic_Medical_Science': {'acc': 0.8, 'num': 30},
'Biology': {'acc': 0.6, 'num': 30},
'Chemistry': {'acc': 0.533, 'num': 30},
'Clinical_Medicine': {'acc': 0.667, 'num': 30},
'Computer_Science': {'acc': 0.8, 'num': 30},
'Design': {'acc': 0.867, 'num': 30},
'Diagnostics_and_Laboratory_Medicine': {'acc': 0.667, 'num': 30},
'Economics': {'acc': 0.833, 'num': 30},
'Electronics': {'acc': 0.433, 'num': 30},
'Energy_and_Power': {'acc': 0.733, 'num': 30},
'Finance': {'acc': 0.767, 'num': 30},
'Geography': {'acc': 0.667, 'num': 30},
'History': {'acc': 0.8, 'num': 30},
'Literature': {'acc': 0.9, 'num': 30},
'Manage': {'acc': 0.733, 'num': 30},
'Marketing': {'acc': 0.9, 'num': 30},
'Materials': {'acc': 0.567, 'num': 30},
'Math': {'acc': 0.8, 'num': 30},
'Mechanical_Engineering': {'acc': 0.767, 'num': 30},
'Music': {'acc': 0.3, 'num': 30},
'Overall': {'acc': 0.732, 'num': 900},
'Overall-Art and Design': {'acc': 0.683, 'num': 120},
'Overall-Business': {'acc': 0.82, 'num': 150},
'Overall-Health and Medicine': {'acc': 0.787, 'num': 150},
'Overall-Humanities and Social Science': {'acc': 0.783, 'num': 120},
'Overall-Science': {'acc': 0.707, 'num': 150},
'Overall-Tech and Engineering': {'acc': 0.648, 'num': 210},
'Pharmacy': {'acc': 0.9, 'num': 30},
'Physics': {'acc': 0.933, 'num': 30},
'Psychology': {'acc': 0.767, 'num': 30},
'Public_Health': {'acc': 0.9, 'num': 30},
'Sociology': {'acc': 0.667, 'num': 30}}
eval out saved to ./val_sglang.json
Overall accuracy: 0.732
```
docs_new/cookbook/autoregressive/GLM/GLM-4.6.mdx
+888
View File
@@ -0,0 +1,888 @@
---
title: GLM-4.6
metatags:
description: "Deploy GLM-4.6 with SGLang - 200K context window, superior coding, advanced reasoning, and enhanced agentic capabilities."
---
## 1. Model Introduction
[GLM-4.6](https://huggingface.co/zai-org/GLM-4.6) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding.
As the latest iteration in the GLM series, GLM-4.6 achieves comprehensive enhancements across multiple domains, including real-world coding, long-context processing, reasoning, searching, writing, and agentic applications. Details are as follows:
- **Longer context window**: The context window has been expanded from 128K to 200K tokens, enabling the model to handle more complex agentic tasks.
- **Superior coding performance**: The model achieves higher scores on code benchmarks and demonstrates better real-world performance in applications such as Claude Code, Cline, Roo Code and Kilo Code, including improvements in generating visually polished front-end pages.
- **Advanced reasoning**: GLM-4.6 shows a clear improvement in reasoning performance and supports tool use during inference, leading to stronger overall capability.
- **More capable agents**: GLM-4.6 exhibits stronger performance in tool use and search-based agents, and integrates more effectively within agent frameworks.
- **Refined writing**: Better aligns with human preferences in style and readability, and performs more naturally in role-playing scenarios.
For more details, please refer to the [official GLM-4.6 documentation](https://docs.z.ai/guides/llm/glm-4.6).
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.
import { GLM46Deployment } from "/src/snippets/autoregressive/glm-46-deployment.jsx";
<GLM46Deployment />
### 3.2 Configuration Tips
For more detailed configuration tips, please refer to [GLM-4.5/GLM-4.6 Usage](../../../docs/basic_usage/glm45).
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
GLM-4.6 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:
```shell Command
python -m sglang.launch_server \
--model zai-org/GLM-4.6 \
--reasoning-parser glm45 \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
**Streaming with Thinking Process:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="zai-org/GLM-4.6",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
To solve this problem, I need to calculate 15% of 240.
Step 1: Convert 15% to decimal: 15% = 0.15
Step 2: Multiply 240 by 0.15
Step 3: 240 × 0.15 = 36
=============== Content =================
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
```
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
#### 4.2.2 Tool Calling
GLM-4.6 supports tool calling capabilities. Enable the tool call parser:
```shell Command
python -m sglang.launch_server \
--model zai-org/GLM-4.6 \
--reasoning-parser glm45 \
--tool-call-parser glm45 \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
**Python Example (with Thinking Process):**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="zai-org/GLM-4.6",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
if tool_call.function:
print(f"🔧 Tool Call: {tool_call.function.name}")
print(f" Arguments: {tool_call.function.arguments}")
# Print content
if delta.content:
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
I should call the function with location="Beijing".
=============== Content =================
🔧 Tool Call: get_weather
Arguments: {"location": "Beijing", "unit": "celsius"}
```
**Note:**
- The reasoning parser shows how the model decides to use a tool
- Tool calls are clearly marked with the function name and arguments
- You can then execute the function and send the result back to continue the conversation
**Handling Tool Call Results:**
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="zai-org/GLM-4.6",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```
## 5. Benchmark
This section uses **industry-standard configurations** for comparable benchmark results.
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: NVIDIA B200 GPU (8x), AMD MI300X (8x), AMD MI325X (8x), AMD MI355X (8x)
- Model: GLM-4.6
- Tensor Parallelism: 8
- SGLang Version: 0.5.6.post1
**Benchmark Methodology:**
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
#### 5.1.1 Standard Test Scenarios
Three core scenarios reflect real-world usage patterns:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
</tr>
</tbody>
</table>
#### 5.1.2 Concurrency Levels
Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
#### 5.1.3 Number of Prompts
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
---
#### 5.1.4 Benchmark Commands
**Scenario 1: Chat (1K/1K) - Most Important**
- **Model Deployment**
```bash Command
python -m sglang.launch_server \
--model zai-org/GLM-4.6 \
--tp 8
```
- Low Concurrency (Latency-Optimized)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.6 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 63.82
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 4210
Total generated tokens (retokenized): 4209
Request throughput (req/s): 0.16
Input token throughput (tok/s): 95.60
Output token throughput (tok/s): 65.97
Peak output token throughput (tok/s): 68.00
Peak concurrent requests: 2
Total token throughput (tok/s): 161.57
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 6379.24
Median E2E Latency (ms): 5085.00
---------------Time to First Token----------------
Mean TTFT (ms): 155.57
Median TTFT (ms): 149.79
P99 TTFT (ms): 207.69
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 14.81
Median TPOT (ms): 14.80
P99 TPOT (ms): 14.84
---------------Inter-Token Latency----------------
Mean ITL (ms): 14.82
Median ITL (ms): 14.82
P95 ITL (ms): 15.17
P99 ITL (ms): 15.36
Max ITL (ms): 25.05
==================================================
```
- Medium Concurrency (Balanced)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.6 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 72.06
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 40725
Total generated tokens (retokenized): 40672
Request throughput (req/s): 1.11
Input token throughput (tok/s): 550.47
Output token throughput (tok/s): 565.14
Peak output token throughput (tok/s): 752.00
Peak concurrent requests: 20
Total token throughput (tok/s): 1115.61
Concurrency: 13.71
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 12348.93
Median E2E Latency (ms): 13164.81
---------------Time to First Token----------------
Mean TTFT (ms): 196.08
Median TTFT (ms): 155.22
P99 TTFT (ms): 377.98
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 24.24
Median TPOT (ms): 24.55
P99 TPOT (ms): 30.42
---------------Inter-Token Latency----------------
Mean ITL (ms): 23.92
Median ITL (ms): 21.40
P95 ITL (ms): 22.49
P99 ITL (ms): 123.83
Max ITL (ms): 486.54
==================================================
```
- High Concurrency (Throughput-Optimized)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.6 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 138.50
Total input tokens: 249831
Total input text tokens: 249831
Total input vision tokens: 0
Total generated tokens: 252162
Total generated tokens (retokenized): 251841
Request throughput (req/s): 3.61
Input token throughput (tok/s): 1803.78
Output token throughput (tok/s): 1820.61
Peak output token throughput (tok/s): 2900.00
Peak concurrent requests: 107
Total token throughput (tok/s): 3624.40
Concurrency: 90.91
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 25183.97
Median E2E Latency (ms): 23968.49
---------------Time to First Token----------------
Mean TTFT (ms): 337.77
Median TTFT (ms): 180.65
P99 TTFT (ms): 906.14
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 49.97
Median TPOT (ms): 52.20
P99 TPOT (ms): 61.81
---------------Inter-Token Latency----------------
Mean ITL (ms): 49.36
Median ITL (ms): 35.05
P95 ITL (ms): 124.91
P99 ITL (ms): 187.69
Max ITL (ms): 440.34
==================================================
```
**Scenario 2: Reasoning (1K/8K)**
- Low Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.6 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 666.64
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 44452
Total generated tokens (retokenized): 44387
Request throughput (req/s): 0.02
Input token throughput (tok/s): 9.15
Output token throughput (tok/s): 66.68
Peak output token throughput (tok/s): 68.00
Peak concurrent requests: 2
Total token throughput (tok/s): 75.83
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 66661.35
Median E2E Latency (ms): 71902.36
---------------Time to First Token----------------
Mean TTFT (ms): 160.21
Median TTFT (ms): 140.32
P99 TTFT (ms): 295.56
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 14.92
Median TPOT (ms): 14.94
P99 TPOT (ms): 15.02
---------------Inter-Token Latency----------------
Mean ITL (ms): 14.96
Median ITL (ms): 14.96
P95 ITL (ms): 15.36
P99 ITL (ms): 15.57
Max ITL (ms): 19.06
==================================================
```
- Medium Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.6 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 503.30
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 318226
Total generated tokens (retokenized): 318025
Request throughput (req/s): 0.16
Input token throughput (tok/s): 78.82
Output token throughput (tok/s): 632.28
Peak output token throughput (tok/s): 752.00
Peak concurrent requests: 19
Total token throughput (tok/s): 711.09
Concurrency: 13.88
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 87349.22
Median E2E Latency (ms): 88248.04
---------------Time to First Token----------------
Mean TTFT (ms): 228.54
Median TTFT (ms): 142.78
P99 TTFT (ms): 569.84
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 21.97
Median TPOT (ms): 22.14
P99 TPOT (ms): 22.47
---------------Inter-Token Latency----------------
Mean ITL (ms): 21.91
Median ITL (ms): 21.80
P95 ITL (ms): 22.30
P99 ITL (ms): 22.78
Max ITL (ms): 137.19
==================================================
```
- High Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.6 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 320 \
--max-concurrency 64 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 772.28
Total input tokens: 158939
Total input text tokens: 158939
Total input vision tokens: 0
Total generated tokens: 1300705
Total generated tokens (retokenized): 1299924
Request throughput (req/s): 0.41
Input token throughput (tok/s): 205.80
Output token throughput (tok/s): 1684.24
Peak output token throughput (tok/s): 2112.00
Peak concurrent requests: 68
Total token throughput (tok/s): 1890.05
Concurrency: 56.17
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 135563.36
Median E2E Latency (ms): 140888.88
---------------Time to First Token----------------
Mean TTFT (ms): 232.45
Median TTFT (ms): 145.59
P99 TTFT (ms): 576.49
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 33.47
Median TPOT (ms): 34.02
P99 TPOT (ms): 35.10
---------------Inter-Token Latency----------------
Mean ITL (ms): 33.30
Median ITL (ms): 32.63
P95 ITL (ms): 34.27
P99 ITL (ms): 104.39
Max ITL (ms): 155.65
==================================================
```
**Scenario 3: Summarization (8K/1K)**
- Low
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.6 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 65.11
Total input tokens: 41941
Total input text tokens: 41941
Total input vision tokens: 0
Total generated tokens: 4210
Total generated tokens (retokenized): 4210
Request throughput (req/s): 0.15
Input token throughput (tok/s): 644.17
Output token throughput (tok/s): 64.66
Peak output token throughput (tok/s): 68.00
Peak concurrent requests: 2
Total token throughput (tok/s): 708.83
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 6508.31
Median E2E Latency (ms): 5263.36
---------------Time to First Token----------------
Mean TTFT (ms): 189.48
Median TTFT (ms): 159.23
P99 TTFT (ms): 304.09
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 15.02
Median TPOT (ms): 15.03
P99 TPOT (ms): 15.27
---------------Inter-Token Latency----------------
Mean ITL (ms): 15.04
Median ITL (ms): 15.03
P95 ITL (ms): 15.46
P99 ITL (ms): 15.65
Max ITL (ms): 24.20
==================================================
```
- Medium Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.6 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 76.43
Total input tokens: 300020
Total input text tokens: 300020
Total input vision tokens: 0
Total generated tokens: 41589
Total generated tokens (retokenized): 41577
Request throughput (req/s): 1.05
Input token throughput (tok/s): 3925.47
Output token throughput (tok/s): 544.15
Peak output token throughput (tok/s): 752.00
Peak concurrent requests: 19
Total token throughput (tok/s): 4469.62
Concurrency: 13.95
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 13329.63
Median E2E Latency (ms): 14141.09
---------------Time to First Token----------------
Mean TTFT (ms): 339.88
Median TTFT (ms): 252.75
P99 TTFT (ms): 906.54
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 25.37
Median TPOT (ms): 25.73
P99 TPOT (ms): 30.94
---------------Inter-Token Latency----------------
Mean ITL (ms): 25.04
Median ITL (ms): 21.68
P95 ITL (ms): 22.69
P99 ITL (ms): 146.98
Max ITL (ms): 483.14
==================================================
```
- High Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.6 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 320 \
--max-concurrency 64 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 136.24
Total input tokens: 1273893
Total input text tokens: 1273893
Total input vision tokens: 0
Total generated tokens: 169680
Total generated tokens (retokenized): 169452
Request throughput (req/s): 2.35
Input token throughput (tok/s): 9350.32
Output token throughput (tok/s): 1245.44
Peak output token throughput (tok/s): 1984.00
Peak concurrent requests: 69
Total token throughput (tok/s): 10595.77
Concurrency: 58.46
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 24889.40
Median E2E Latency (ms): 25123.37
---------------Time to First Token----------------
Mean TTFT (ms): 355.82
Median TTFT (ms): 268.84
P99 TTFT (ms): 858.64
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 46.62
Median TPOT (ms): 49.04
P99 TPOT (ms): 58.88
---------------Inter-Token Latency----------------
Mean ITL (ms): 46.36
Median ITL (ms): 32.46
P95 ITL (ms): 135.23
P99 ITL (ms): 204.27
Max ITL (ms): 508.14
==================================================
```
#### 5.1.5 Understanding the Results
**Key Metrics:**
- **Request Throughput (req/s)**: Number of requests processed per second
- **Output Token Throughput (tok/s)**: Total tokens generated per second
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
**Why These Configurations Matter:**
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
- **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
**Interpreting Results:**
- Compare your results against baseline numbers for your hardware
- Higher throughput at same latency = better performance
- Lower TTFT = more responsive user experience
- Lower TPOT = faster generation speed
### 5.2 Accuracy Benchmark
Document model accuracy on standard benchmarks:
#### 5.2.1 GSM8K Benchmark
- Benchmark Command
```bash Command
python -m sglang.test.few_shot_gsm8k \
--num-questions 200 \
--port 30000
```
- Test Result
```text Output
Accuracy: 0.975
Invalid: 0.000
Latency: 16.574 s
Output throughput: 1194.637 token/s
```
docs_new/cookbook/autoregressive/GLM/GLM-4.6V.mdx
+382
View File
@@ -0,0 +1,382 @@
---
title: GLM-4.6V
metatags:
description: "Deploy GLM-4.6V vision-language model with SGLang - native function calling, 128K context, multimodal document understanding and frontend replication."
---
## 1. Model Introduction
GLM-4.6V series model includes two versions: GLM-4.6V (106B), a foundation model designed for cloud and high-performance cluster scenarios, and GLM-4.6V-Flash (9B), a lightweight model optimized for local deployment and low-latency applications. GLM-4.6V scales its context window to 128k tokens in training, and achieves SoTA performance in visual understanding among models of similar parameter scales. Crucially, GLM team integrated native Function Calling capabilities for the first time. This effectively bridges the gap between "visual perception" and "executable action" providing a unified technical foundation for multimodal agents in real-world business scenarios.
Beyond achieves SoTA performance across major multimodal benchmarks at comparable model scales. GLM-4.6V introduces several key features:
- **Native Multimodal Function Calling** Enables native vision-driven tool use. Images, screenshots, and document pages can be passed directly as tool inputs without text conversion, while visual outputs (charts, search images, rendered pages) are interpreted and integrated into the reasoning chain. This closes the loop from perception to understanding to execution. Please refer to this [example](#tool-call-example).
- **Interleaved Image-Text Content Generation** Supports high-quality mixed media creation from complex multimodal inputs. GLM-4.6V takes a multimodal context—spanning documents, user inputs, and tool-retrieved images—and synthesizes coherent, interleaved image-text content tailored to the task. During generation it can actively call search and retrieval tools to gather and curate additional text and visuals, producing rich, visually grounded content.
- **Multimodal Document Understanding** GLM-4.6V can process up to 128K tokens of multi-document or long-document input, directly interpreting richly formatted pages as images. It understands text, layout, charts, tables, and figures jointly, enabling accurate comprehension of complex, image-heavy documents without requiring prior conversion to plain text.
- **Frontend Replication & Visual Editing** Reconstructs pixel-accurate HTML/CSS from UI screenshots and supports natural-language-driven edits. It detects layout, components, and styles visually, generates clean code, and applies iterative visual modifications through simple user instructions.
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
### 2.1 Docker Installation (Recommended)
```shell Command
docker pull lmsysorg/sglang:latest
```
**Advantages:**
- Ready to use out of the box, no manual environment configuration needed
- Avoids dependency conflict issues
- Easy to migrate between different environments
### 2.2 Build from Source
If you need to use the latest development version or require custom modifications, you can build from source:
```bash Command
# Install SGLang using UV (recommended)
git clone https://github.com/sgl-project/sglang.git
cd sglang
uv venv
source .venv/bin/activate
uv pip install -e "python[all]" --index-url=https://pypi.org/simple
pip install nvidia-cudnn-cu12==9.16.0.29
# Install ffmpeg to support video input
sudo apt update
sudo apt install ffmpeg
```
**Use Cases:**
- Need to customize and modify SGLang source code
- Want to use the latest development features
- Participate in SGLang project development
For general installation instructions, you can also refer to the [official SGLang installation guide](../../../docs/get-started/installation).
## 3. Model Deployment
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the interactive configuration generator below to customize your deployment settings. Select your hardware platform, model size, quantization method, and other options to generate the appropriate launch command.
import { GLM46VDeployment } from "/src/snippets/autoregressive/glm-46v-deployment.jsx";
<GLM46VDeployment />
### 3.2 Configuration Tips
- **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size * number of images in current running requests.)
- **TP=8 Configuration**: When using Tensor Parallelism (TP) of 8, the vision attention's 12 heads cannot be evenly divided. You can resolve this by adding `--mm-enable-dp-encoder` (which the generator above handles automatically).
- **Fast Model Loading**: For large models (like the 106B version), you can speed up model loading by using `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'`.
- For more detailed configuration tips, please refer to [GLM-4.5V/GLM-4.6V Usage](../../../docs/basic_usage/glmv).
## 4. Example APIs
### Image Input Example
#### API Payload
```python Example
curl_command = f"""
curl -s http://localhost:{30000}/v1/chat/completions \\
-H "Content-Type: application/json" \\
-d '{{
"model": "default",
"messages": [
{{
"role": "user",
"content": [
{{
"type": "image_url",
"image_url": {{
"url": "/home/jobuser/sgl_logo.png"
}}
}},
{{
"type": "text",
"text": "What is the image"
}}
]
}}
],
"temperature": "0",
"max_completion_tokens": "1000",
"max_tokens": "1000"
}}'
"""
response = subprocess.check_output(curl_command, shell=True).decode()
print(response)
```
#### API Response
```text Output
{"id":"b61596ca71394dd699fd8abd4f650c44","object":"chat.completion","created":1765259019,"model":"default","choices":[{"index":0,"message":{"role":"assistant","content":"The image is a logo featuring the text \"SGL\" (in a bold, orange-brown font) alongside a stylized icon. The icon includes a network-like structure with circular nodes (suggesting connectivity or a tree/graph structure) and a tag with \"</>\" (a common symbol for coding, web development, or software). The color scheme uses warm orange-brown tones with a black background, giving it a tech-focused, modern aesthetic (likely representing a company, project, or tool related to software, web development, or digital technology).<|begin_of_box|>SGL logo (stylized text + network/coding icon)<|end_of_box|>","reasoning_content":"Okay, let's see. The image has a logo with the text \"SGL\" and a little icon on the left. The icon looks like a network or a tree structure with circles, and there's a tag with \"</>\" which is a common symbol for coding or web development. The colors are orange and brown tones, with a black background. So probably a logo for a company or project named SGL, maybe related to software, web development, or a tech company.","tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":151336}],"usage":{"prompt_tokens":2222,"total_tokens":2448,"completion_tokens":226,"prompt_tokens_details":null,"reasoning_tokens":0},"metadata":{"weight_version":"default"}}
```
### Video Input Example
#### API Payload
```python Example
curl_command = f"""
curl -s http://localhost:{30000}/v1/chat/completions \\
-H "Content-Type: application/json" \\
-d '{{
"model": "default",
"messages": [
{{
"role": "user",
"content": [
{{
"type": "video_url",
"video_url": {{
"url": "/home/jobuser/jobs_presenting_ipod.mp4"
}}
}},
{{
"type": "text",
"text": "What is the image"
}}
]
}}
],
"temperature": "0",
"max_completion_tokens": "1000",
"max_tokens": "1000"
}}'
"""
response = subprocess.check_output(curl_command, shell=True).decode()
print(response)
```
#### API Response
```text Output
{"id":"520e0a079e5d4b17b82a6af619315a97","object":"chat.completion","created":1765259029,"model":"default","choices":[{"index":0,"message":{"role":"assistant","content":"The image is a still from a presentation by a man on a stage. He is pointing to a small pocket on his jeans and asking the audience what the pocket is for. The video is being shared by Evan Carmichael. The man then reveals that the pocket is for an iPod Nano.","reasoning_content":"Based on the visual evidence in the video, here is a breakdown of what is being shown:\n\n* **Subject:** The video features a man on a stage, giving a presentation. He is wearing a black t-shirt and dark jeans.\n* **Action:** The man is pointing to a pocket on his jeans. He is asking the audience a question about the purpose of this pocket.\n* **Context:** The presentation is being filmed, and the video is being shared by \"Evan Carmichael,\" a well-known motivational speaker and content creator. The source of the clip is credited to \"JoshuaG.\"\n* **Reveal:** The man then reveals the answer to his question. He pulls a small, white, rectangular device out of the pocket. He identifies this device as an \"iPod Nano.\"\n\nIn summary, the image is a still from a presentation where a speaker is explaining the purpose of the small pocket found on many pairs of jeans.","tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":151336}],"usage":{"prompt_tokens":30276,"total_tokens":30532,"completion_tokens":256,"prompt_tokens_details":null,"reasoning_tokens":0},"metadata":{"weight_version":"default"}}
```
### Tool Call Example
#### API Payload
```python Example
from openai import OpenAI
import argparse
import sys
import base64
def image_to_base64(image_path):
"""Convert image file to base64 data URL format for OpenAI API"""
with open(image_path, 'rb') as image_file:
image_data = image_file.read()
base64_string = base64.b64encode(image_data).decode('utf-8')
return f"data:image/png;base64,{base64_string}"
openai_api_key = "EMPTY"
openai_api_base = "http://127.0.0.1:30000/v1"
client = OpenAI(api_key=openai_api_key, base_url=openai_api_base)
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current temperature for a given location.",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "City and country e.g. Beijing, China",
}
},
"required": ["location"],
"additionalProperties": False,
},
},
}
]
messages = [
{
"role": "user",
"content": "Please help me check today’s weather in Beijing, and tell me whether the tool returned an image."
},
{
"role": "assistant",
"tool_calls": [
{
"id": "call_bk32t88BGpSdbtDgzT044Rh4",
"type": "function",
"function": {
"name": 'get_weather',
"arguments": '{"location":"Beijing, China"}'
}
}
]
},
{
"role": "tool",
"tool_call_id": "call_bk32t88BGpSdbtDgzT044Rh4",
"content": [
{
"type": "text",
"text": "Weather report generated: Beijing, November 7, 2025, sunny, temperature 2°C."
},
{
"type": "image_url",
"image_url": {
"url": "/home/jobuser/sgl_logo.png"
}
}
]
},
]
response = client.chat.completions.create(
model="zai-org/GLM-4.6V",
messages=messages,
timeout=900,
tools=tools
)
print(response.choices[0].message.content.strip())
```
#### Output
```text Output
The weather in Beijing today (November 7, 2025) is sunny with a temperature of 2°C.
Yes, the tool returned an image (the SGL logo).
```
## 5. Benchmark
### 5.1. Text Benchmark: Latency, Throughput and Accuracy
#### Command
```shell Command
python3 ./benchmark/gsm8k/bench_sglang.py
```
#### Result Output
```text Output
Accuracy: 0.925
Invalid: 0.000
Latency: 15.327 s
Output throughput: 1788.375 token/s
```
### 5.2. Multimodal Benchmark - Latency and Throughput
#### Command
```shell Command
python3 -m sglang.bench_serving \
--backend sglang-oai-chat \
--port 30000 \
--model zai-org/GLM-4.6V \
--dataset-name image \
--image-count 2 \
--image-resolution 720p \
--random-input-len 128 \
--random-output-len 1024 \
--num-prompts 128 \
--max-concurrency 8
```
#### Result Output
```text Output
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Traffic request rate: inf
Max request concurrency: 8
Successful requests: 128
Benchmark duration (s): 89.27
Total input tokens: 315390
Total input text tokens: 8702
Total input vision tokens: 306688
Total generated tokens: 66020
Total generated tokens (retokenized): 31037
Request throughput (req/s): 1.43
Input token throughput (tok/s): 3533.17
Output token throughput (tok/s): 739.59
Peak output token throughput (tok/s): 823.00
Peak concurrent requests: 12
Total token throughput (tok/s): 4272.76
Concurrency: 7.67
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 5349.20
Median E2E Latency (ms): 5380.98
---------------Time to First Token----------------
Mean TTFT (ms): 1724.04
Median TTFT (ms): 1688.16
P99 TTFT (ms): 6152.34
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 8.15
Median TPOT (ms): 7.77
P99 TPOT (ms): 23.97
---------------Inter-Token Latency----------------
Mean ITL (ms): 10.00
Median ITL (ms): 8.44
P95 ITL (ms): 9.23
P99 ITL (ms): 116.02
Max ITL (ms): 173.48
==================================================
```
### 5.3. Multimodal Accuracy Benchmark - MMMU
#### Command
```shell Command
python3 benchmark/mmmu/bench_sglang.py --response-answer-regex "<\|begin_of_box\|>(.*)<\|end_of_box\|>" --port 30000 --concurrency 64 --extra-request-body '{"max_tokens": 4096}'
```
#### Result Output
```text Output
Benchmark time: 487.2229107860476
answers saved to: ./answer_sglang.json
Evaluating...
answers saved to: ./answer_sglang.json
{'Accounting': {'acc': 0.962, 'num': 26},
'Agriculture': {'acc': 0.5, 'num': 30},
'Architecture_and_Engineering': {'acc': 0.733, 'num': 15},
'Art': {'acc': 0.833, 'num': 30},
'Art_Theory': {'acc': 0.9, 'num': 30},
'Basic_Medical_Science': {'acc': 0.733, 'num': 30},
'Biology': {'acc': 0.586, 'num': 29},
'Chemistry': {'acc': 0.654, 'num': 26},
'Clinical_Medicine': {'acc': 0.633, 'num': 30},
'Computer_Science': {'acc': 0.76, 'num': 25},
'Design': {'acc': 0.867, 'num': 30},
'Diagnostics_and_Laboratory_Medicine': {'acc': 0.633, 'num': 30},
'Economics': {'acc': 0.862, 'num': 29},
'Electronics': {'acc': 0.5, 'num': 18},
'Energy_and_Power': {'acc': 0.875, 'num': 16},
'Finance': {'acc': 0.857, 'num': 28},
'Geography': {'acc': 0.714, 'num': 28},
'History': {'acc': 0.767, 'num': 30},
'Literature': {'acc': 0.897, 'num': 29},
'Manage': {'acc': 0.759, 'num': 29},
'Marketing': {'acc': 1.0, 'num': 26},
'Materials': {'acc': 0.833, 'num': 18},
'Math': {'acc': 0.76, 'num': 25},
'Mechanical_Engineering': {'acc': 0.619, 'num': 21},
'Music': {'acc': 0.286, 'num': 28},
'Overall': {'acc': 0.761, 'num': 803},
'Overall-Art and Design': {'acc': 0.729, 'num': 118},
'Overall-Business': {'acc': 0.884, 'num': 138},
'Overall-Health and Medicine': {'acc': 0.773, 'num': 150},
'Overall-Humanities and Social Science': {'acc': 0.78, 'num': 118},
'Overall-Science': {'acc': 0.728, 'num': 136},
'Overall-Tech and Engineering': {'acc': 0.671, 'num': 143},
'Pharmacy': {'acc': 0.933, 'num': 30},
'Physics': {'acc': 0.929, 'num': 28},
'Psychology': {'acc': 0.733, 'num': 30},
'Public_Health': {'acc': 0.933, 'num': 30},
'Sociology': {'acc': 0.724, 'num': 29}}
eval out saved to ./val_sglang.json
Overall accuracy: 0.761
```
docs_new/cookbook/autoregressive/GLM/GLM-4.7-Flash.mdx
+931
View File
@@ -0,0 +1,931 @@
---
title: GLM-4.7-Flash
metatags:
description: "Deploy GLM-4.7-Flash 30B-A3B MoE model with SGLang - lightweight, efficient inference optimized for single-GPU deployment."
---
## 1. Model Introduction
[GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash) is a lightweight and high-speed model in the GLM-4.7 series developed by Zhipu AI, featuring state-of-the-art capabilities in reasoning, function calling, and efficient local deployment.
As a compact variant in the GLM-4.7 family, GLM-4.7-Flash is a **30B-A3B MoE** model designed to balance performance and efficiency:
- **Lightweight Architecture**: 30B total parameters with only 3B active parameters, enabling efficient inference
- **Enhanced Reasoning**: Inherits the reasoning capabilities from GLM-4.7 with optimized performance
- **Superior Coding**: Strong code generation and understanding capabilities
- **Advanced Tool Use**: Robust tool calling and agent capabilities for complex workflows
- **Optimized for Local Deployment**: Designed for single-GPU deployment scenarios
For more details, please refer to the [official GLM-4.7 documentation](https://docs.z.ai/guides/llm/glm-4.7).
**Key Features:**
- **Efficient MoE Architecture**: 30B-A3B sparse activation for optimal performance/efficiency trade-off
- **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
- **Hardware Optimization**: Specifically tuned for NVIDIA H100/H200/B200 GPUs
- **High Performance**: Optimized for both throughput and latency scenarios
**Available Models:**
- **BF16 (Full precision)**: [zai-org/GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash)
**License:**
Please refer to the [official GLM-4.7-Flash model card](https://huggingface.co/zai-org/GLM-4.7-Flash) for license details.
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.
import { GLM47FlashDeployment } from "/src/snippets/autoregressive/glm-47-flash-deployment.jsx";
<GLM47FlashDeployment />
### 3.2 Configuration Tips
For more detailed configuration tips, please refer to [GLM-4.7 Usage](../../../docs/basic_usage/glm45).
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
GLM-4.7-Flash supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:
```shell Command
python -m sglang.launch_server \
--model zai-org/GLM-4.7-Flash \
--reasoning-parser glm45 \
--attention-backend triton \
--tp 1 \
--host 0.0.0.0 \
--port 8000
```
**Streaming with Thinking Process:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="zai-org/GLM-4.7-Flash",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
To solve this problem, I need to calculate 15% of 240.
Step 1: Convert 15% to decimal: 15% = 0.15
Step 2: Multiply 240 by 0.15
Step 3: 240 × 0.15 = 36
=============== Content =================
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
```
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
#### 4.2.2 Tool Calling
GLM-4.7-Flash supports tool calling capabilities. Enable the tool call parser:
```shell Command
python -m sglang.launch_server \
--model zai-org/GLM-4.7-Flash \
--reasoning-parser glm45 \
--tool-call-parser glm47 \
--attention-backend triton \
--tp 1 \
--host 0.0.0.0 \
--port 8000
```
**Python Example (with Thinking Process):**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="zai-org/GLM-4.7-Flash",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Accumulate tool calls (tool call deltas may stream in multiple chunks)
if hasattr(delta, 'tool_calls') and delta.tool_calls:
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {
'name': None,
'arguments': ''
}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]['name'] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
# Print content
if delta.content:
print(delta.content, end="", flush=True)
# Print accumulated tool calls
if tool_calls_accumulator:
print("\n=============== Tool Calls =================", flush=True)
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
print()
```
**Output Example:**
```text Output
=============== Thinking =================
The user is asking for the weather in Beijing. I have the get_weather function available which can provide weather information for a location. The required parameter is "location" and the
user has provided "Beijing". There's an optional parameter "unit" for temperature unit, but the user hasn't specified which unit they prefer, and since it's optional, I should not ask about it or make up a value for it. I'll call the function with just the location parameter.I'll check the current weather in Beijing for you.
=============== Tool Calls =================
Tool Call: get_weather
Arguments: {"location": "Beijing"}
```
**Note:**
- The reasoning parser shows how the model decides to use a tool
- Tool calls are clearly marked with the function name and arguments
- You can then execute the function and send the result back to continue the conversation
**Handling Tool Call Results:**
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="zai-org/GLM-4.7-Flash",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```
## 5. Benchmark
This section uses **industry-standard configurations** for comparable benchmark results.
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: NVIDIA B200 (1x)
- Model: GLM-4.7-Flash
- Tensor Parallelism: 1
- SGLang Version: 0.5.7
**Benchmark Methodology:**
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
#### 5.1.1 Standard Test Scenarios
Three core scenarios reflect real-world usage patterns:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
</tr>
</tbody>
</table>
#### 5.1.2 Concurrency Levels
Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
#### 5.1.3 Number of Prompts
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
---
#### 5.1.4 Benchmark Commands
**Scenario 1: Chat (1K/1K) - Most Important**
- **Model Deployment**
```bash Command
python -m sglang.launch_server \
--model zai-org/GLM-4.7-Flash \
--attention-backend triton \
--tp 1
```
- Low Concurrency (Latency-Optimized)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.7-Flash \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 38.94
Total input tokens: 6101
Total input text tokens: 6101
Total generated tokens: 4220
Total generated tokens (retokenized): 4220
Request throughput (req/s): 0.26
Input token throughput (tok/s): 156.67
Output token throughput (tok/s): 108.37
Peak output token throughput (tok/s): 125.00
Peak concurrent requests: 2
Total token throughput (tok/s): 265.03
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 3891.12
Median E2E Latency (ms): 3061.48
P90 E2E Latency (ms): 7172.25
P99 E2E Latency (ms): 9042.62
---------------Time to First Token----------------
Mean TTFT (ms): 131.36
Median TTFT (ms): 94.55
P99 TTFT (ms): 435.93
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 8.75
Median TPOT (ms): 8.82
P99 TPOT (ms): 9.39
---------------Inter-Token Latency----------------
Mean ITL (ms): 8.93
Median ITL (ms): 8.98
P95 ITL (ms): 9.83
P99 ITL (ms): 10.20
Max ITL (ms): 18.50
==================================================
```
- Medium Concurrency (Balanced)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.7-Flash \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 52.73
Total input tokens: 39668
Total input text tokens: 39668
Total generated tokens: 40805
Total generated tokens (retokenized): 40775
Request throughput (req/s): 1.52
Input token throughput (tok/s): 752.27
Output token throughput (tok/s): 773.83
Peak output token throughput (tok/s): 1040.00
Peak concurrent requests: 21
Total token throughput (tok/s): 1526.10
Concurrency: 13.98
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 9217.90
Median E2E Latency (ms): 9642.50
P90 E2E Latency (ms): 15147.02
P99 E2E Latency (ms): 18237.06
---------------Time to First Token----------------
Mean TTFT (ms): 299.02
Median TTFT (ms): 105.98
P99 TTFT (ms): 1109.29
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 18.03
Median TPOT (ms): 18.00
P99 TPOT (ms): 26.51
---------------Inter-Token Latency----------------
Mean ITL (ms): 17.52
Median ITL (ms): 16.07
P95 ITL (ms): 18.14
P99 ITL (ms): 89.43
Max ITL (ms): 763.13
==================================================
```
- High Concurrency (Throughput-Optimized)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.7-Flash \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 91.48
Total input tokens: 249831
Total input text tokens: 249831
Total generated tokens: 252662
Total generated tokens (retokenized): 250941
Request throughput (req/s): 5.47
Input token throughput (tok/s): 2730.87
Output token throughput (tok/s): 2761.82
Peak output token throughput (tok/s): 4199.00
Peak concurrent requests: 109
Total token throughput (tok/s): 5492.69
Concurrency: 90.54
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 16566.04
Median E2E Latency (ms): 16134.36
P90 E2E Latency (ms): 30167.60
P99 E2E Latency (ms): 34034.04
---------------Time to First Token----------------
Mean TTFT (ms): 433.94
Median TTFT (ms): 123.26
P99 TTFT (ms): 1760.09
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 32.26
Median TPOT (ms): 33.56
P99 TPOT (ms): 38.78
---------------Inter-Token Latency----------------
Mean ITL (ms): 31.99
Median ITL (ms): 24.06
P95 ITL (ms): 79.62
P99 ITL (ms): 103.03
Max ITL (ms): 1369.20
==================================================
```
**Scenario 2: Reasoning (1K/8K)**
- Low Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.7-Flash \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 525.43
Total input tokens: 6101
Total input text tokens: 6101
Total generated tokens: 44462
Total generated tokens (retokenized): 44451
Request throughput (req/s): 0.02
Input token throughput (tok/s): 11.61
Output token throughput (tok/s): 84.62
Peak output token throughput (tok/s): 125.00
Peak concurrent requests: 2
Total token throughput (tok/s): 96.23
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 52540.19
Median E2E Latency (ms): 53694.45
P90 E2E Latency (ms): 94742.08
P99 E2E Latency (ms): 101224.18
---------------Time to First Token----------------
Mean TTFT (ms): 97.45
Median TTFT (ms): 95.28
P99 TTFT (ms): 105.64
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 10.94
Median TPOT (ms): 11.25
P99 TPOT (ms): 13.09
---------------Inter-Token Latency----------------
Mean ITL (ms): 11.80
Median ITL (ms): 11.51
P95 ITL (ms): 15.83
P99 ITL (ms): 16.86
Max ITL (ms): 19.96
==================================================
```
- Medium Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.7-Flash \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 473.92
Total input tokens: 39668
Total input text tokens: 39668
Total generated tokens: 318306
Total generated tokens (retokenized): 317860
Request throughput (req/s): 0.17
Input token throughput (tok/s): 83.70
Output token throughput (tok/s): 671.65
Peak output token throughput (tok/s): 1040.00
Peak concurrent requests: 19
Total token throughput (tok/s): 755.35
Concurrency: 13.80
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 81746.73
Median E2E Latency (ms): 78508.54
P90 E2E Latency (ms): 155292.49
P99 E2E Latency (ms): 166769.99
---------------Time to First Token----------------
Mean TTFT (ms): 117.50
Median TTFT (ms): 101.97
P99 TTFT (ms): 182.88
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 20.36
Median TPOT (ms): 20.48
P99 TPOT (ms): 22.63
---------------Inter-Token Latency----------------
Mean ITL (ms): 20.52
Median ITL (ms): 20.42
P95 ITL (ms): 23.41
P99 ITL (ms): 26.29
Max ITL (ms): 90.48
==================================================
```
- High Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.7-Flash \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 320 \
--max-concurrency 64 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 714.72
Total input tokens: 158939
Total input text tokens: 158939
Total generated tokens: 1301025
Total generated tokens (retokenized): 1289431
Request throughput (req/s): 0.45
Input token throughput (tok/s): 222.38
Output token throughput (tok/s): 1820.33
Peak output token throughput (tok/s): 3200.00
Peak concurrent requests: 68
Total token throughput (tok/s): 2042.71
Concurrency: 55.68
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 124364.58
Median E2E Latency (ms): 129250.98
P90 E2E Latency (ms): 219175.80
P99 E2E Latency (ms): 247741.77
---------------Time to First Token----------------
Mean TTFT (ms): 149.40
Median TTFT (ms): 114.78
P99 TTFT (ms): 288.60
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 30.51
Median TPOT (ms): 31.75
P99 TPOT (ms): 33.32
---------------Inter-Token Latency----------------
Mean ITL (ms): 30.56
Median ITL (ms): 30.82
P95 ITL (ms): 33.20
P99 ITL (ms): 80.54
Max ITL (ms): 117.72
==================================================
```
**Scenario 3: Summarization (8K/1K)**
- Low Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.7-Flash \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 58.27
Total input tokens: 41941
Total input text tokens: 41941
Total generated tokens: 4220
Total generated tokens (retokenized): 4220
Request throughput (req/s): 0.17
Input token throughput (tok/s): 719.73
Output token throughput (tok/s): 72.42
Peak output token throughput (tok/s): 112.00
Peak concurrent requests: 2
Total token throughput (tok/s): 792.15
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 5825.08
Median E2E Latency (ms): 4624.26
P90 E2E Latency (ms): 12690.22
P99 E2E Latency (ms): 13177.96
---------------Time to First Token----------------
Mean TTFT (ms): 296.01
Median TTFT (ms): 195.59
P99 TTFT (ms): 717.88
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 12.63
Median TPOT (ms): 13.07
P99 TPOT (ms): 16.68
---------------Inter-Token Latency----------------
Mean ITL (ms): 13.13
Median ITL (ms): 13.17
P95 ITL (ms): 17.02
P99 ITL (ms): 17.47
Max ITL (ms): 19.84
==================================================
```
- Medium Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.7-Flash \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 89.59
Total input tokens: 300020
Total input text tokens: 300020
Total generated tokens: 41669
Total generated tokens (retokenized): 41656
Request throughput (req/s): 0.89
Input token throughput (tok/s): 3348.77
Output token throughput (tok/s): 465.10
Peak output token throughput (tok/s): 752.00
Peak concurrent requests: 19
Total token throughput (tok/s): 3813.87
Concurrency: 14.39
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 16120.74
Median E2E Latency (ms): 16246.55
P90 E2E Latency (ms): 27279.72
P99 E2E Latency (ms): 34577.93
---------------Time to First Token----------------
Mean TTFT (ms): 1943.94
Median TTFT (ms): 382.19
P99 TTFT (ms): 8980.41
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 27.87
Median TPOT (ms): 28.26
P99 TPOT (ms): 40.55
---------------Inter-Token Latency----------------
Mean ITL (ms): 27.27
Median ITL (ms): 21.74
P95 ITL (ms): 23.32
P99 ITL (ms): 232.65
Max ITL (ms): 4282.01
==================================================
```
- High Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.7-Flash \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 320 \
--max-concurrency 64 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 167.01
Total input tokens: 1273893
Total input text tokens: 1273893
Total generated tokens: 170000
Total generated tokens (retokenized): 169226
Request throughput (req/s): 1.92
Input token throughput (tok/s): 7627.82
Output token throughput (tok/s): 1017.93
Peak output token throughput (tok/s): 1984.00
Peak concurrent requests: 69
Total token throughput (tok/s): 8645.75
Concurrency: 59.68
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 31147.52
Median E2E Latency (ms): 30603.34
P90 E2E Latency (ms): 54889.44
P99 E2E Latency (ms): 67665.30
---------------Time to First Token----------------
Mean TTFT (ms): 428.87
Median TTFT (ms): 441.69
P99 TTFT (ms): 1232.68
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 58.06
Median TPOT (ms): 62.79
P99 TPOT (ms): 82.23
---------------Inter-Token Latency----------------
Mean ITL (ms): 57.93
Median ITL (ms): 33.30
P95 ITL (ms): 247.98
P99 ITL (ms): 409.63
Max ITL (ms): 1421.21
==================================================
```
#### 5.1.5 Understanding the Results
**Key Metrics:**
- **Request Throughput (req/s)**: Number of requests processed per second
- **Output Token Throughput (tok/s)**: Total tokens generated per second
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
**Why These Configurations Matter:**
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
- **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
**Interpreting Results:**
- Compare your results against baseline numbers for your hardware
- Higher throughput at same latency = better performance
- Lower TTFT = more responsive user experience
- Lower TPOT = faster generation speed
### 5.2 Accuracy Benchmark
Document model accuracy on standard benchmarks:
#### 5.2.1 GSM8K Benchmark
- Benchmark Command
```bash Command
python -m sglang.test.few_shot_gsm8k \
--num-questions 200 \
--port 30000
```
- Result
```text Output
Accuracy: 0.845
Invalid: 0.000
Latency: 8.431 s
Output throughput: 2195.387 token/s
```
docs_new/cookbook/autoregressive/GLM/GLM-4.7.mdx
+546
View File
@@ -0,0 +1,546 @@
---
title: GLM-4.7
metatags:
description: "Deploy GLM-4.7 with SGLang on AMD GPUs - state-of-the-art reasoning, enhanced coding, and robust tool calling capabilities."
---
## 1. Model Introduction
[GLM-4.7](https://huggingface.co/zai-org/GLM-4.7) is the latest and most powerful language model in the GLM series developed by Zhipu AI, featuring state-of-the-art capabilities in reasoning, function calling, and multi-modal understanding.
As the newest iteration in the GLM series, GLM-4.7 achieves significant improvements across all domains:
- **Extended Context Window**: Expanded context window supporting even longer documents and complex multi-turn conversations
- **Enhanced Reasoning**: Improved reasoning capabilities with better chain-of-thought processing
- **Superior Coding**: Significantly improved code generation and understanding, with better real-world application performance
- **Advanced Tool Use**: More robust tool calling and agent capabilities for complex workflows
- **Optimized Performance**: Better throughput and latency characteristics across all hardware platforms
For more details, please refer to the [official GLM-4.7 documentation](https://docs.z.ai/guides/llm/glm-4.7).
**Key Features:**
- **State-of-the-Art Reasoning**: Enhanced reasoning capabilities for the most complex problem-solving tasks
- **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
- **Hardware Optimization**: Specifically tuned for AMD MI300X/MI325X/MI355X GPUs
- **High Performance**: Optimized for both throughput and latency scenarios
**Available Models:**
- **BF16 (Full precision)**: [zai-org/GLM-4.7](https://huggingface.co/zai-org/GLM-4.7) - Recommended for MI300X/MI325X/MI355X
- **FP8 (8-bit quantized)**: [zai-org/GLM-4.7-FP8](https://huggingface.co/zai-org/GLM-4.7-FP8) - Recommended for MI300X/MI325X/MI355X
**License:**
Please refer to the [official GLM-4.7 model card](https://huggingface.co/zai-org/GLM-4.7) for license details.
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.
import { GLM47Deployment } from "/src/snippets/autoregressive/glm-47-deployment.jsx";
<GLM47Deployment />
### 3.2 Configuration Tips
For more detailed configuration tips, please refer to [GLM-4.7 Usage](../../../docs/basic_usage/glm45).
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
GLM-4.7 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:
```shell Command
python -m sglang.launch_server \
--model zai-org/GLM-4.7 \
--reasoning-parser glm47 \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
**Streaming with Thinking Process:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="zai-org/GLM-4.7",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
To solve this problem, I need to calculate 15% of 240.
Step 1: Convert 15% to decimal: 15% = 0.15
Step 2: Multiply 240 by 0.15
Step 3: 240 × 0.15 = 36
=============== Content =================
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
```
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
#### 4.2.2 Tool Calling
GLM-4.7 supports tool calling capabilities. Enable the tool call parser:
```shell Command
python -m sglang.launch_server \
--model zai-org/GLM-4.7 \
--reasoning-parser glm47 \
--tool-call-parser glm47 \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
**Python Example (with Thinking Process):**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="zai-org/GLM-4.7",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
if tool_call.function:
print(f"Tool Call: {tool_call.function.name}")
print(f" Arguments: {tool_call.function.arguments}")
# Print content
if delta.content:
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
I should call the function with location="Beijing".
=============== Content =================
Tool Call: get_weather
Arguments: {"location": "Beijing", "unit": "celsius"}
```
**Note:**
- The reasoning parser shows how the model decides to use a tool
- Tool calls are clearly marked with the function name and arguments
- You can then execute the function and send the result back to continue the conversation
**Handling Tool Call Results:**
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="zai-org/GLM-4.7",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```
## 5. Benchmark
This section uses **industry-standard configurations** for comparable benchmark results.
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: AMD MI300X (8x), AMD MI325X (8x), AMD MI355X (8x)
- Model: GLM-4.7
- Tensor Parallelism: 8
- SGLang Version: 0.5.6.post1
**Benchmark Methodology:**
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
#### 5.1.1 Standard Test Scenarios
Three core scenarios reflect real-world usage patterns:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
</tr>
</tbody>
</table>
#### 5.1.2 Concurrency Levels
Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
#### 5.1.3 Number of Prompts
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
---
#### 5.1.4 Benchmark Commands
**Scenario 1: Chat (1K/1K) - Most Important**
- **Model Deployment**
```bash Command
python -m sglang.launch_server \
--model zai-org/GLM-4.7 \
--tp 8
```
- Low Concurrency (Latency-Optimized)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.7 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
- Medium Concurrency (Balanced)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.7 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
- High Concurrency (Throughput-Optimized)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.7 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100 \
--request-rate inf
```
**Scenario 2: Reasoning (1K/8K)**
- Low Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.7 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
- Medium Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.7 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
- High Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.7 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 320 \
--max-concurrency 64 \
--request-rate inf
```
**Scenario 3: Summarization (8K/1K)**
- Low Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.7 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
- Medium Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.7 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
- High Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-4.7 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 320 \
--max-concurrency 64 \
--request-rate inf
```
#### 5.1.5 Understanding the Results
**Key Metrics:**
- **Request Throughput (req/s)**: Number of requests processed per second
- **Output Token Throughput (tok/s)**: Total tokens generated per second
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
**Why These Configurations Matter:**
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
- **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
**Interpreting Results:**
- Compare your results against baseline numbers for your hardware
- Higher throughput at same latency = better performance
- Lower TTFT = more responsive user experience
- Lower TPOT = faster generation speed
### 5.2 Accuracy Benchmark
Document model accuracy on standard benchmarks:
#### 5.2.1 GSM8K Benchmark
- Benchmark Command
```bash Command
python -m sglang.test.few_shot_gsm8k \
--num-questions 200 \
--port 30000
```
docs_new/cookbook/autoregressive/GLM/GLM-5.1.mdx
+641
View File
@@ -0,0 +1,641 @@
---
title: GLM-5.1
metatags:
description: "Deploy GLM-5.1 with SGLang on NVIDIA H100/H200/B200/GB300 and AMD MI300X/MI325X/MI355X."
tag: NEW
---
## 1. Model Introduction
**Available Models:**
- **BF16 (Full precision)**: [zai-org/GLM-5.1](https://huggingface.co/zai-org/GLM-5.1)
- **FP8 (8-bit quantized)**: [zai-org/GLM-5.1-FP8](https://huggingface.co/zai-org/GLM-5.1-FP8)
**License:** MIT
## 2. SGLang Installation
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and capabilities. SGLang supports serving GLM-5.1 on NVIDIA H100, H200, B200, GB300, and AMD MI300X/MI325X/MI355X GPUs.
import { GLM51Deployment } from '/src/snippets/autoregressive/glm-51-deployment.jsx'
<GLM51Deployment />
### 3.2 Configuration Tips
- Speculative decoding (MTP) can significantly reduce latency for interactive use cases.
- **DP Attention**: Enables data parallel attention for higher throughput under high concurrency. Note that DP attention trades off low-concurrency latency for high-concurrency throughput — disable it if your workload is latency-sensitive with few concurrent requests.
- The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload.
- BF16 model always requires **2x GPUs** compared to FP8 on NVIDIA hardware.
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Hardware</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>FP8</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>BF16</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H100</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=16</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=32</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H200</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=16</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>B200</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=16</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GB300</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=4</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MI300X/MI325X</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MI355X</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
</tr>
</tbody>
</table>
- **AMD GPUs**: Both BF16 and FP8 checkpoints are supported on MI300X/MI325X/MI355X at tp=8. Use `--nsa-prefill-backend tilelang --nsa-decode-backend tilelang` for the NSA attention backend. Add `--chunked-prefill-size 131072` and `--watchdog-timeout 1200` (20 minutes for weight loading). FP8 uses approximately half the memory of BF16 (~89 GB/GPU vs ~175 GB/GPU). EAGLE speculative decoding is not currently supported on AMD for GLM-5.1.
- **GB300**: Only the FP8 checkpoint is recommended on GB300, with `tp=4`. For high-throughput DP attention on GB300, use `--dp 4`.
- For other configuration tips, please refer to [DeepSeek V3.2 documentation](../../../docs/basic_usage/deepseek_v32). GLM-5.1 and DeepSeek V3.2 share the same model structure, so the optimization techniques between these two models are also common (MTP, DSA kernel, Context Parallel...).
- Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` for GLM-5.1-FP8 if you want to enable the [IndexCache](https://github.com/THUDM/IndexCache) method. This feature is supported through [this PR](https://github.com/sgl-project/sglang/pull/21405) and introduces only a small accuracy loss. However, if you are running rigorous accuracy evaluations, it is not recommended to enable this feature.
## 4. Model Invocation
Deploy GLM-5.1 with the following command (FP8 on H200, all features enabled):
```shell Command
SGLANG_ENABLE_SPEC_V2=1 sglang serve \
--model-path zai-org/GLM-5.1-FP8 \
--tp 8 \
--tool-call-parser glm47 \
--reasoning-parser glm45 \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--mem-fraction-static 0.85 \
--host 0.0.0.0 \
--port 30000
```
### 4.1 MI300X/MI325X/MI355X (ROCm) Server Command
The following ROCm commands are additional options for AMD GPUs and do not replace the NVIDIA instructions above.
#### FP8 (Recommended)
```shell Command
sglang serve \
--model-path zai-org/GLM-5.1-FP8 \
--tp 8 \
--trust-remote-code \
--tool-call-parser glm47 \
--reasoning-parser glm45 \
--nsa-prefill-backend tilelang \
--nsa-decode-backend tilelang \
--chunked-prefill-size 131072 \
--mem-fraction-static 0.80 \
--watchdog-timeout 1200 \
--host 0.0.0.0 \
--port 30000
```
#### BF16
```shell Command
sglang serve \
--model-path zai-org/GLM-5.1 \
--tp 8 \
--trust-remote-code \
--nsa-prefill-backend tilelang \
--nsa-decode-backend tilelang \
--chunked-prefill-size 131072 \
--mem-fraction-static 0.80 \
--watchdog-timeout 1200 \
--host 0.0.0.0 \
--port 30000
```
### 4.2 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.3 Advanced Usage
#### 4.3.1 Reasoning Parser
GLM-5.1 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response.
To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time:
- **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed.
- **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process.
**Example 1: Thinking Mode (Default)**
Thinking mode is enabled by default. The model will reason step-by-step before answering, and the thinking process is returned via `reasoning_content`:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Thinking mode is enabled by default, no extra parameters needed
response = client.chat.completions.create(
model="zai-org/GLM-5.1-FP8",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
max_tokens=2048,
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
1. **Understand the Goal:** The user wants to find 15% of 240, and they want the solution explained step-by-step.
2. **Identify the Core Mathematical Concept:** "Percent" means "per hundred" or "out of 100". Finding "X% of Y" translates to the mathematical operation: $(X / 100) \times Y$.
3. **Step-by-Step Breakdown:**
* *Step 1: Convert the percentage to a decimal (or fraction).* 15% means 15 out of 100, which is $15/100$ or $0.15$.
* *Step 2: Multiply the decimal by the given number.* Multiply $0.15$ by $240$.
* *Step 3: Perform the calculation.*
* $0.15 \times 240$
* I can break this down further to make it easy to follow:
* $0.10 \times 240 = 24$ (which is 10%)
* $0.05 \times 240 = 12$ (which is 5%, half of 10%)
* $24 + 12 = 36$
* Alternatively, standard multiplication:
* $240 \times 15 = 3600$
* Move decimal two places left -> $36$
* *Step 4: State the final answer clearly.*
4. **Draft the Response (incorporating the steps clearly):**
* *Introduction:* State the problem clearly.
* *Step 1:* Explain how to convert 15% to a decimal.
* *Step 2:* Explain the multiplication step.
* *Step 3:* Show the actual math (I'll provide the standard multiplication and the "mental math" trick as it adds value).
* *Conclusion:* Give the final answer.
5. **Refine the Output (Self-Correction/Polishing during drafting):**
* *Drafting Step 1:* To find 15% of 240, first convert 15% into a decimal. Since percent means "per hundred," you divide 15 by 100. 15 ÷ 100 = 0.15.
* *Drafting Step 2:* Next, multiply this decimal by the number you are finding the percentage of (which is 240). So, calculate 0.15 × 240.
* *Drafting Step 3 (Standard way):* 0.15 × 240 = 36.
* *Adding the alternative mental math way:* It's often helpful to break it down into 10% and 5%.
* 10% of 240 = 24 (move the decimal point one place to the left)
* 5% is half of 10%, so half of 24 = 12
* Add them together: 24 + 12 = 36.
* *Final Answer:* 15% of 240 is 36.
6. **Final Review against User Prompt:** Does it solve the problem? Yes. Is it step-by-step? Yes. Is it clear? Yes. (Proceed to generate output).
=============== Content =================
Here is the step-by-step solution to find 15% of 240:
**Step 1: Convert the percentage to a decimal.**
To convert a percentage to a decimal, divide it by 100 (or simply move the decimal point two places to the left).
* 15% = 15 ÷ 100 = **0.15**
**Step 2: Multiply the decimal by the number.**
Now, multiply the decimal (0.15) by the number you are finding the percentage of (240).
* 0.15 × 240 = **36**
*(Alternative mental math method for Step 2)*:
If you don't want to multiply by 0.15 directly, you can break 15% down into 10% and 5%:
* **10% of 240** = 24 (just move the decimal point one place to the left)
* **5% of 240** = 12 (5% is half of 10%, so just divide 24 by 2)
* **Add them together**: 24 + 12 = **36**
**Answer:**
15% of 240 is **36**.
```
**Example 2: Instruct Mode (Thinking Off)**
To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Disable thinking mode via chat_template_kwargs
response = client.chat.completions.create(
model="zai-org/GLM-5.1-FP8",
messages=[
{"role": "user", "content": "What is 15% of 240?"}
],
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
max_tokens=2048,
stream=True
)
# In Instruct mode, the model responds directly without reasoning_content
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if delta.content:
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
15% of 240 is 36.
Here is how to calculate it:
1. Convert the percentage to a decimal: 15% = 0.15
2. Multiply the decimal by the number: 0.15 × 240 = 36
```
#### 4.3.2 Tool Calling
GLM-5.1 supports tool calling capabilities. Enable the tool call parser during deployment. Thinking mode is on by default; to disable it for tool calling requests, pass `extra_body={"chat_template_kwargs": {"enable_thinking": False}}`.
**Python Example (with Thinking Process):**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="zai-org/GLM-5.1-FP8",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
if tool_call.function:
print(f"Tool Call: {tool_call.function.name}")
print(f" Arguments: {tool_call.function.arguments}")
# Print content
if delta.content:
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
The user wants to know the weather in Beijing. I'll call the get_weather function with "Beijing" as the location.
=============== Content =================
Tool Call: get_weather
Arguments:
Tool Call: None
Arguments: {
Tool Call: None
Arguments: "location": "Be
Tool Call: None
Arguments: ijing"
Tool Call: None
Arguments: }
```
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: H200 (8x)
- Model: GLM-5.1-FP8
- Tensor Parallelism: 8
- SGLang Version: commit 947927bdb
#### 5.1.1 Latency Benchmark
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-5.1-FP8 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 35.78
Total input tokens: 6101
Total input text tokens: 6101
Total generated tokens: 4220
Total generated tokens (retokenized): 4213
Request throughput (req/s): 0.28
Input token throughput (tok/s): 170.54
Output token throughput (tok/s): 117.96
Peak output token throughput (tok/s): 148.00
Peak concurrent requests: 2
Total token throughput (tok/s): 288.50
Concurrency: 1.00
Accept length: 3.48
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 3576.31
Median E2E Latency (ms): 2935.97
P90 E2E Latency (ms): 5908.97
P99 E2E Latency (ms): 8588.08
---------------Time to First Token----------------
Mean TTFT (ms): 290.88
Median TTFT (ms): 282.34
P99 TTFT (ms): 332.27
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 7.54
Median TPOT (ms): 6.97
P99 TPOT (ms): 9.04
---------------Inter-Token Latency----------------
Mean ITL (ms): 7.80
Median ITL (ms): 6.81
P95 ITL (ms): 13.51
P99 ITL (ms): 26.99
Max ITL (ms): 29.50
==================================================
```
#### 5.1.2 Throughput Benchmark
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-5.1-FP8 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 1000 \
--max-concurrency 100 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 411.74
Total input tokens: 502493
Total input text tokens: 502493
Total generated tokens: 500251
Total generated tokens (retokenized): 499614
Request throughput (req/s): 2.43
Input token throughput (tok/s): 1220.41
Output token throughput (tok/s): 1214.97
Peak output token throughput (tok/s): 2648.00
Peak concurrent requests: 105
Total token throughput (tok/s): 2435.38
Concurrency: 96.30
Accept length: 3.50
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 39648.76
Median E2E Latency (ms): 39058.12
P90 E2E Latency (ms): 57009.82
P99 E2E Latency (ms): 68880.33
---------------Time to First Token----------------
Mean TTFT (ms): 20613.80
Median TTFT (ms): 21429.21
P99 TTFT (ms): 29543.17
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 38.73
Median TPOT (ms): 36.52
P99 TPOT (ms): 67.09
---------------Inter-Token Latency----------------
Mean ITL (ms): 38.13
Median ITL (ms): 16.57
P95 ITL (ms): 86.01
P99 ITL (ms): 164.88
Max ITL (ms): 1307.02
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 GSM8K Benchmark
- Benchmark Command
```bash Command
python3 benchmark/gsm8k/bench_sglang.py --port 30000
```
- Test Result
```text Output
Accuracy: 0.955
Invalid: 0.000
Latency: 32.470 s
Output throughput: 642.044 token/s
```
#### 5.2.2 MMLU Benchmark
- Benchmark Command
```bash Command
python3 benchmark/mmlu/bench_sglang.py --port 30000
```
- Test Result
```text Output
subject: abstract_algebra, #q:100, acc: 0.860
subject: anatomy, #q:135, acc: 0.874
subject: astronomy, #q:152, acc: 0.941
subject: business_ethics, #q:100, acc: 0.880
subject: clinical_knowledge, #q:265, acc: 0.932
subject: college_biology, #q:144, acc: 0.972
subject: college_chemistry, #q:100, acc: 0.640
subject: college_computer_science, #q:100, acc: 0.900
subject: college_mathematics, #q:100, acc: 0.810
subject: college_medicine, #q:173, acc: 0.873
subject: college_physics, #q:102, acc: 0.912
subject: computer_security, #q:100, acc: 0.880
subject: conceptual_physics, #q:235, acc: 0.928
subject: econometrics, #q:114, acc: 0.807
subject: electrical_engineering, #q:145, acc: 0.897
subject: elementary_mathematics, #q:378, acc: 0.937
subject: formal_logic, #q:126, acc: 0.778
subject: global_facts, #q:100, acc: 0.710
subject: high_school_biology, #q:310, acc: 0.961
subject: high_school_chemistry, #q:203, acc: 0.847
subject: high_school_computer_science, #q:100, acc: 0.960
subject: high_school_european_history, #q:165, acc: 0.891
subject: high_school_geography, #q:198, acc: 0.960
subject: high_school_government_and_politics, #q:193, acc: 0.984
subject: high_school_macroeconomics, #q:390, acc: 0.923
subject: high_school_mathematics, #q:270, acc: 0.696
subject: high_school_microeconomics, #q:238, acc: 0.962
subject: high_school_physics, #q:151, acc: 0.821
subject: high_school_psychology, #q:545, acc: 0.956
subject: high_school_statistics, #q:216, acc: 0.889
subject: high_school_us_history, #q:204, acc: 0.941
subject: high_school_world_history, #q:237, acc: 0.945
subject: human_aging, #q:223, acc: 0.857
subject: human_sexuality, #q:131, acc: 0.908
subject: international_law, #q:121, acc: 0.934
subject: jurisprudence, #q:108, acc: 0.907
subject: logical_fallacies, #q:163, acc: 0.933
subject: machine_learning, #q:112, acc: 0.830
subject: management, #q:103, acc: 0.942
subject: marketing, #q:234, acc: 0.940
subject: medical_genetics, #q:100, acc: 0.990
subject: miscellaneous, #q:783, acc: 0.959
subject: moral_disputes, #q:346, acc: 0.873
subject: moral_scenarios, #q:895, acc: 0.837
subject: nutrition, #q:306, acc: 0.922
subject: philosophy, #q:311, acc: 0.897
subject: prehistory, #q:324, acc: 0.929
subject: professional_accounting, #q:282, acc: 0.844
subject: professional_law, #q:1534, acc: 0.714
subject: professional_medicine, #q:272, acc: 0.941
subject: professional_psychology, #q:612, acc: 0.913
subject: public_relations, #q:110, acc: 0.791
subject: security_studies, #q:245, acc: 0.878
subject: sociology, #q:201, acc: 0.940
subject: us_foreign_policy, #q:100, acc: 0.920
subject: virology, #q:166, acc: 0.596
subject: world_religions, #q:171, acc: 0.936
Total latency: 165.275
Average accuracy: 0.877
```
### 5.3 AMD GPU Benchmarks
#### 5.3.1 GSM8K Benchmark (MI325/MI35x)
- MI325/MI35x Test (GLM-5.1 BF16, `tp=8`, TileLang NSA backends)
```bash Command
python3 benchmark/gsm8k/bench_sglang.py --num-questions 200
```
```text Output
Accuracy: 0.970
Invalid: 0.000
```
Results from [AMD nightly CI](https://github.com/sgl-project/sglang/actions/runs/22556197510/attempts/2#summary-65346783629). See also [sglang#18911](https://github.com/sgl-project/sglang/pull/18911).
docs_new/cookbook/autoregressive/GLM/GLM-5.mdx
+667
View File
@@ -0,0 +1,667 @@
---
title: GLM-5
metatags:
description: "Deploy GLM-5 with SGLang on NVIDIA H100/H200/B200 and AMD MI300X/MI325X/MI355X — state-of-the-art reasoning, enhanced coding, and robust tool calling capabilities."
tag: NEW
---
## 1. Model Introduction
[GLM-5](https://huggingface.co/zai-org/GLM-5) is the most powerful language model in the GLM series developed by Zhipu AI, targeting complex systems engineering and long-horizon agentic tasks. Scaling from GLM-4.5's 355B parameters (32B active) to 744B parameters (40B active), GLM-5 integrates DeepSeek Sparse Attention (DSA) to largely reduce deployment cost while preserving long-context capacity.
With advances in both pre-training (28.5T tokens) and post-training via [slime](https://github.com/THUDM/slime) (a novel asynchronous RL infrastructure), GLM-5 delivers significant improvements over GLM-4.7 and achieves best-in-class performance among open-source models on reasoning, coding, and agentic tasks.
**Key Features:**
- **Systems Engineering & Agentic Tasks**: Purpose-built for complex systems engineering and long-horizon agentic tasks
- **State-of-the-Art Performance**: Best-in-class among open-source models on reasoning (HLE, AIME, GPQA), coding (SWE-bench, Terminal-Bench), and agentic tasks (BrowseComp, Vending Bench 2)
- **DeepSeek Sparse Attention (DSA)**: Reduces deployment cost while preserving long-context capacity
- **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
- **Speculative Decoding**: EAGLE-based speculative decoding support for lower latency
**Available Models:**
- **BF16 (Full precision)**: [zai-org/GLM-5](https://huggingface.co/zai-org/GLM-5)
- **FP8 (8-bit quantized)**: [zai-org/GLM-5-FP8](https://huggingface.co/zai-org/GLM-5-FP8)
**License:** MIT
## 2. SGLang Installation
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and capabilities. SGLang supports serving GLM-5 on NVIDIA H100, H200, B200, and AMD MI300X/MI325X/MI355X GPUs.
import { GLM5Deployment } from '/src/snippets/autoregressive/glm-5-deployment.jsx'
<GLM5Deployment />
### 3.2 Configuration Tips
- Speculative decoding (MTP) can significantly reduce latency for interactive use cases.
- **DP Attention**: Enables data parallel attention for higher throughput under high concurrency. Note that DP attention trades off low-concurrency latency for high-concurrency throughput — disable it if your workload is latency-sensitive with few concurrent requests.
- The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload.
- BF16 model always requires **2x GPUs** compared to FP8 on NVIDIA hardware.
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Hardware</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>FP8</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>BF16</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H100</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=16</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=32</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H200</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=16</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>B200</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=16</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MI300X/MI325X</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MI355X</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
</tr>
</tbody>
</table>
- **B200 (FP8)**: Use `--ep 1 --attention-backend nsa --nsa-decode-backend trtllm --nsa-prefill-backend trtllm --moe-runner-backend flashinfer_trtllm --enable-flashinfer-allreduce-fusion` for optimized NSA and MoE backends on Blackwell. Also add `--quantization fp8` for FP8 weight quantization.
- **AMD GPUs**: Use `--nsa-prefill-backend tilelang --nsa-decode-backend tilelang` for the NSA attention backend. Add `--chunked-prefill-size 131072` and `--watchdog-timeout 1200` (20 minutes for weight loading). EAGLE speculative decoding is not currently supported on AMD for GLM-5.
- For other configuration tips, please refer to [DeepSeek V3.2 documentation](../../../docs/basic_usage/deepseek_v32). GLM-5 and DeepSeek V3.2 share the same model structure, so the optimization techniques between these two models are also common (MTP, DSA kernel, Context Parallel...).
- Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` for GLM-5-FP8 if you want to enable the [IndexCache](https://github.com/THUDM/IndexCache) method. This feature is supported through [this PR](https://github.com/sgl-project/sglang/pull/21405) and introduces only a small accuracy loss. However, if you are running rigorous accuracy evaluations, it is not recommended to enable this feature.
<Warning>
**FP8 KV Cache**: `--kv-cache-dtype fp8_e4m3` quantizes the KV cache to FP8 at runtime. Since these FP8 model checkpoints do not include pre-calibrated KV cache scaling factors, SGLang defaults to a scale of 1.0, which may cause noticeable accuracy degradation on reasoning-heavy tasks. It is not included in the generated commands above; add it manually only if memory constraints require the trade-off.
</Warning>
## 4. Model Invocation
Deploy GLM-5 with the following command (FP8 on H200, all features enabled):
```shell Command
sglang serve \
--model zai-org/GLM-5-FP8 \
--tp 8 \
--tool-call-parser glm47 \
--reasoning-parser glm45 \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--mem-fraction-static 0.85 \
--host 0.0.0.0 \
--port 30000
```
### 4.1 MI300X/MI325X/MI355X (ROCm) Server Command
The following ROCm command is an additional option for AMD GPUs and does not replace the NVIDIA instructions above.
```shell Command
sglang serve \
--model zai-org/GLM-5 \
--tp 8 \
--trust-remote-code \
--nsa-prefill-backend tilelang \
--nsa-decode-backend tilelang \
--chunked-prefill-size 131072 \
--mem-fraction-static 0.80 \
--watchdog-timeout 1200 \
--host 0.0.0.0 \
--port 30000
```
### 4.2 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.3 Advanced Usage
#### 4.3.1 Reasoning Parser
GLM-5 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response.
To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time:
- **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed.
- **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process.
**Example 1: Thinking Mode (Default)**
Thinking mode is enabled by default. The model will reason step-by-step before answering, and the thinking process is returned via `reasoning_content`:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Thinking mode is enabled by default, no extra parameters needed
response = client.chat.completions.create(
model="zai-org/GLM-5-FP8",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
max_tokens=2048,
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
The user wants me to solve a math problem: "What is 15% of 240?".
Step 1: Understand the problem. I need to calculate a percentage of a number.
Formula: Percentage × Number = Result.
Step 2: Convert the percentage to a decimal or fraction.
15% = 15/100 or 0.15.
Step 3: Perform the multiplication.
Method A: Decimal multiplication.
0.15 × 240.
Break it down:
10% of 240 = 24.
5% is half of 10%, so 12.
15% = 10% + 5% = 24 + 12 = 36.
Method B: Fraction multiplication.
15/100 × 240.
Simplify 240/100 = 2.4.
15 × 2.4.
10 × 2.4 = 24.
5 × 2.4 = 12.
24 + 12 = 36.
Method C: Direct multiplication.
240 × 0.15.
240 × 0.10 = 24.
240 × 0.05 = 12.
24 + 12 = 36.
Step 4: Final Verification.
Is 36 reasonable?
10% is 24. 20% is 48.
15% is halfway between 10% and 20%.
Halfway between 24 and 48 is 36.
The result is correct.
Step 5: Structure the final response. I will present the calculation clearly, perhaps showing the fractional or decimal method, or the mental math shortcut (10% + 5%).
=============== Content =================
Here is the step-by-step solution:
**Step 1: Convert the percentage to a decimal.**
To convert 15% to a decimal, divide by 100.
$$15\% = \frac{15}{100} = 0.15$$
**Step 2: Multiply the decimal by the number.**
Now, multiply 0.15 by 240.
$$0.15 \times 240$$
**Step 3: Perform the calculation.**
You can break this down to make it easier:
$$0.15 = 0.10 + 0.05$$
* First, find 10% of 240:
$$0.10 \times 240 = 24$$
* Next, find 5% (which is half of 10%):
$$\frac{24}{2} = 12$$
* Add the two results together:
$$24 + 12 = 36$$
**Answer:**
15% of 240 is **36**.
```
**Example 2: Instruct Mode (Thinking Off)**
To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Disable thinking mode via chat_template_kwargs
response = client.chat.completions.create(
model="zai-org/GLM-5-FP8",
messages=[
{"role": "user", "content": "What is 15% of 240?"}
],
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
max_tokens=2048,
stream=True
)
# In Instruct mode, the model responds directly without reasoning_content
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if delta.content:
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
To find **15% of 240**, follow these steps:
### Step 1: Convert the Percentage to a Decimal
First, convert the percentage to a decimal by dividing by 100.
\[
15\% = \frac{15}{100} = 0.15
\]
### Step 2: Multiply by the Number
Next, multiply the decimal by the number you want to find the percentage of.
\[
0.15 \times 240
\]
### Step 3: Perform the Multiplication
Calculate the multiplication:
\[
0.15 \times 240 = 36
\]
### Final Answer
\[
\boxed{36}
\]
```
#### 4.3.2 Tool Calling
GLM-5 supports tool calling capabilities. Enable the tool call parser during deployment. Thinking mode is on by default; to disable it for tool calling requests, pass `extra_body={"chat_template_kwargs": {"enable_thinking": False}}`.
**Python Example (with Thinking Process):**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="zai-org/GLM-5-FP8",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
if tool_call.function:
print(f"Tool Call: {tool_call.function.name}")
print(f" Arguments: {tool_call.function.arguments}")
# Print content
if delta.content:
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
The user is asking for the weather in Beijing. I have access to a get_weather function that can provide current weather information. Let me check what parameters are required:
- location: required, should be "Beijing"
- unit: optional (not in required array), can be "celsius" or "fahrenheit"
Since the user didn't specify a unit preference and it's optional, I should not ask about it or make up a value. I'll just call the function with the required location parameter.I'll get the current weather in Beijing for you.
=============== Content =================
Tool Call: get_weather
Arguments:
Tool Call: None
Arguments: {
Tool Call: None
Arguments: "location": "Be
Tool Call: None
Arguments: ijing"
Tool Call: None
Arguments: }
```
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: H200 (8x)
- Model: GLM-5-FP8
- Tensor Parallelism: 8
- SGLang Version: commit 947927bdb
#### 5.1.1 Latency Benchmark
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-5-FP8 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 35.78
Total input tokens: 6101
Total input text tokens: 6101
Total generated tokens: 4220
Total generated tokens (retokenized): 4213
Request throughput (req/s): 0.28
Input token throughput (tok/s): 170.54
Output token throughput (tok/s): 117.96
Peak output token throughput (tok/s): 148.00
Peak concurrent requests: 2
Total token throughput (tok/s): 288.50
Concurrency: 1.00
Accept length: 3.48
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 3576.31
Median E2E Latency (ms): 2935.97
P90 E2E Latency (ms): 5908.97
P99 E2E Latency (ms): 8588.08
---------------Time to First Token----------------
Mean TTFT (ms): 290.88
Median TTFT (ms): 282.34
P99 TTFT (ms): 332.27
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 7.54
Median TPOT (ms): 6.97
P99 TPOT (ms): 9.04
---------------Inter-Token Latency----------------
Mean ITL (ms): 7.80
Median ITL (ms): 6.81
P95 ITL (ms): 13.51
P99 ITL (ms): 26.99
Max ITL (ms): 29.50
==================================================
```
#### 5.1.2 Throughput Benchmark
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model zai-org/GLM-5-FP8 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 1000 \
--max-concurrency 100 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 411.74
Total input tokens: 502493
Total input text tokens: 502493
Total generated tokens: 500251
Total generated tokens (retokenized): 499614
Request throughput (req/s): 2.43
Input token throughput (tok/s): 1220.41
Output token throughput (tok/s): 1214.97
Peak output token throughput (tok/s): 2648.00
Peak concurrent requests: 105
Total token throughput (tok/s): 2435.38
Concurrency: 96.30
Accept length: 3.50
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 39648.76
Median E2E Latency (ms): 39058.12
P90 E2E Latency (ms): 57009.82
P99 E2E Latency (ms): 68880.33
---------------Time to First Token----------------
Mean TTFT (ms): 20613.80
Median TTFT (ms): 21429.21
P99 TTFT (ms): 29543.17
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 38.73
Median TPOT (ms): 36.52
P99 TPOT (ms): 67.09
---------------Inter-Token Latency----------------
Mean ITL (ms): 38.13
Median ITL (ms): 16.57
P95 ITL (ms): 86.01
P99 ITL (ms): 164.88
Max ITL (ms): 1307.02
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 GSM8K Benchmark
- Benchmark Command
```bash Command
python3 benchmark/gsm8k/bench_sglang.py --port 30000
```
- Test Result
```text Output
Accuracy: 0.955
Invalid: 0.000
Latency: 32.470 s
Output throughput: 642.044 token/s
```
#### 5.2.2 MMLU Benchmark
- Benchmark Command
```bash Command
python3 benchmark/mmlu/bench_sglang.py --port 30000
```
- Test Result
```text Output
subject: abstract_algebra, #q:100, acc: 0.860
subject: anatomy, #q:135, acc: 0.874
subject: astronomy, #q:152, acc: 0.941
subject: business_ethics, #q:100, acc: 0.880
subject: clinical_knowledge, #q:265, acc: 0.932
subject: college_biology, #q:144, acc: 0.972
subject: college_chemistry, #q:100, acc: 0.640
subject: college_computer_science, #q:100, acc: 0.900
subject: college_mathematics, #q:100, acc: 0.810
subject: college_medicine, #q:173, acc: 0.873
subject: college_physics, #q:102, acc: 0.912
subject: computer_security, #q:100, acc: 0.880
subject: conceptual_physics, #q:235, acc: 0.928
subject: econometrics, #q:114, acc: 0.807
subject: electrical_engineering, #q:145, acc: 0.897
subject: elementary_mathematics, #q:378, acc: 0.937
subject: formal_logic, #q:126, acc: 0.778
subject: global_facts, #q:100, acc: 0.710
subject: high_school_biology, #q:310, acc: 0.961
subject: high_school_chemistry, #q:203, acc: 0.847
subject: high_school_computer_science, #q:100, acc: 0.960
subject: high_school_european_history, #q:165, acc: 0.891
subject: high_school_geography, #q:198, acc: 0.960
subject: high_school_government_and_politics, #q:193, acc: 0.984
subject: high_school_macroeconomics, #q:390, acc: 0.923
subject: high_school_mathematics, #q:270, acc: 0.696
subject: high_school_microeconomics, #q:238, acc: 0.962
subject: high_school_physics, #q:151, acc: 0.821
subject: high_school_psychology, #q:545, acc: 0.956
subject: high_school_statistics, #q:216, acc: 0.889
subject: high_school_us_history, #q:204, acc: 0.941
subject: high_school_world_history, #q:237, acc: 0.945
subject: human_aging, #q:223, acc: 0.857
subject: human_sexuality, #q:131, acc: 0.908
subject: international_law, #q:121, acc: 0.934
subject: jurisprudence, #q:108, acc: 0.907
subject: logical_fallacies, #q:163, acc: 0.933
subject: machine_learning, #q:112, acc: 0.830
subject: management, #q:103, acc: 0.942
subject: marketing, #q:234, acc: 0.940
subject: medical_genetics, #q:100, acc: 0.990
subject: miscellaneous, #q:783, acc: 0.959
subject: moral_disputes, #q:346, acc: 0.873
subject: moral_scenarios, #q:895, acc: 0.837
subject: nutrition, #q:306, acc: 0.922
subject: philosophy, #q:311, acc: 0.897
subject: prehistory, #q:324, acc: 0.929
subject: professional_accounting, #q:282, acc: 0.844
subject: professional_law, #q:1534, acc: 0.714
subject: professional_medicine, #q:272, acc: 0.941
subject: professional_psychology, #q:612, acc: 0.913
subject: public_relations, #q:110, acc: 0.791
subject: security_studies, #q:245, acc: 0.878
subject: sociology, #q:201, acc: 0.940
subject: us_foreign_policy, #q:100, acc: 0.920
subject: virology, #q:166, acc: 0.596
subject: world_religions, #q:171, acc: 0.936
Total latency: 165.275
Average accuracy: 0.877
```
### 5.3 AMD GPU Benchmarks
#### 5.3.1 GSM8K Benchmark (MI325/MI35x)
- MI325/MI35x Test (GLM-5 BF16, `tp=8`, TileLang NSA backends)
```bash Command
python3 benchmark/gsm8k/bench_sglang.py --num-questions 200
```
```text Output
Accuracy: 0.970
Invalid: 0.000
```
Results from [AMD nightly CI](https://github.com/sgl-project/sglang/actions/runs/22556197510/attempts/2#summary-65346783629). See also [sglang#18911](https://github.com/sgl-project/sglang/pull/18911).
docs_new/cookbook/autoregressive/GLM/GLM-Glyph.mdx
+829
View File
@@ -0,0 +1,829 @@
---
title: GLM Glyph
metatags:
description: "Deploy GLM-Glyph with SGLang - community contribution guide for Zhipu AI's GLM Glyph model deployment."
---
import { GLMGlyphDeployment } from '/src/snippets/autoregressive/glm-glyph-deployment.jsx';
## 1. Model Introduction
[Glyph](https://huggingface.co/zai-org/Glyph) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding.
**Hardware Support:** NVIDIA B200/H100/H200, AMD MI300X/MI325X/MI355X
**Key Features:**
- **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving
- **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
- **High Performance**: Optimized for both throughput and latency scenarios
**Available Models:**
- **BF16 (Full precision)**: [zai-org/Glyph](https://huggingface.co/zai-org/Glyph)
- **FP8 (8-bit quantized)**: [zai-org/Glyph-FP8](https://huggingface.co/zai-org/Glyph-FP8)
**License:**
Please refer to the [official Glyph model card](https://huggingface.co/zai-org/Glyph) for license details.
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and other options.
<GLMGlyphDeployment />
### 3.2 Configuration Tips
For more detailed configuration tips, please refer to [GLM-4.5/GLM-4.6 Usage](../../../docs/basic_usage/glm45).
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Advanced Usage
#### 4.2.1 Thinking Mode
Glyph supports thinking mode for enhanced reasoning. Enable the reasoning parser during deployment to separate the thinking and content sections:
```shell Command
python -m sglang.launch_server \
--model-path zai-org/Glyph \
--reasoning-parser glm45 \
--tp 4
```
**Streaming with Thinking Process:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="zai-org/Glyph",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
**Disable Thinking Mode:**
To disable thinking mode for a specific request:
```python Example
response = client.chat.completions.create(
model="zai-org/Glyph",
messages=[{"role": "user", "content": "What is the capital of France?"}],
extra_body={"chat_template_kwargs": {"enable_thinking": False}}
)
```
#### 4.2.2 Tool Calling
Glyph supports tool calling capabilities. Enable the tool call parser:
```shell Command
python -m sglang.launch_server \
--model-path zai-org/Glyph \
--reasoning-parser glm45 \
--tool-call-parser glm45 \
--tp 4
```
**Python Example (with Thinking Process):**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="zai-org/Glyph",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Accumulate tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================\n", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {
'name': None,
'arguments': ''
}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]['name'] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
# Print content
if delta.content:
print(delta.content, end="", flush=True)
# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
print()
```
**Output Example:**
```text Output
=============== Thinking =================
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
I should call the function with location="Beijing".
=============== Content =================
Tool Call: get_weather
Arguments: {"location": "Beijing", "unit": "celsius"}
```
**Note:**
- The reasoning parser shows how the model decides to use a tool
- Tool calls are clearly marked with the function name and arguments
- You can then execute the function and send the result back to continue the conversation
**Handling Tool Call Results:**
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="zai-org/Glyph",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```
## 5. Benchmark
This section uses **industry-standard configurations** for comparable benchmark results.
### 5.1 Speed Benchmark
**Test Environment:**
- Model: Glyph
- SGLang Version: 0.5.6.post1
**Benchmark Methodology:**
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
#### 5.1.1 Standard Scenario Benchmark
- **Model Deployment**
```bash Command
python -m sglang.launch_server \
--model zai-org/Glyph \
--tp 2
```
##### 5.1.1.1 Low Concurrency
- **Benchmark Command**:
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/Glyph \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
- **Test Results**:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 17.03
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4220
Request throughput (req/s): 0.59
Input token throughput (tok/s): 358.17
Output token throughput (tok/s): 247.74
Peak output token throughput (tok/s): 251.00
Peak concurrent requests: 3
Total token throughput (tok/s): 605.91
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 1702.14
Median E2E Latency (ms): 1361.72
---------------Time to First Token----------------
Mean TTFT (ms): 22.35
Median TTFT (ms): 22.61
P99 TTFT (ms): 23.76
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 3.99
Median TPOT (ms): 3.99
P99 TPOT (ms): 4.01
---------------Inter-Token Latency----------------
Mean ITL (ms): 3.99
Median ITL (ms): 3.99
P95 ITL (ms): 4.03
P99 ITL (ms): 4.12
Max ITL (ms): 7.46
==================================================
```
##### 5.1.1.2 Medium Concurrency
- **Benchmark Command**:
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/Glyph \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
- **Test Results**:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 16.27
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 40805
Total generated tokens (retokenized): 40804
Request throughput (req/s): 4.92
Input token throughput (tok/s): 2438.06
Output token throughput (tok/s): 2507.94
Peak output token throughput (tok/s): 3069.00
Peak concurrent requests: 26
Total token throughput (tok/s): 4946.00
Concurrency: 13.44
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 2733.43
Median E2E Latency (ms): 2892.98
---------------Time to First Token----------------
Mean TTFT (ms): 33.10
Median TTFT (ms): 27.73
P99 TTFT (ms): 49.34
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 5.33
Median TPOT (ms): 5.39
P99 TPOT (ms): 5.86
---------------Inter-Token Latency----------------
Mean ITL (ms): 5.30
Median ITL (ms): 4.89
P95 ITL (ms): 5.54
P99 ITL (ms): 21.17
Max ITL (ms): 25.14
==================================================
```
##### 5.1.1.3 High Concurrency
- **Benchmark Command**:
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/Glyph \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100 \
--request-rate inf
```
- **Test Results**:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 25.67
Total input tokens: 249831
Total input text tokens: 249831
Total input vision tokens: 0
Total generated tokens: 252662
Total generated tokens (retokenized): 252657
Request throughput (req/s): 19.48
Input token throughput (tok/s): 9733.69
Output token throughput (tok/s): 9843.99
Peak output token throughput (tok/s): 13398.00
Peak concurrent requests: 127
Total token throughput (tok/s): 19577.68
Concurrency: 89.49
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 4593.75
Median E2E Latency (ms): 4431.03
---------------Time to First Token----------------
Mean TTFT (ms): 48.66
Median TTFT (ms): 35.88
P99 TTFT (ms): 120.61
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 9.10
Median TPOT (ms): 9.55
P99 TPOT (ms): 11.00
---------------Inter-Token Latency----------------
Mean ITL (ms): 9.01
Median ITL (ms): 6.51
P95 ITL (ms): 23.19
P99 ITL (ms): 25.54
Max ITL (ms): 52.93
==================================================
```
#### 5.1.2 Reasoning Scenario Benchmark
##### 5.1.2.1 Low Concurrency
- **Benchmark Command**:
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/Glyph \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
- **Test Results**:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 201.53
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 44462
Total generated tokens (retokenized): 44455
Request throughput (req/s): 0.05
Input token throughput (tok/s): 30.27
Output token throughput (tok/s): 220.63
Peak output token throughput (tok/s): 251.00
Peak concurrent requests: 2
Total token throughput (tok/s): 250.90
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 20151.45
Median E2E Latency (ms): 21576.31
---------------Time to First Token----------------
Mean TTFT (ms): 2362.23
Median TTFT (ms): 23.03
P99 TTFT (ms): 21310.14
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 4.00
Median TPOT (ms): 4.00
P99 TPOT (ms): 4.01
---------------Inter-Token Latency----------------
Mean ITL (ms): 4.00
Median ITL (ms): 4.00
P95 ITL (ms): 4.05
P99 ITL (ms): 4.08
Max ITL (ms): 5.67
==================================================
```
##### 5.1.2.2 Medium Concurrency
- **Benchmark Command**:
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/Glyph \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
- **Test Results**:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 118.67
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 318306
Total generated tokens (retokenized): 318270
Request throughput (req/s): 0.67
Input token throughput (tok/s): 334.27
Output token throughput (tok/s): 2682.26
Peak output token throughput (tok/s): 3264.00
Peak concurrent requests: 19
Total token throughput (tok/s): 3016.53
Concurrency: 13.74
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 20387.23
Median E2E Latency (ms): 20466.09
---------------Time to First Token----------------
Mean TTFT (ms): 132.47
Median TTFT (ms): 27.19
P99 TTFT (ms): 583.15
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 5.09
Median TPOT (ms): 5.13
P99 TPOT (ms): 5.19
---------------Inter-Token Latency----------------
Mean ITL (ms): 5.09
Median ITL (ms): 5.08
P95 ITL (ms): 5.18
P99 ITL (ms): 5.57
Max ITL (ms): 522.26
==================================================
```
##### 5.1.2.3 High Concurrency
- **Benchmark Command**:
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/Glyph \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 320 \
--max-concurrency 64 \
--request-rate inf
```
- **Test Results**:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 150.00
Total input tokens: 158939
Total input text tokens: 158939
Total input vision tokens: 0
Total generated tokens: 1301025
Total generated tokens (retokenized): 1300901
Request throughput (req/s): 2.13
Input token throughput (tok/s): 1059.59
Output token throughput (tok/s): 8673.49
Peak output token throughput (tok/s): 11899.00
Peak concurrent requests: 71
Total token throughput (tok/s): 9733.09
Concurrency: 54.71
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 25645.42
Median E2E Latency (ms): 26913.26
---------------Time to First Token----------------
Mean TTFT (ms): 163.75
Median TTFT (ms): 93.67
P99 TTFT (ms): 426.19
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 6.27
Median TPOT (ms): 6.39
P99 TPOT (ms): 6.59
---------------Inter-Token Latency----------------
Mean ITL (ms): 6.27
Median ITL (ms): 0.17
P95 ITL (ms): 32.94
P99 ITL (ms): 67.89
Max ITL (ms): 136.00
==================================================
```
#### 5.1.3 Summarization Scenario Benchmark
#### 5.1.3.1 Low Concurrency
- **Benchmark Command**:
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/Glyph \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
- **Test Results**:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 17.44
Total input tokens: 41941
Total input text tokens: 41941
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4220
Request throughput (req/s): 0.57
Input token throughput (tok/s): 2405.19
Output token throughput (tok/s): 242.00
Peak output token throughput (tok/s): 250.00
Peak concurrent requests: 2
Total token throughput (tok/s): 2647.19
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 1742.54
Median E2E Latency (ms): 1412.47
---------------Time to First Token----------------
Mean TTFT (ms): 53.48
Median TTFT (ms): 45.05
P99 TTFT (ms): 98.57
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 4.01
Median TPOT (ms): 4.01
P99 TPOT (ms): 4.03
---------------Inter-Token Latency----------------
Mean ITL (ms): 4.01
Median ITL (ms): 4.01
P95 ITL (ms): 4.06
P99 ITL (ms): 4.09
Max ITL (ms): 4.95
==================================================
```
##### 5.1.3.2 Medium Concurrency
- **Benchmark Command**:
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/Glyph \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
- **Test Results**:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 16.90
Total input tokens: 300020
Total input text tokens: 300020
Total input vision tokens: 0
Total generated tokens: 41669
Total generated tokens (retokenized): 41668
Request throughput (req/s): 4.73
Input token throughput (tok/s): 17753.58
Output token throughput (tok/s): 2465.75
Peak output token throughput (tok/s): 3005.00
Peak concurrent requests: 25
Total token throughput (tok/s): 20219.33
Concurrency: 13.68
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 2890.33
Median E2E Latency (ms): 3069.55
---------------Time to First Token----------------
Mean TTFT (ms): 41.46
Median TTFT (ms): 31.75
P99 TTFT (ms): 93.18
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 5.52
Median TPOT (ms): 5.58
P99 TPOT (ms): 6.14
---------------Inter-Token Latency----------------
Mean ITL (ms): 5.48
Median ITL (ms): 5.13
P95 ITL (ms): 5.93
P99 ITL (ms): 20.76
Max ITL (ms): 36.01
==================================================
```
##### 5.1.3.3 High Concurrency
- **Benchmark Command**:
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model zai-org/Glyph \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 320 \
--max-concurrency 64 \
--request-rate inf
```
- **Test Results**:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 35.54
Total input tokens: 1273893
Total input text tokens: 1273893
Total input vision tokens: 0
Total generated tokens: 170000
Total generated tokens (retokenized): 169994
Request throughput (req/s): 9.01
Input token throughput (tok/s): 35848.57
Output token throughput (tok/s): 4783.96
Peak output token throughput (tok/s): 8396.00
Peak concurrent requests: 80
Total token throughput (tok/s): 40632.53
Concurrency: 59.26
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 6580.96
Median E2E Latency (ms): 6248.74
---------------Time to First Token----------------
Mean TTFT (ms): 345.27
Median TTFT (ms): 96.06
P99 TTFT (ms): 2823.92
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 12.26
Median TPOT (ms): 12.53
P99 TPOT (ms): 23.58
---------------Inter-Token Latency----------------
Mean ITL (ms): 11.76
Median ITL (ms): 6.57
P95 ITL (ms): 27.66
P99 ITL (ms): 91.24
Max ITL (ms): 2609.64
==================================================
```
### 5.2 Accuracy Benchmark
Document model accuracy on standard benchmarks:
#### 5.2.1 GSM8K Benchmark
- Benchmark Command
```bash Command
python -m sglang.test.few_shot_gsm8k \
--num-questions 200
```
- Test Result
```text Output
Accuracy: 0.890
Invalid: 0.000
Latency: 3.718 s
Output throughput: 5245.606 token/s
```
docs_new/cookbook/autoregressive/GLM/GLM-OCR.mdx
+227
View File
@@ -0,0 +1,227 @@
---
title: GLM-OCR
metatags:
description: "Deploy GLM-OCR with SGLang - state-of-the-art OCR performance for complex document understanding."
---
## 1. Model Introduction
[GLM-OCR](https://huggingface.co/zai-org/GLM-OCR) is a multimodal OCR model for complex document understanding, built on the GLM-V encoder–decoder architecture. It introduces Multi-Token Prediction (MTP) loss and stable full-task reinforcement learning to improve training efficiency, recognition accuracy, and generalization.
The model integrates the CogViT visual encoder pre-trained on large-scale image–text data, a lightweight cross-modal connector with efficient token downsampling, and a GLM-0.5B language decoder. Combined with a two-stage pipeline of layout analysis and parallel recognition based on PP-DocLayout-V3, GLM-OCR delivers robust and high-quality OCR performance across diverse document layouts.
**Hardware Support:** NVIDIA B200/H100/H200
**Key Features:**
- **State-of-the-Art Performance**: Achieves 94.62 on OmniDocBench V1.5, ranking #1, and delivers SOTA results across major document understanding benchmarks, including formula recognition, table recognition, and information extraction.
- **Optimized for Real-World Scenarios**: Specifically optimized for practical business cases, maintaining stable and accurate performance on complex tables, code documents, seals, and other challenging layouts.
- **Efficient Inference**: With only 0.9B parameters, GLM-OCR supports deployment via vLLM and SGLang, significantly reducing inference latency and compute cost—well suited for high-concurrency and edge deployments.
- **Easy to Use**: Fully open-sourced with a complete SDK and inference toolchain, enabling one-line invocation and seamless integration into existing systems.
For more details, please refer to the [official GLM-OCR model card](https://huggingface.co/zai-org/GLM-OCR).
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and deployment options. You can optionally enable MTP (Multi-Token Prediction) for faster inference using EAGLE speculative decoding.
import { GLMOCRDeployment } from '/src/snippets/autoregressive/glm-ocr-deployment.jsx'
<GLMOCRDeployment />
### 3.2 Configuration Tips
- **CUDA IPC Transport**: The `SGLANG_USE_CUDA_IPC_TRANSPORT=1` environment variable enables CUDA IPC for transferring multimodal features, which significantly improves TTFT.
- **MTP (Multi-Token Prediction)**: Enable MTP to use EAGLE speculative decoding for faster inference. This feature predicts multiple tokens at once to reduce latency.
- **Memory Management**: For memory-constrained environments, you may need to adjust `--mem-fraction-static` and/or `--max-running-requests`.
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
### 4.2 Advanced Usage
#### 4.2.1 OCR Image Processing
GLM-OCR supports OCR tasks on various document types. Here's a basic example:
```python Example
import time
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:30000/v1",
timeout=3600
)
messages = [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
}
},
{
"type": "text",
"text": "Please extract all text from this image."
}
]
}
]
start = time.time()
response = client.chat.completions.create(
model="zai-org/GLM-OCR",
messages=messages,
max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```
**Example Output:**
```text Output
Response costs: 2.29s
Generated text: CINNAMON SUGAR
1 x 17,000 17,000
SUB TOTAL 17,000
GRAND TOTAL 17,000
CASH IDR 20,000
CHANGE DUE 3,000
```
#### 4.2.2 Complex Document Processing
GLM-OCR excels at processing complex documents including:
- **Tables**: Accurate extraction of tabular data with structure preservation
- **Formulas**: Mathematical formula recognition
- **Code Documents**: Source code extraction from screenshots
- **Seals and Stamps**: Recognition of seals and stamps in documents
- **Multi-layout Documents**: Mixed content with text, images, and tables
```python Example
import time
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:30000/v1",
timeout=3600
)
# Example: Processing a document with tables
messages = [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "YOUR_DOCUMENT_IMAGE_URL"
}
},
{
"type": "text",
"text": "Please extract the table content from this document and format it as markdown."
}
]
}
]
response = client.chat.completions.create(
model="zai-org/GLM-OCR",
messages=messages,
max_tokens=4096
)
print(response.choices[0].message.content)
```
## 5. Benchmark
### 5.1 Accuracy Benchmark
Document model accuracy on standard benchmarks:
#### 5.1.1 OCRBench Benchmark
- Benchmark Command
```bash Command
python3 -m lmms_eval \
--model openai_compatible \
--model_args "model_version=zai-org/GLM-OCR" \
--tasks ocrbench \
--batch_size 128 \
--log_samples \
--log_samples_suffix "openai_compatible" \
--output_path ./logs
```
- Test Result
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "12.5%"}} />
<col style={{width: "12.5%"}} />
<col style={{width: "12.5%"}} />
<col style={{width: "12.5%"}} />
<col style={{width: "12.5%"}} />
<col style={{width: "12.5%"}} />
<col style={{width: "12.5%"}} />
<col style={{width: "12.5%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Tasks</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Version</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Filter</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>n-shot</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Metric</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}></th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Value</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Stderr</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ocrbench</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yaml</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>none</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>ocrbench_accuracy</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>↑</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.806</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A</td>
</tr>
</tbody>
</table>
#### 5.1.2 OmniDocBench V1.5
GLM-OCR achieves **94.62** on OmniDocBench V1.5, ranking #1 among all models, demonstrating state-of-the-art performance across major document understanding benchmarks.
docs_new/cookbook/autoregressive/Google/Gemma4.mdx
+1290
View File
File diff suppressed because it is too large Load Diff
docs_new/cookbook/autoregressive/InclusionAI/LLaDA-2.1.mdx
+701
View File
@@ -0,0 +1,701 @@
---
title: LLaDA 2.1
metatags:
description: "Deploy LLaDA 2.1 with SGLang - large-scale discrete diffusion language model with parallel token generation, iterative denoising, MoE architecture, and reinforcement learning for reasoning."
tag: NEW
---
import { LLaDA21Deployment } from '/src/snippets/autoregressive/llada-21-deployment.jsx';
## 1. Model Introduction
[LLaDA 2.1](https://github.com/inclusionAI/LLaDA2.X) is a series of large-scale discrete diffusion language models (dLLMs) developed by the InclusionAI team at Ant Group. Unlike traditional autoregressive models that generate text left-to-right one token at a time, LLaDA 2.1 uses a diffusion-based approach — drafting tokens in parallel and refining them through iterative denoising, enabling self-correction during generation.
**Key Features:**
- **Token Editing (T2T + M2T)**: Combines Mask-to-Token (M2T) and Token-to-Token (T2T) editing, allowing the model to not only unmask tokens but also revise already-generated tokens mid-flight
- **Dual Decoding Modes**: Speed Mode (S) for maximum throughput with T2T refinement, and Quality Mode (Q) for conservative thresholds and higher benchmark scores
- **MoE Architecture**: Both variants use Mixture-of-Experts architecture for efficient scaling
- **First Large-Scale RL for dLLMs**: Implements the first reinforcement learning framework specifically designed for diffusion language models, improving reasoning and instruction-following
- **Lightning-Fast Decoding**: Up to 892 tokens/s on HumanEval+ for the 100B model
**Available Models:**
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "20.0%"}} />
<col style={{width: "20.0%"}} />
<col style={{width: "20.0%"}} />
<col style={{width: "20.0%"}} />
<col style={{width: "20.0%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Parameters</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Architecture</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Context Length</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>HuggingFace</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**LLaDA2.1-mini**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16B</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>MoE (20 layers, 16 attention heads)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>32,768 tokens</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[inclusionAI/LLaDA2.1-mini](https://huggingface.co/inclusionAI/LLaDA2.1-mini)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**LLaDA2.1-flash**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>100B</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>MoE</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>32,768 tokens</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[inclusionAI/LLaDA2.1-flash](https://huggingface.co/inclusionAI/LLaDA2.1-flash)</td>
</tr>
</tbody>
</table>
**License:**
Apache 2.0. Please refer to the [official LLaDA2.X repository](https://github.com/inclusionAI/LLaDA2.X) for details.
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, and decoding mode. SGLang supports serving LLaDA-2.1 on NVIDIA H100, H200, B200, and AMD MI300X, MI325X, MI355X GPUs.
<LLaDA21Deployment />
### 3.2 Configuration Tips
**dLLM-Specific Parameters:**
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Recommended Value</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--dllm-algorithm`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Diffusion decoding algorithm</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`JointThreshold`</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--trust-remote-code`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Required for LLaDA model loading</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Always enabled</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--mem-fraction-static`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Static memory fraction for KV cache</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`0.8`</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--max-running-requests`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Maximum concurrent requests</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`1` (for best quality)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--attention-backend`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Attention computation backend</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`flashinfer`</td>
</tr>
</tbody>
</table>
**Decoding Mode Comparison:**
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Mode</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Threshold</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Speed</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Quality</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Best For</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Quality Mode (Q)**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Conservative</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Moderate</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Higher benchmark scores</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Accuracy-critical tasks</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Speed Mode (S)**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Aggressive</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Very fast, relies on T2T editing</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Slightly lower</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Throughput-critical tasks</td>
</tr>
</tbody>
</table>
**Hardware Requirements:**
- **LLaDA2.1-mini (16B)**: ~47 GB VRAM, runs on a single GPU (TP=1)
- **LLaDA2.1-flash (100B)**: Requires multi-GPU setup (TP=4 on H100/H200, TP=2 on B200)
## 4. Model Invocation
### 4.1 Deployment
Start the server using the command generated above, for example:
```shell Command
python -m sglang.launch_server \
--model-path inclusionAI/LLaDA2.1-mini \
--dllm-algorithm JointThreshold \
--tp 1 \
--trust-remote-code \
--mem-fraction-static 0.8 \
--max-running-requests 1 \
--attention-backend flashinfer \
--host 0.0.0.0 \
--port 8000
```
### 4.2 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
**Simple Completion Example:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="inclusionAI/LLaDA2.1-mini",
messages=[
{"role": "user", "content": "Explain what a diffusion language model is in simple terms."}
],
max_tokens=1024
)
print(response.choices[0].message.content)
```
**Output Example:**
```text Output
Sure! Let's break it down in simple terms.
A **diffusion language model** is a type of artificial intelligence that learns to generate text—like sentences, stories, or emails—by studying a lot of written text.
Here’s how it works, using a simple real-life analogy:
Imagine you have a big book full of stories. A diffusion language model is trying to learn how to write a new story. Instead of being told the rules, it starts by looking at all the words in the book and trying to understand how words usually go together.
Now, think of the process like this:
1. **Start with random noise**: The model begins with a completely random set of words (like a scribble on paper).
2. ** ** "clean up" the noise**: It gradually "denoises" the noise by turning it into meaningful text, word by word, based on what it learned learned from the book.
3. **Learn from patterns**: As it does this, it learns patterns—like how words often follow each other, or how sentences start.
4. **Generate new text**: Once it’s learned the patterns, it can create new, coherent sentences or stories by starting from a and and building it up word by word.
So, the "diffusion" part comes from the idea of going from random noise to clear, meaningful text—like turning a scribble into a full story.
In short:
A diffusion language model is an AI that learns to write text by reading lots of books and gradually turning random noise into coherent, meaningful sentences based on what it learned.
```
### 4.3 Advanced Usage
#### 4.3.1 Streaming
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="inclusionAI/LLaDA2.1-mini",
messages=[
{"role": "user", "content": "Write a Python function to compute the Fibonacci sequence."}
],
max_tokens=2048,
stream=True
)
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if delta.content:
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
````text Output
Here are several ways to implement the Fibonacci sequence in Python:
## 1. Recursive Approach (Simple but Inefficient)
```python
def fibonacci_recursive(n):
"""
Compute the nth Fibonacci number using recursion.
Args:
n (int): The position in the Fibonacci sequence (0-indexed)
Returns:
int: The nth Fibonacci number
Raises:
ValueError: If n is negative
"""
if n < 0:
raise ValueError("n must be non-negative")
if n <= 1:
return n
return fibonacci_recursive(n - 1) + fibonacci_recursive(n - 2)
# Example usage
print(fibonacci_recursive(10)) # Output: 55
```
## 2. Iterative Approach (Efficient)
...
````
#### 4.3.2 Code Generation
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="inclusionAI/LLaDA2.1-mini",
messages=[
{"role": "user", "content": "Write a Python function that checks if a string is a palindrome. Include docstring and test cases."}
],
max_tokens=2048
)
print(response.choices[0].message.content)
```
**Output Example:**
````text Output
```python
def is_palindrome(s):
"""
Check if a string is a palindrome.
A palindrome is a word, phrase, or sequence that reads the same backward as forward.
This function ignores case, spaces, punctuation, and non characters characters.
Args:
s (str): The string to check
Returns:
bool: True if the string is a palindrome, False otherwise
Examples:
>>> is_palindrome("racecar")
True
>>> is_palindrome("A man a plan a canal Panama")
True
>>> is_palindrome("race a car")
False
>>> is_palindrome("")
True
>>> is_palindrome("a")
True
"""
# Remove non-alphanumeric characters and convert to lowercase
cleaned = ''.join(char.lower() for char in s if char.isalnum())
# Check if the cleaned string reads the same forwards and backwards
return cleaned == cleaned[::-1]
# Test cases
def test_is_palindrome():
"""Test the is_palindrome function with various inputs."""
# Test basic palindromes
assert is_palindrome("racecar") == True
assert is_palindrome("level") == True
assert is_palindrome("madam") == True
assert is_palindrome("radar") == True
# Test palindromes with spaces and punctuation
assert is_palindrome("A man a plan a canal Panama") == True
assert is_palindrome("race a car") == False
assert is_palindrome("Was it a car or a cat I saw?") == True
assert is_palindrome("Madam, I'm Adam") == True
# Test edge cases
assert is_palindrome("") == True
assert is_palindrome("a") == True
assert is_palindrome("A") == True
assert is_palindrome("Aa") == True
# Test non-palindromes
assert is_palindrome("hello") == False
assert is_palindrome("world") == False
assert is_palindrome("python") == False
# Test single characters
assert is_palindrome("1") == True
assert is_palindrome("1") == True
print("All tests passed!")
# Run the tests
if __name__ == "__main__":
# Example usage
print("Testing isalindrome function:")
print(f"'racecar' {is_palindrome('racecar')}")
print(f"'A man a plan a canal Panama': {is_palindrome('A man a plan a canal Panama')}")
print(f"'race a car': {is_palindrome('race a car')}")
print(f"'hello': {is_palindrome('hello')}")
# Run tests
test_is_palindrome()
```
This implementation includes:
1. **Comprehensive function** `is_palindrome()` that:
- Ignores case by converting to lowercase
- Removes all non-alphanumeric characters (spaces, punctuation, etc.)
- Uses string slicing (`[::-1]`) to reverse the string
2. **Detailed docstring** explaining:
- What the function does
- How it works
- Return value
- Examples of usage
3. **Extensive test cases** covering:
- Basic palindromes
- Palindromes with spaces and punctuation
- Edge cases (empty string, single character)
- Non-palindromes
- Mixed case scenarios
4. **Test function** that uses assertions to verify the function works correctly
The function efficiently handles real-world palindrome checking by ignoring case, spaces, and punctuation, making it suitable for phrases like "A man a plan a canal Panama".
````
## 5. Benchmark
This section uses **industry-standard configurations** for comparable benchmark results.
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: NVIDIA B200 (4x)
- SGLang Version: 0.5.8+
#### 5.1.1 LLaDA2.1-mini
**Model Deployment:**
```bash Command
python -m sglang.launch_server \
--model-path inclusionAI/LLaDA2.1-mini \
--dllm-algorithm JointThreshold \
--tp 1 \
--trust-remote-code \
--mem-fraction-static 0.8 \
--max-running-requests 1 \
--attention-backend flashinfer
```
- Latency Benchmark
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model inclusionAI/LLaDA2.1-mini \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
- **Latency Result**:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 9.90
Total input tokens: 6101
Total input text tokens: 6101
Total generated tokens: 4220
Total generated tokens (retokenized): 3433
Request throughput (req/s): 1.01
Input token throughput (tok/s): 616.26
Output token throughput (tok/s): 426.26
Peak output token throughput (tok/s): 1010.00
Peak concurrent requests: 3
Total token throughput (tok/s): 1042.53
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 988.87
Median E2E Latency (ms): 655.27
P90 E2E Latency (ms): 1952.50
P99 E2E Latency (ms): 2932.19
---------------Time to First Token----------------
Mean TTFT (ms): 152.74
Median TTFT (ms): 150.37
P99 TTFT (ms): 229.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 2.16
Median TPOT (ms): 2.08
P99 TPOT (ms): 3.72
---------------Inter-Token Latency----------------
Mean ITL (ms): 2.10
Median ITL (ms): 1.99
P95 ITL (ms): 4.03
P99 ITL (ms): 6.34
Max ITL (ms): 26.59
==================================================
```
- Throughput Benchmark
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model inclusionAI/LLaDA2.1-mini \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100 \
--request-rate inf
```
- **Throughput Result**:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 467.74
Total input tokens: 249831
Total input text tokens: 249831
Total generated tokens: 252662
Total generated tokens (retokenized): 189717
Request throughput (req/s): 1.07
Input token throughput (tok/s): 534.12
Output token throughput (tok/s): 540.17
Peak output token throughput (tok/s): 1753.00
Peak concurrent requests: 105
Total token throughput (tok/s): 1074.30
Concurrency: 90.77
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 84912.27
Median E2E Latency (ms): 86564.26
P90 E2E Latency (ms): 110567.26
P99 E2E Latency (ms): 114303.38
---------------Time to First Token----------------
Mean TTFT (ms): 83920.39
Median TTFT (ms): 85669.54
P99 TTFT (ms): 112969.91
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 2.67
Median TPOT (ms): 1.65
P99 TPOT (ms): 4.43
---------------Inter-Token Latency----------------
Mean ITL (ms): 1.69
Median ITL (ms): 1.46
P95 ITL (ms): 3.96
P99 ITL (ms): 4.84
Max ITL (ms): 92.08
==================================================
```
#### 5.1.2 LLaDA2.1-flash
**Model Deployment:**
```bash Command
python -m sglang.launch_server \
--model-path inclusionAI/LLaDA2.1-flash \
--dllm-algorithm JointThreshold \
--tp 4 \
--trust-remote-code \
--mem-fraction-static 0.8 \
--max-running-requests 1 \
--attention-backend flashinfer
```
- Latency Benchmark
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model inclusionAI/LLaDA2.1-flash \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
- **Latency Result**:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 14.46
Total input tokens: 6101
Total input text tokens: 6101
Total generated tokens: 4220
Total generated tokens (retokenized): 3276
Request throughput (req/s): 0.69
Input token throughput (tok/s): 421.79
Output token throughput (tok/s): 291.75
Peak output token throughput (tok/s): 676.00
Peak concurrent requests: 3
Total token throughput (tok/s): 713.53
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 1445.16
Median E2E Latency (ms): 968.06
P90 E2E Latency (ms): 3101.86
P99 E2E Latency (ms): 4208.49
---------------Time to First Token----------------
Mean TTFT (ms): 231.63
Median TTFT (ms): 242.67
P99 TTFT (ms): 341.33
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 3.04
Median TPOT (ms): 2.79
P99 TPOT (ms): 5.33
---------------Inter-Token Latency----------------
Mean ITL (ms): 3.05
Median ITL (ms): 2.41
P95 ITL (ms): 7.25
P99 ITL (ms): 8.27
Max ITL (ms): 29.27
==================================================
```
- Throughput Benchmark
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model inclusionAI/LLaDA2.1-flash \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100 \
--request-rate inf
```
- **Throughput Result**:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 671.85
Total input tokens: 249831
Total input text tokens: 249831
Total generated tokens: 252662
Total generated tokens (retokenized): 177961
Request throughput (req/s): 0.74
Input token throughput (tok/s): 371.85
Output token throughput (tok/s): 376.07
Peak output token throughput (tok/s): 1521.00
Peak concurrent requests: 103
Total token throughput (tok/s): 747.92
Concurrency: 91.28
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 122658.36
Median E2E Latency (ms): 125265.55
P90 E2E Latency (ms): 159554.07
P99 E2E Latency (ms): 165174.88
---------------Time to First Token----------------
Mean TTFT (ms): 121009.17
Median TTFT (ms): 124437.80
P99 TTFT (ms): 163579.29
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 4.73
Median TPOT (ms): 2.16
P99 TPOT (ms): 7.13
---------------Inter-Token Latency----------------
Mean ITL (ms): 2.38
Median ITL (ms): 1.40
P95 ITL (ms): 6.89
P99 ITL (ms): 8.60
Max ITL (ms): 176.78
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 GSM8K Benchmark
```bash Command
python -m sglang.test.few_shot_gsm8k \
--num-questions 200 \
--port 8000
```
**Results:**
```text Output
Accuracy: 0.895
Invalid: 0.000
Latency: 100.552 s
Output throughput: 262.094 token/s
```
docs_new/cookbook/autoregressive/InclusionAI/Ling-2.5-1T.mdx
+221
View File
@@ -0,0 +1,221 @@
---
title: Ling-2.5-1T
metatags:
description: "Deploy Ling-2.5-1T with SGLang - 1T parameter MoE model with 63B active parameters, trillion-scale context length up to 1M tokens, and agentic tool calling capabilities."
tag: NEW
---
## 1. Model Introduction
[Ling-2.5-1T](https://huggingface.co/inclusionAI/Ling-2.5-1T) is the latest flagship instant model in the Ling family. Thinking models raise the ceiling of intelligence, while instant models expand its reach by balancing efficiency and performance—making AGI not only more powerful, but also more accessible. Ling-2.5-1T delivers comprehensive upgrades across model architecture, token efficiency, and preference alignment, designed to bring universally accessible AI to a new level of quality.
**Key Features:**
- **Trillion-Scale Model**: 1T total parameters with 63B active parameters (up from 51B in the previous generation). Pre-training corpus expanded from 20T to 29T tokens. Leveraging an efficient hybrid linear attention architecture (1:7 MLA + Lightning Linear Attention), the model delivers exceptionally high throughput while processing context lengths of up to 1M tokens.
- **Token Efficiency**: By introducing a composite reward mechanism combining "Correctness" and "Process Redundancy", Ling-2.5-1T further pushes the frontier of efficiency-performance balance in instant models. At comparable token efficiency levels, Ling-2.5-1T's reasoning capabilities significantly outperform its predecessor, approaching the level of frontier "thinking models" that typically consume ~4x the output tokens.
- **Preference Alignment**: Through refined alignment strategies—such as bidirectional RL feedback and Agent-based instruction constraint verification—Ling-2.5-1T achieves substantial improvements over the previous generation in preference alignment tasks, including creative writing and instruction following.
- **Agentic Capabilities**: Trained with Agentic RL in large-scale high-fidelity interactive environments, Ling-2.5-1T is compatible with mainstream agent platforms such as Claude Code, OpenCode, and OpenClaw. It achieves leading open-source performance on the general tool-calling benchmark, BFCL-V4.
- **Context Length**: 256K -> 1M (YaRN)
**Available Models:**
- **BF16**: [inclusionAI/Ling-2.5-1T](https://huggingface.co/inclusionAI/Ling-2.5-1T)
**License:** MIT
## 2. SGLang Installation
Ling-2.5-1T requires a specific SGLang Docker image:
```bash Command
# For H200/B200
docker pull lmsysorg/sglang:nightly-dev-20260213-a0ebaa64
# For GB200/GB300
docker pull lmsysorg/sglang:nightly-dev-cu13-20260213-a0ebaa64
```
For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/installation).
Ling-2.5-1T is also supported via the **nightly PyPI builds**. See the [SGLang Installation (PyPI)](../../../docs/get-started/installation) guide for setup instructions.
## 3. Model Deployment
Ling-2.5-1T is a trillion-parameter BF16 model that requires multi-node deployment (at least 2 nodes). Use the configuration selector below to generate the deployment command for your hardware platform.
import { Ling251TDeployment } from '/src/snippets/autoregressive/ling-25-1t-deployment.jsx'
<Ling251TDeployment />
### Configuration Tips
- The `--trust-remote-code` flag is required for this model due to custom modeling code.
- `--tp-size` can be set to a maximum of 8 for this model. If you have more GPUs available, increase `--pp-size` to scale across additional nodes.
- Adding `--model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}'` enables faster model loading.
- On H200/GB200/GB300 with 2-node deployment, `--mem-frac 0.95` is required to avoid OOM since the model occupies most of the GPU memory. For better throughput, consider 4-node deployment (ref [model card](https://huggingface.co/inclusionAI/Ling-2.5-1T#run-inference) for more details).
## 4. Model Invocation
### 4.1 Basic Usage
For example, launch the server on 2 H200 nodes:
```bash Command
export MASTER_IP=10.10.0.1 # The IP of Node 0
export PORT=30000
export DIST_PORT=50000
# Node 0:
python3 -m sglang.launch_server \
--model-path inclusionAI/Ling-2.5-1T \
--trust-remote-code \
--tp-size 8 \
--pp-size 2 \
--nnodes 2 \
--node-rank 0 \
--host 0.0.0.0 \
--port ${PORT} \
--dist-init-addr ${MASTER_IP}:${DIST_PORT} \
--tool-call-parser qwen \
--model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}' \
--mem-frac 0.95
# Node 1:
python3 -m sglang.launch_server \
--model-path inclusionAI/Ling-2.5-1T \
--trust-remote-code \
--tp-size 8 \
--pp-size 2 \
--nnodes 2 \
--node-rank 1 \
--dist-init-addr ${MASTER_IP}:${DIST_PORT} \
--tool-call-parser qwen \
--model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}' \
--mem-frac 0.95
```
Once the server is running, send requests to the master node:
```bash Command
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "auto", "messages": [{"role": "user", "content": "What is the capital of France?"}]}'
```
Output:
```json Config
{
"id": "e82af153da844ee6aed7a27a3187f2f4",
"object": "chat.completion",
"created": 1771216764,
"model": "auto",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "The capital of France is **Paris**.\n\n**Additional details:**\n* It is the largest city in France.\n* It is located in the north-central part of the country along the Seine River.\n* Paris is often referred to as \"The City of Light\" (*La Ville Lumière*).",
"reasoning_content": null,
"tool_calls": null
},
"logprobs": null,
"finish_reason": "stop",
"matched_stop": 156895
}
],
"usage": {
"prompt_tokens": 25,
"total_tokens": 93,
"completion_tokens": 68,
"prompt_tokens_details": null,
"reasoning_tokens": 0
}
}
```
For more API usage examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Tool Calling Example
```bash Command
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "inclusionAI/Ling-2.5-1T",
"messages": [{"role": "user", "content": "Search for the latest news about AI"}],
"tools": [{
"type": "function",
"function": {
"name": "search",
"description": "Search for information on the internet",
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string", "description": "The search query"}
},
"required": ["query"]
}
}
}],
"tool_choice": "auto"
}'
```
Output:
```json Config
{
"id": "b968e45c7d414f7482c8ffc0f9c6b688",
"object": "chat.completion",
"created": 1771216520,
"model": "inclusionAI/Ling-2.5-1T",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": null,
"reasoning_content": null,
"tool_calls": [
{
"id": "call_e75f711d8ad840ed9d382c9e",
"index": 0,
"type": "function",
"function": {
"name": "search",
"arguments": "{\"query\": \"latest news about AI\"}"
}
}
]
},
"logprobs": null,
"finish_reason": "tool_calls",
"matched_stop": null
}
],
"usage": {
"prompt_tokens": 173,
"total_tokens": 196,
"completion_tokens": 23,
"prompt_tokens_details": null,
"reasoning_tokens": 0
}
}
```
## 5. Benchmark
### GSM8K
- Benchmark Command
```bash Command
python3 benchmark/gsm8k/bench_sglang.py
```
- Test Result
```text Output
Accuracy: 0.960
Invalid: 0.000
Latency: 45.410 s
Output throughput: 560.642 token/s
```
docs_new/cookbook/autoregressive/InclusionAI/Ring-2.5-1T.mdx
+266
View File
@@ -0,0 +1,266 @@
---
title: Ring-2.5-1T
metatags:
description: "Deploy Ring-2.5-1T with SGLang - world's first open-source 1T parameter reasoning model with hybrid linear attention, deep reasoning, and agentic tool calling capabilities."
tag: NEW
---
## 1. Model Introduction
[Ring-2.5-1T](https://huggingface.co/inclusionAI/Ring-2.5-1T) is the world's first open-source trillion-parameter reasoning model based on hybrid linear attention architecture, developed by InclusionAI. Building on Ring-1T, Ring-2.5-1T demonstrates substantial improvements in generation efficiency, reasoning depth, and long-horizon task execution capabilities.
**Key Features:**
- **Trillion-Scale Model**: ~1T total parameters with 63B activation parameters using a hybrid linear attention architecture (1:7 MLA + Lightning Linear Attention)
- **Generation Efficiency**: Reduces memory access overhead by over 10x and increases generation throughput by more than 3x for sequences exceeding 32K tokens
- **Deep Reasoning**: Achieves gold medal level for both IMO 2025 and CMO 2025, with dense rewards for rigorous reasoning process feedback
- **Long-horizon Task Execution**: Enhanced autonomous execution capability through large-scale fully-async agentic RL training
- **Tool Calling**: Supports function calling with XML-style tool call format
- **Context Length**: 128K -> 256K (YaRN)
**Available Models:**
- **FP8 (8-bit quantized)**: [inclusionAI/Ring-2.5-1T](https://huggingface.co/inclusionAI/Ring-2.5-1T)
**License:** MIT
## 2. SGLang Installation
Ring-2.5-1T requires a specific SGLang Docker image:
```bash Command
# For H200/B200
docker pull lmsysorg/sglang:nightly-dev-20260213-a0ebaa64
# For GB200/GB300
docker pull lmsysorg/sglang:nightly-dev-cu13-20260213-a0ebaa64
# For MI300X/325X
docker pull lmsysorg/sglang:v0.5.9-rocm700-mi30x
# For MI355X
docker pull lmsysorg/sglang:v0.5.9-rocm700-mi35x
```
For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/installation).
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform.
import { Ring251TDeployment } from '/src/snippets/autoregressive/ring-25-1t-deployment.jsx'
<Ring251TDeployment />
### 3.2 Configuration Tips
- The `--trust-remote-code` flag is required for this model due to custom modeling code.
- The model uses FP8 quantization (compressed-tensors format).
## 4. Model Invocation
Deploy Ring-2.5-1T with the following command (on H200, all features enabled):
```shell Command
sglang serve \
--model-path inclusionAI/Ring-2.5-1T \
--tp 8 \
--trust-remote-code \
--host 0.0.0.0 \
--port 30000
```
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
To enable reasoning output separation, add `--reasoning-parser deepseek-r1` when launching the server. The thinking process is returned via `reasoning_content` in the streaming response.
```shell Command
sglang serve \
--model-path inclusionAI/Ring-2.5-1T \
--tp 8 \
--trust-remote-code \
--reasoning-parser deepseek-r1 \
--host 0.0.0.0 \
--port 30000
```
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="inclusionAI/Ring-2.5-1T",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
max_tokens=2048,
stream=True
)
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
print(delta.reasoning_content, end="", flush=True)
if delta.content:
print(delta.content, end="", flush=True)
print()
```
<details>
<summary>Output Example</summary>
````text Output
We are asked: "Solve this problem step by step: What is 15% of 240?" This is a straightforward percentage calculation. We need to show step-by-step solution.
We can compute 15% of 240 as (15/100)*240 = 0.15 * 240 = 36.
But we need to present step by step. Also ensure it's clear.
We could also break down: 10% of 240 = 24, then 5% = 12, so 15% = 36.
But any method is fine.
We'll produce a solution with explanation: "To find 15% of 240, multiply 240 by 0.15 (or 15/100)."
We'll show:
15% = 15/100 = 0.15
Then 0.15 × 240 = 36.
Alternatively: (15/100) × 240 = (15 × 240) / 100 = 3600/100 = 36.
Finally, answer: 36.
We can also illustrate stepwise: "First, convert the percentage to a decimal: 15% = 0.15. Then multiply by the number: 0.15 × 240 = 36."
We'll present as a final answer: \boxed{36}.
However, we need to provide step-by-step solution as per instructions. We'll write a full explanation.
We can also use the fraction method: 15% of 240 = (15/100)*240 = (15*240)/100 = 3600/100 = 36.
Alr.
I think that's it.
**Step 1:** Write 15% as a fraction or decimal.
\[ 15\% = \frac{15}{100} = 0.15\]
**Step 2:** Multiply the number (240) by this fraction/decimal.
\[ 240 \times 0.15 = 36\]
Alternatively, using the fraction:
\[ \frac{15}{100} \times 240 = \frac{15 \times 240}{100} = \frac{3600}{100} = 36\]
**Conclusion:** 15% of 240 is 36.
\[ \boxed{36} \]
````
</details>
#### 4.2.2 Tool Calling
To enable tool calling, add `--tool-call-parser qwen` when launching the server.
```shell Command
sglang serve \
--model-path inclusionAI/Ring-2.5-1T \
--tp 8 \
--trust-remote-code \
--tool-call-parser qwen \
--host 0.0.0.0 \
--port 30000
```
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
}
},
"required": ["location"]
}
}
}
]
response = client.chat.completions.create(
model="inclusionAI/Ring-2.5-1T",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools
)
print(response.choices[0].message.tool_calls)
```
**Output Example:**
```text Output
[ChatCompletionMessageFunctionToolCall(id='call_770360e31d194ed79d32cd8c', function=Function(arguments='{"location": "Beijing"}', name='get_weather'), type='function', index=0)]
```
## 5. Benchmark
### GSM8K
- Deployment Command
```bash Command
sglang serve \
--model-path inclusionAI/Ring-2.5-1T \
--tp-size 8 \
--trust-remote-code
```
- Benchmark Command
```bash Command
python3 benchmark/gsm8k/bench_sglang.py --temperature 1.2 --top-p 0.8 --max-new-tokens 32768 --num-questions 200 --tokenizer-path inclusionAI/Ring-2.5-1T --enable-thinking
```
- Test Result
```text Output
Accuracy: 0.955
Invalid: 0.010
Latency: 615.833 s
Output throughput: 412.360 token/s
```
docs_new/cookbook/autoregressive/InternLM/Intern-S1.mdx
+28
View File
@@ -0,0 +1,28 @@
---
title: Intern-S1
metatags:
description: "Deploy Intern-S1 with SGLang - community contribution guide for InternLM's Intern-S1 model deployment."
---
## 📝 Community Contribution Welcome
This guide is currently under development. We welcome community contributions!
If you have experience deploying **Intern-S1** with SGLang, please help us complete this documentation.
## 🚀 How to Contribute
```shell Command
git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git
cd sglang-cookbook
git checkout -b add-intern-s1-guide
# Edit this file and submit a PR
```
## 📚 Reference
- [GLM-4.6V](../GLM/GLM-4.6V)
---
**Let's build this together!** 🌟
docs_new/cookbook/autoregressive/InternVL/InternVL3.5.mdx
+29
View File
@@ -0,0 +1,29 @@
---
title: InternVL3.5
metatags:
description: "Deploy InternVL3.5 vision-language model with SGLang - community contribution guide for OpenGVLab's multimodal model."
---
## 📝 Community Contribution Welcome
This guide is currently under development. We welcome community contributions!
If you have experience deploying **InternVL3.5** with SGLang, please help us complete this documentation.
## 🚀 How to Contribute
```shell Command
git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git
cd sglang-cookbook
git checkout -b add-internvl3-5-guide
# Edit this file and submit a PR
```
## 📚 Reference
- [GLM-4.6V](../GLM/GLM-4.6V)
---
**Let's build this together!** 🌟
docs_new/cookbook/autoregressive/Jina/Jina-reranker-m0.mdx
+28
View File
@@ -0,0 +1,28 @@
---
title: Jina-reranker-m0
metatags:
description: "Deploy Jina-reranker-m0 with SGLang - community contribution guide for Jina AI's reranker model deployment."
---
## 📝 Community Contribution Welcome
This guide is currently under development. We welcome community contributions!
If you have experience deploying **Jina-reranker-m0** with SGLang, please help us complete this documentation.
## 🚀 How to Contribute
```shell Command
git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git
cd sglang-cookbook
git checkout -b add-jina-reranker-m0-guide
# Edit this file and submit a PR
```
## 📚 Reference
- [DeepSeek-V3.2](../DeepSeek/DeepSeek-V3_2.md)
---
**Let's build this together!** 🌟
docs_new/cookbook/autoregressive/Llama/Llama3.1.mdx
+644
View File
@@ -0,0 +1,644 @@
---
title: Llama-3.1
metatags:
description: "Deploy Llama 3.1 (8B/70B/405B) with SGLang - 128K context, tool use, multilingual support, and speculative decoding optimization."
---
## 1. Model Introduction
Llama 3.1 is a collection of pretrained and instruction tuned generative models, released in July 2024 by Meta. These models are available in 8B, 70B and 405B sizes, with the 405B variant being the most capable fully-open source model at the time.
These models bring open intelligence to all, with several new features and improvements:
- **Stronger General Intelligence**: These models showcase significant improvements in coding, state-of-the-art tool use, and overall stronger reasoning capabilities.
- **Extended Context Length**: Llama 3.1 extends the context length to 128K tokens to improve performance over long context tasks such as summarization and code reasoning.
- **Tool Use**: Llama 3.1 is trained to interact with a search engine, python interpreter and mathematical engine, and also improves zero-shot tool use capabilities to interact with potentially unseen tools.
- **Multilinguality**: Llama 3.1 supports 7 languages in addition to English: French, German, Hindi, Italian, Portuguese, Spanish, and Thai.
For further details, please refer to the [Llama 3.1 blog](https://ai.meta.com/blog/meta-llama-3-1/) and the [Llama 3.1 model card](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md).note
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to generate a launch command for Llama 3.1 collection of models.
import { Llama31Deployment } from "/src/snippets/autoregressive/llama31-deployment.jsx";
<Llama31Deployment />
### 3.2 Configuration Tips
**Speculative Decoding (NVIDIA GPUs):**
- Using Speculative Decoding for latency-sensitive scenarios:
- `--speculative-algorithm EAGLE3`: Speculative decoding algorithm
- `--speculative-num-steps 3`: Number of speculative verification rounds
- `--speculative-eagle-topk 1`: Top-k sampling for draft tokens
- `--speculative-num-draft-tokens 4`: Number of draft tokens per step
- `--speculative-draft-model-path`: The path of the draft model weights. This can be a local folder or a Hugging Face repo ID such as [`yuhuili/EAGLE3-LLaMA3.1-Instruct-8B`](https://huggingface.co/yuhuili/EAGLE3-LLaMA3.1-Instruct-8B).
**AMD GPU Deployment:**
- **Hardware-Aware TP**: MI355X (256GB memory) supports lower TP values compared to MI300X/MI325X (192GB)
- **Verified TP Configurations**:
- MI300X/MI325X: 405B BF16 (TP=8), 405B FP8 (TP=4), 70B/8B (TP=1)
- MI355X: 405B BF16 (TP=4), 405B FP8 (TP=2), 70B/8B (TP=1)
- **FP8 Model Variants**:
- 405B: Use Meta's official `meta-llama/Llama-3.1-405B-Instruct-FP8`
- 70B/8B: Use AMD's optimized `amd/Llama-3.1-{size}-Instruct-FP8-KV`
- **Tool Calling**: Enable with `--tool-call-parser llama3` for Instruct models
## 4. Model Invocation
### 4.1 Basic Usage
SGLang exposes an OpenAI-compatible endpoint. First, start the server
```shell Command
sglang serve \
--model-path Meta-Llama/Llama-3.1-405B-Instruct \
--tp 8
```
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY",
)
resp = client.chat.completions.create(
model="Meta-Llama/Llama-3.1-405B-Instruct",
messages=[
{"role": "system", "content": "You are a helpful coding assistant."},
{"role": "user", "content": "Write a Python function that retries a request with exponential backoff."},
],
temperature=0.2,
max_tokens=512,
)
print(resp.choices[0].message.content)
```
**Output Example:**
````text Output
**Exponential Backoff Retry Function in Python**
=====================================================
Below is a Python function that uses the `requests` library to retry a request with exponential backoff.
```python
import requests
import time
import random
def exponential_backoff_retry(url, method, retries=3, backoff_factor=1, max_delay=60):
"""
Retry a request with exponential backoff.
Args:
url (str): The URL to make the request to.
method (str): The HTTP method to use (e.g. 'GET', 'POST', etc.).
retries (int): The number of retries to attempt. Defaults to 3.
backoff_factor (int): The factor to multiply the delay by for each retry. Defaults to 1.
max_delay (int): The maximum delay to wait between retries in seconds. Defaults to 60.
Returns:
The response object from the successful request.
"""
delay = 1
for attempt in range(retries + 1):
try:
response = requests.request(method, url)
response.raise_for_status() # Raise an exception for HTTP errors
return response
except requests.RequestException as e:
if attempt < retries:
# Calculate the delay for this retry
delay = min(delay * backoff_factor, max_delay)
# Add a random jitter to the delay to prevent thundering herd problem
delay += random.uniform(0, delay * 0.1)
# Wait for the calculated delay before retrying
time.sleep(delay)
else:
# If all retries have failed, raise the exception
raise e
...
````
### 4.2 Advanced Usage
#### 4.2.1 Tool Calling
Llama3 supports tool calling capabilities. First, start the server with tool call parser enabled:
```shell Command
sglang serve \
--model-path Meta-Llama/Llama-3.1-405B-Instruct \
--tool-call-parser llama3 \
--tp 8
```
**Python Example**
```python Example
from openai import OpenAI
client = OpenAI(api_key="None", base_url=f"http://0.0.0.0:8000/v1")
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the weather in a given location",
"parameters": {
"type": "object",
"properties": {
"city": {
"type": "string",
"description": "The city to find the weather for, e.g. 'San Francisco'",
},
"unit": {
"type": "string",
"description": "The unit to fetch the temperature in",
"enum": ["celsius", "fahrenheit"],
},
},
"required": ["city", "unit"],
},
},
}
]
response = client.chat.completions.create(
model="meta-llama/Llama-3.1-405B-Instruct",
messages=[
{
"role": "user",
"content": "What's the weather like in Boston today?",
}
],
temperature=0.7,
stream=True,
tools=tools,
)
arguments = []
tool_calls_accumulator = {}
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if hasattr(delta, 'tool_calls') and delta.tool_calls:
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {
'name': None,
'arguments': ''
}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]['name'] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
# Print content
if delta.content:
print(delta.content, end="", flush=True)
# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"🔧 Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
print()
```
Reference: [SGLang Tool Parser Documentation](../../../docs/advanced_features/tool_parser#OpenAI-Compatible-API)
**Output Example**
```text Output
🔧 Tool Call: get_weather
Arguments: {"city": "Boston", "unit": "fahrenheit"}
```
**Handling Tool Call Results**
After getting the tool call, you can execute the function:
```python Example
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather like in Boston today?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Boston", "unit": "fahrenheit"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Boston", "fahrenheit")
}
]
final_response = client.chat.completions.create(
model="Meta-Llama/Llama-3.1-405B-Instruct",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The current weather in Boston is **22°C** and **sunny**. A perfect day to spend outside"
```
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: NVIDIA A100 GPU (8x)
- Model: Meta-Llama/Llama-3.1-70B
- Tensor Parallelism: 8
- sglang version: 0.5.6
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.
#### 5.1.1 Standard Scenario Benchmark
- Model Deployment Command:
```shell Command
sglang serve \
--model-path Meta-Llama/Llama-3.1-70B \
--tp 8
```
##### 5.1.1.1 Low Concurrency
- Benchmark Command:
```shell Command
sglang serve \
--backend sglang \
--model Meta-Llama/Llama-3.1-70B \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 79.81
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4208
Request throughput (req/s): 0.13
Input token throughput (tok/s): 76.44
Output token throughput (tok/s): 52.88
Peak output token throughput (tok/s): 54.00
Peak concurrent requests: 2
Total token throughput (tok/s): 129.32
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 7977.81
Median E2E Latency (ms): 6373.48
---------------Time to First Token----------------
Mean TTFT (ms): 131.61
Median TTFT (ms): 131.77
P99 TTFT (ms): 163.88
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 18.63
Median TPOT (ms): 18.63
P99 TPOT (ms): 18.65
---------------Inter-Token Latency----------------
Mean ITL (ms): 18.64
Median ITL (ms): 18.64
P95 ITL (ms): 18.69
P99 ITL (ms): 18.74
Max ITL (ms): 21.95
==================================================
```
##### 5.1.1.2 Medium Concurrency
```shell Command
sglang serve \
--backend sglang \
--model-path Meta-Llama/Llama-3.1-70B \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 79.47
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 40805
Total generated tokens (retokenized): 38450
Request throughput (req/s): 1.01
Input token throughput (tok/s): 499.17
Output token throughput (tok/s): 513.48
Peak output token throughput (tok/s): 674.00
Peak concurrent requests: 20
Total token throughput (tok/s): 1012.65
Concurrency: 13.47
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 13376.67
Median E2E Latency (ms): 14130.48
---------------Time to First Token----------------
Mean TTFT (ms): 264.84
Median TTFT (ms): 147.02
P99 TTFT (ms): 791.93
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 26.09
Median TPOT (ms): 26.08
P99 TPOT (ms): 34.65
---------------Inter-Token Latency----------------
Mean ITL (ms): 25.76
Median ITL (ms): 23.95
P95 ITL (ms): 24.72
P99 ITL (ms): 98.32
Max ITL (ms): 478.92
==================================================
```
##### 5.1.1.3 High Concurrency
```shell Command
sglang serve \
--backend sglang \
--model-path Meta-Llama/Llama-3.1-70B \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 131.64
Total input tokens: 249831
Total input text tokens: 249831
Total input vision tokens: 0
Total generated tokens: 252662
Total generated tokens (retokenized): 243641
Request throughput (req/s): 3.80
Input token throughput (tok/s): 1897.87
Output token throughput (tok/s): 1919.38
Peak output token throughput (tok/s): 3100.00
Peak concurrent requests: 107
Total token throughput (tok/s): 3817.25
Concurrency: 89.70
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 23616.71
Median E2E Latency (ms): 22770.44
---------------Time to First Token----------------
Mean TTFT (ms): 245.98
Median TTFT (ms): 184.22
P99 TTFT (ms): 1251.67
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 47.19
Median TPOT (ms): 48.67
P99 TPOT (ms): 56.37
---------------Inter-Token Latency----------------
Mean ITL (ms): 46.34
Median ITL (ms): 33.46
P95 ITL (ms): 108.61
P99 ITL (ms): 166.11
Max ITL (ms): 1107.09
==================================================
```
#### 5.1.2 Summarization Scenario Benchmark
##### 5.1.2.1 Low Concurrency
```shell Command
sglang serve \
--backend sglang \
--model-path Meta-Llama/Llama-3.1-70B\
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 83.25
Total input tokens: 41941
Total input text tokens: 41941
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4220
Request throughput (req/s): 0.12
Input token throughput (tok/s): 503.77
Output token throughput (tok/s): 50.69
Peak output token throughput (tok/s): 54.00
Peak concurrent requests: 2
Total token throughput (tok/s): 554.46
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 8322.45
Median E2E Latency (ms): 6873.36
---------------Time to First Token----------------
Mean TTFT (ms): 395.25
Median TTFT (ms): 318.02
P99 TTFT (ms): 850.80
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 18.80
Median TPOT (ms): 18.81
P99 TPOT (ms): 19.03
---------------Inter-Token Latency----------------
Mean ITL (ms): 18.83
Median ITL (ms): 18.81
P95 ITL (ms): 19.06
P99 ITL (ms): 19.08
Max ITL (ms): 23.08
==================================================
```
##### 5.1.2.2 Medium Concurrency
```shell Command
sglang serve \
--backend sglang \
--model-path Meta-Llama/Llama-3.1-70B \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 107.12
Total input tokens: 300020
Total input text tokens: 300020
Total input vision tokens: 0
Total generated tokens: 41669
Total generated tokens (retokenized): 41603
Request throughput (req/s): 0.75
Input token throughput (tok/s): 2800.81
Output token throughput (tok/s): 389.00
Peak output token throughput (tok/s): 624.00
Peak concurrent requests: 19
Total token throughput (tok/s): 3189.81
Concurrency: 14.18
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 18988.30
Median E2E Latency (ms): 20290.66
---------------Time to First Token----------------
Mean TTFT (ms): 603.42
Median TTFT (ms): 531.82
P99 TTFT (ms): 2607.95
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 36.94
Median TPOT (ms): 36.73
P99 TPOT (ms): 79.19
---------------Inter-Token Latency----------------
Mean ITL (ms): 35.36
Median ITL (ms): 25.72
P95 ITL (ms): 27.07
P99 ITL (ms): 439.74
Max ITL (ms): 2529.51
==================================================
```
##### 5.1.2.3 High Concurrency
```shell Command
sglang serve \
--backend sglang \
--model-path Meta-Llama/Llama-3.1-70B \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 320 \
--max-concurrency 64
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 215.66
Total input tokens: 1273893
Total input text tokens: 1273893
Total input vision tokens: 0
Total generated tokens: 170000
Total generated tokens (retokenized): 169035
Request throughput (req/s): 1.48
Input token throughput (tok/s): 5906.92
Output token throughput (tok/s): 788.27
Peak output token throughput (tok/s): 1920.00
Peak concurrent requests: 69
Total token throughput (tok/s): 6695.19
Concurrency: 60.01
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 40443.85
Median E2E Latency (ms): 39813.12
---------------Time to First Token----------------
Mean TTFT (ms): 633.32
Median TTFT (ms): 616.38
P99 TTFT (ms): 1912.97
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 74.95
Median TPOT (ms): 82.85
P99 TPOT (ms): 118.46
---------------Inter-Token Latency----------------
Mean ITL (ms): 75.08
Median ITL (ms): 34.12
P95 ITL (ms): 261.18
P99 ITL (ms): 828.12
Max ITL (ms): 1970.03
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 GSM8K Benchmark
- **Benchmark Command:**
```shell Command
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
```
- **Results**:
```text Output
Accuracy: 0.830
Invalid: 0.000
Latency: 11.794 s
Output throughput: 1406.961 token/s
```
docs_new/cookbook/autoregressive/Llama/Llama3.3-70B.mdx
+229
View File
@@ -0,0 +1,229 @@
---
title: Llama-3.3-70B
metatags:
description: "Deploy Llama-3.3-70B-Instruct with SGLang on AMD GPUs - 128K context, enhanced reasoning, tool calling, and multilingual support."
---
## 1. Model Introduction
[Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) is Meta's latest 70 billion parameter instruction-tuned language model, featuring improved performance and efficiency over Llama 3.1. With a 128K token context window and enhanced capabilities across reasoning, coding, and multilingual tasks, Llama 3.3 delivers state-of-the-art results while maintaining accessibility for production deployment.
**Key Features:**
- **Enhanced Performance**: Improved instruction following, reasoning, and task completion over Llama 3.1
- **Tool Calling**: Native support for function calling and tool use scenarios
- **Multilingual Support**: Optimized for 8 languages (English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai)
- **Extended Context**: 128K token context window for processing long documents and complex tasks
- **Efficient Deployment**: 70B parameters enable deployment on single GPU with AMD MI300X
**License:**
Llama 3.3 is licensed under the Llama 3.3 Community License. See [LICENSE](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct/blob/main/LICENSE) for details.
For more details, please refer to the [official Llama models repository](https://github.com/meta-llama/llama-models).
## 2. SGLang Installation
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for AMD GPUs (MI300X, MI325X, MI355X).
### 3.1 Interactive Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your AMD GPU setup.
import { Llama33Deployment } from "/src/snippets/autoregressive/llama33-70b-deployment.jsx";
<Llama33Deployment />
### 3.2 Configuration Tips
**AMD GPU Deployment:**
- All AMD GPUs (MI300X, MI325X, MI355X) support TP=1 for both BF16 and FP8 variants
- **FP8 Model Variant**: Use AMD's optimized `amd/Llama-3.3-70B-Instruct-FP8-KV`
- **Tool Calling**: Enable with `--tool-call-parser llama3` for function calling support
- **Higher Throughput**: Optional TP=2 or TP=4 can be used for increased throughput
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Advanced Usage
#### 4.2.1 Tool Calling
Llama 3.3 70B Instruct supports native tool calling. Enable the tool parser during deployment:
```shell Command
python -m sglang.launch_server \
--model-path meta-llama/Llama-3.3-70B-Instruct \
--tool-call-parser llama3 \
--tp 1 \
--host 0.0.0.0 \
--port 30000
```
**Python Example:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request
response = client.chat.completions.create(
model="meta-llama/Llama-3.3-70B-Instruct",
messages=[
{"role": "user", "content": "What's the weather in Tokyo?"}
],
tools=tools,
temperature=0.7
)
# Check for tool calls
message = response.choices[0].message
if message.tool_calls:
tool_call = message.tool_calls[0]
print(f"Function: {tool_call.function.name}")
print(f"Arguments: {tool_call.function.arguments}")
```
**Handling Tool Call Results:**
```python Example
# After executing the function, send the result back
def get_weather(location, unit="celsius"):
# Your weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Build conversation with tool result
messages = [
{"role": "user", "content": "What's the weather in Tokyo?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Tokyo", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Tokyo", "celsius")
}
]
final_response = client.chat.completions.create(
model="meta-llama/Llama-3.3-70B-Instruct",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The current weather in Tokyo is 22°C and sunny. A perfect day!"
```
#### 4.2.2 Long Context Processing
Leverage the 128K context window for processing long documents:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Example with long document
long_document = "..." * 10000 # Your long document here
response = client.chat.completions.create(
model="meta-llama/Llama-3.3-70B-Instruct",
messages=[
{"role": "user", "content": f"Summarize this document:\n\n{long_document}"}
],
temperature=0.7,
max_tokens=1000
)
print(response.choices[0].message.content)
```
## 5. Benchmarking
Use the SGLang benchmarking suite to test model performance with different workload patterns:
### 5.1 Basic Benchmark Command
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--dataset-name random \
--num-prompts 1000 \
--random-input 1024 \
--random-output 1024 \
--max-concurrency 16
```
### 5.2 Adjusting Benchmark Parameters
**Input/Output Length**: Adjust `--random-input` and `--random-output` to test different workload patterns:
- Short conversations: `--random-input 1024 --random-output 1024`
- Long outputs: `--random-input 1024 --random-output 8192`
- Long inputs: `--random-input 8192 --random-output 1024`
**Concurrency Levels**: Adjust `--max-concurrency` to test different load scenarios:
- Low concurrency (latency-focused): `--max-concurrency 1 --num-prompts 100`
- Medium concurrency (balanced): `--max-concurrency 16 --num-prompts 1000`
- High concurrency (throughput-focused): `--max-concurrency 100 --num-prompts 2000`
---
## 📚 Additional Resources
- [Meta Llama Models Repository](https://github.com/meta-llama/llama-models)
- [Llama 3.3 Model Card](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct)
- [SGLang Documentation](/)
- [AMD ROCm Documentation](https://rocm.docs.amd.com/)
docs_new/cookbook/autoregressive/Llama/Llama4.mdx
+474
View File
@@ -0,0 +1,474 @@
---
title: Llama 4
metatags:
description: "Deploy Llama 4 Scout and Maverick with SGLang - Meta's latest generation open-source LLMs with industry-leading performance."
---
import { Llama4ScoutDeployment } from '/src/snippets/autoregressive/llama4-scout-deployment.jsx';
import { Llama4MaverickDeployment } from '/src/snippets/autoregressive/llama4-maverick-deployment.jsx';
## 1. Model Introduction
[Llama 4](https://github.com/meta-llama/llama-models/blob/main/models/llama4/MODEL_CARD.md) is Meta's latest generation of open-source LLM model with industry-leading performance.
SGLang has supported Llama 4 Scout (109B) and Llama 4 Maverick (400B) since [v0.4.5](https://github.com/sgl-project/sglang/releases/tag/v0.4.5).
Ongoing optimizations are tracked in the [Roadmap](https://github.com/sgl-project/sglang/issues/5118).
This generation delivers comprehensive upgrades across the board:
The highly capable Llama 4 Maverick with 17B active parameters out of ~400B total, with 128 experts.
The efficient Llama 4 Scout also has 17B active parameters out of ~109B total, using just 16 experts.
Both models leverage early fusion for native multimodality, enabling them to process text and image inputs. Maverick and Scout are both trained on up to 40 trillion tokens on data encompassing 200 languages (with specific fine-tuning support for 12 languages including Arabic, Spanish, German, and Hindi).
For more details, please refer to the official llama4 Repository:https://www.llama.com/models/llama-4/
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
<Llama4ScoutDeployment />
<Llama4MaverickDeployment />
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
### 4.2 Advanced Usage
#### 4.2.1 Launch the docker
```shell Command
docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x
```
```shell Command
docker run -d -it --ipc=host --network=host --privileged \
--cap-add=CAP_SYS_ADMIN \
--device=/dev/kfd --device=/dev/dri --device=/dev/mem \
--group-add video --cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
-v /:/work \
-e SHELL=/bin/bash \
--name Llama4 \
lmsysorg/sglang:v0.5.9-rocm720-mi30x \
/bin/bash
```
#### 4.2.2 Launch the server
### Llama-4-Scout
8-GPU deployment command:
```bash Command
sglang serve \
--model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
--tp 8 \
--context-length 1000000 \
--trust-remote-code
```
### Llama-4-Maverick
8-GPU deployment command:
```bash Command
sglang serve \
--model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
--tp 8 \
--context-length 1000000 \
--trust-remote-code
```
## 5. Benchmark
### 5.1 Speed Benchmark
Test Environment:
Hardware: AMD MI300x GPU
Model: Llama-4-Scout
Tensor Parallelism: 8
sglang version: 0.5.9
- **Model Deployment**
```bash Command
sglang serve \
--model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
--tp 8 \
--context-length 1000000 \
--trust-remote-code
```
### 5.1.1 Low Concurrency (Latency-Optimized)
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model meta-llama/Llama-4-Scout-17B-16E-Instruct \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 74.62
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4211
Request throughput (req/s): 0.14
Input token throughput (tok/s): 82.88
Output token throughput (tok/s): 57.42
Peak output token throughput (tok/s): 146.00
Peak concurrent requests: 2
Total token throughput (tok/s): 140.20
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 7459.48
Median E2E Latency (ms): 4489.77
---------------Time to First Token----------------
Mean TTFT (ms): 4246.98
Median TTFT (ms): 68.57
P99 TTFT (ms): 48091.05
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 7.49
Median TPOT (ms): 7.40
P99 TPOT (ms): 7.40
---------------Inter-Token Latency----------------
Mean ITL (ms): 7.49
Median ITL (ms): 7.49
P95 ITL (ms): 7.47
P99 ITL (ms): 7.52
Max ITL (ms): 10.44
==================================================
```
### 5.1.2 Medium Concurrency (Balanced)
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model meta-llama/Llama-4-Scout-17B-16E-Instruct \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 45.41
Total input tokens: 49668
Total input text tokens: 49668
Total input vision tokens: 0
Total generated tokens: 40805
Total generated tokens (retokenized): 40516
Request throughput (req/s): 2.26
Input token throughput (tok/s): 1120.46
Output token throughput (tok/s): 1152.47
Peak output token throughput (tok/s): 1520.00
Peak concurrent requests: 21
Total token throughput (tok/s): 2272.84
Concurrency: 14.76
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 6089.22
Median E2E Latency (ms): 6568.80
---------------Time to First Token----------------
Mean TTFT (ms): 124.44
Median TTFT (ms): 87.42
P99 TTFT (ms): 268.72
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 11.88
Median TPOT (ms): 12.00
P99 TPOT (ms): 15.49
---------------Inter-Token Latency----------------
Mean ITL (ms): 11.72
Median ITL (ms): 10.54
P95 ITL (ms): 11.22
P99 ITL (ms): 67.88
Max ITL (ms): 74.05
==================================================
```
### 5.1.3 High Concurrency (Throughput-Optimized)
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model meta-llama/Llama-4-Scout-17B-16E-Instruct \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 85.84
Total input tokens: 249841
Total input text tokens: 249841
Total input vision tokens: 0
Total generated tokens: 252662
Total generated tokens (retokenized): 250498
Request throughput (req/s): 5.84
Input token throughput (tok/s): 2910.84
Output token throughput (tok/s): 2944.82
Peak output token throughput (tok/s): 4100.00
Peak concurrent requests: 110
Total token throughput (tok/s): 5854.65
Concurrency: 92.24
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 15844.00
Median E2E Latency (ms): 15262.56
---------------Time to First Token----------------
Mean TTFT (ms): 204.46
Median TTFT (ms): 129.96
P99 TTFT (ms): 528.54
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 41.56
Median TPOT (ms): 42.90
P99 TPOT (ms): 47.48
---------------Inter-Token Latency----------------
Mean ITL (ms): 40.99
Median ITL (ms): 24.46
P95 ITL (ms): 84.46
P99 ITL (ms): 87.64
Max ITL (ms): 226.06
==================================================
```
### 5.2 Speed Benchmark
Test Environment:
Hardware: AMD MI300x GPU
Model: Llama-4-Maverick
Tensor Parallelism: 8
sglang version: 0.5.9
- **Model Deployment**
```bash Command
sglang serve \
--model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
--tp 8 \
--context-length 1000000 \
--trust-remote-code
```
### 5.2.1 Low Concurrency (Latency-Optimized)
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model meta-llama/Llama-4-Maverick-17B-128E-Instruct \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 68.08
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4202
Request throughput (req/s): 0.15
Input token throughput (tok/s): 89.62
Output token throughput (tok/s): 61.99
Peak output token throughput (tok/s): 168.00
Peak concurrent requests: 2
Total token throughput (tok/s): 151.61
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 6805.62
Median E2E Latency (ms): 2733.91
---------------Time to First Token----------------
Mean TTFT (ms): 4296.56
Median TTFT (ms): 57.45
P99 TTFT (ms): 38633.95
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 5.95
Median TPOT (ms): 5.96
P99 TPOT (ms): 5.97
---------------Inter-Token Latency----------------
Mean ITL (ms): 5.96
Median ITL (ms): 5.96
P95 ITL (ms): 6.02
P99 ITL (ms): 6.08
Max ITL (ms): 7.02
==================================================
```
### 5.2.2 Medium Concurrency (Balanced)
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model meta-llama/Llama-4-Maverick-17B-128E-Instruct \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 30.72
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 40805
Total generated tokens (retokenized): 40923
Request throughput (req/s): 2.60
Input token throughput (tok/s): 1291.39
Output token throughput (tok/s): 1328.41
Peak output token throughput (tok/s): 1760.00
Peak concurrent requests: 22
Total token throughput (tok/s): 2619.80
Concurrency: 13.92
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 5345.15
Median E2E Latency (ms): 5679.73
---------------Time to First Token----------------
Mean TTFT (ms): 259.30
Median TTFT (ms): 72.60
P99 TTFT (ms): 1063.45
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 10.53
Median TPOT (ms): 10.22
P99 TPOT (ms): 20.27
---------------Inter-Token Latency----------------
Mean ITL (ms): 9.99
Median ITL (ms): 9.10
P95 ITL (ms): 9.87
P99 ITL (ms): 55.62
Max ITL (ms): 868.54
==================================================
```
### 5.2.3 High Concurrency (Throughput-Optimized)
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model meta-llama/Llama-4-Maverick-17B-128E-Instruct \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 90.95
Total input tokens: 249831
Total input text tokens: 249831
Total input vision tokens: 0
Total generated tokens: 252662
Total generated tokens (retokenized): 251625
Request throughput (req/s): 5.50
Input token throughput (tok/s): 2746.77
Output token throughput (tok/s): 2777.90
Peak output token throughput (tok/s): 3700.00
Peak concurrent requests: 109
Total token throughput (tok/s): 5524.67
Concurrency: 93.04
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 16924.17
Median E2E Latency (ms): 16294.85
---------------Time to First Token----------------
Mean TTFT (ms): 188.19
Median TTFT (ms): 128.96
P99 TTFT (ms): 534.81
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 33.63
Median TPOT (ms): 35.37
P99 TPOT (ms): 38.26
---------------Inter-Token Latency----------------
Mean ITL (ms): 33.19
Median ITL (ms): 27.66
P95 ITL (ms): 76.91
P99 ITL (ms): 78.82
Max ITL (ms): 268.17
==================================================
```
### 5.3 Accuracy Benchmark
#### 5.3.1 GSM8K Benchmark
- **Benchmark Command:**
```shell Command
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
```
- Llama-4-Scout-17B-16E-Instruct
```text Output
Accuracy: 0.945
Invalid: 0.000
Latency: 12.731 s
Output throughput: 1595.418 token/s
```
- Llama-4-Maverick-17B-128E-Instruct
```text Output
Accuracy: 0.895
Invalid: 0.000
Latency: 9.739 s
Output throughput: 2405.505 token/s
```
docs_new/cookbook/autoregressive/MiniMax/MiniMax-M2.5.mdx
+1041
View File
File diff suppressed because it is too large Load Diff
docs_new/cookbook/autoregressive/MiniMax/MiniMax-M2.7.mdx
+723
View File
@@ -0,0 +1,723 @@
---
title: MiniMax-M2.7
metatags:
description: "Deploy MiniMax-M2.7 with SGLang on NVIDIA and AMD GPUs — model self-evolution, professional software engineering, and native agent teams."
tag: NEW
---
## 1. Model Introduction
[MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7) is MiniMax's first model deeply participating in its own evolution. Built for real-world productivity, M2.7 excels at building complex agent harnesses and completing highly elaborate productivity tasks, leveraging Agent Teams, complex Skills, and dynamic tool search.
Key highlights:
- **Model Self-Evolution**: During development, M2.7 updates its own memory, builds complex skills for RL experiments, and improves its own learning process. An internal version autonomously optimized a programming scaffold over 100+ rounds, achieving a **30% performance improvement**. On MLE Bench Lite, M2.7 achieved a **66.6% medal rate**.
- **Professional Software Engineering**: Delivers outstanding real-world programming capabilities. On SWE-Pro, M2.7 achieved **56.22%**, with strong results on SWE Multilingual (76.5) and Multi SWE Bench (52.7). On Terminal Bench 2 (57.0%) and NL2Repo (39.8%), M2.7 demonstrates deep understanding of complex engineering systems.
- **Professional Work**: Achieved an ELO score of **1495** on GDPval-AA (highest among open-source models). On Toolathon, M2.7 reached **46.3%** accuracy (global top tier).
- **Native Agent Teams**: Supports multi-agent collaboration with stable role identity and autonomous decision-making.
For more details, see the [official MiniMax-M2.7 blog post](https://www.minimax.io/news/minimax-m27-en).
**License**: [Modified-MIT (MiniMax Model License)](https://github.com/MiniMax-AI/MiniMax-M2.7/blob/main/LICENSE)
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
**Docker Images by Hardware Platform:**
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Hardware Platform</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Docker Image</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>NVIDIA A100 / H100 / H200 / B200</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.10.post1`</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>NVIDIA B300 / GB300</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.10.post1-cu130`</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AMD MI300X / MI325X</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.10.post1-rocm720-mi30x`</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AMD MI355X</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.10.post1-rocm720-mi35x`</td>
</tr>
</tbody>
</table>
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, deployment strategy, and feature capabilities.
import { MiniMaxM27Deployment } from '/src/snippets/autoregressive/minimax-m27-deployment.jsx'
<MiniMaxM27Deployment />
### 3.2 Configuration Tips
**Key Parameters:**
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Recommended Value</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--tool-call-parser`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Tool call parser for function calling support</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`minimax-m2`</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Reasoning parser for thinking mode</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`minimax-append-think`</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--trust-remote-code`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Required for MiniMax model loading</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Always enabled</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--mem-fraction-static`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Static memory fraction for KV cache</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`0.85`</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--tp`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Tensor parallelism size</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`2` / `4` / `8` depending on hardware</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--ep`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Expert parallelism size</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`8` (NVIDIA 8-GPU) or EP=TP (AMD)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--kv-cache-dtype`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>KV cache data type (AMD only)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`fp8_e4m3`</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--attention-backend`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Attention backend (AMD only)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`triton`</td>
</tr>
</tbody>
</table>
**Hardware Requirements: NVIDIA**
- **4-GPU deployment**: Requires 4× high-memory GPUs (e.g., H200, B200, A100, H100) with TP=4
- **8-GPU deployment**: Requires 8× GPUs (e.g., H200, B200, A100, H100) with TP=8 and EP=8
**Hardware Requirements: NVIDIA GB300**
- **2-GPU deployment**: GB300 (275GB per die) can host the model with TP=2
- **4-GPU deployment**: Maximum single-node TP for GB300, recommended for higher throughput
**Hardware Requirements: AMD**
- **2-GPU deployment**: Requires 2× high-memory GPUs (e.g., MI300X, MI325X, MI355X) with TP=2, EP=2
- **4-GPU deployment**: Requires 4× GPUs (e.g., MI300X, MI325X, MI355X) with TP=4, EP=4
- **8-GPU deployment**: Requires 8× GPUs (e.g., MI300X, MI325X, MI355X) with TP=8, EP=8
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
**Deployment Command:**
```bash Command
sglang serve \
--model-path MiniMaxAI/MiniMax-M2.7 \
--tp 4 \
--tool-call-parser minimax-m2 \
--reasoning-parser minimax-append-think \
--trust-remote-code \
--mem-fraction-static 0.85
```
**Testing Deployment:**
After startup, you can test the SGLang OpenAI-compatible API with the following command:
```bash Command
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "MiniMaxAI/MiniMax-M2.7",
"messages": [
{"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]},
{"role": "user", "content": [{"type": "text", "text": "Who won the world series in 2020?"}]}
]
}'
```
**Simple Completion Example:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="MiniMaxAI/MiniMax-M2.7",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Who won the world series in 2020?"}
],
max_tokens=1024
)
print(response.choices[0].message.content)
```
**Example Output**:
```text Output
<think>The user asks: "Who won the World Series in 2020?" That's a simple factual question. The answer: the Los Angeles Dodgers won the 2020 MLB World Series, defeating the Tampa Bay Rays. So answer accordingly.
We must be mindful of policy: it's a factual question about sports. It's allowed. Provide answer with brief context.
We should answer concisely.
Hence final answer: The Los Angeles Dodgers won the 2020 World Series, defeating the Tampa Bay Rays in six games (best-of-seven series). Possibly mention it was played at a neutral site due to COVID-19, at Globe Life Field in Arlington, Texas.
We must avoid disallowed content, no issue.
Thus final.
</think>
The **Los Angeles Dodgers** won the 2020 World Series. They defeated the **Tampa Bay Rays** in six games (4‑2) in a best‑of‑seven series that was played at Globe Life Field in Arlington, Texas, under the MLB bubble‑like arrangements for the COVID‑19 pandemic.
```
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
MiniMax-M2.7 supports Thinking mode. Enable the reasoning parser during deployment to separate the thinking and the content sections:
```bash Command
sglang serve \
--model-path MiniMaxAI/MiniMax-M2.7 \
--tp 4 \
--reasoning-parser minimax-append-think \
--trust-remote-code \
--mem-fraction-static 0.85
```
**Streaming with Thinking Process**
With `minimax-append-think`, the thinking content is wrapped in `<think>...</think>` tags within the `content` field. You can parse these tags on the client side to separate the thinking and content sections:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="MiniMaxAI/MiniMax-M2.7",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
max_tokens=2048,
stream=True
)
# Process the stream, separating <think>...</think> from content
in_think = False
think_printed_header = False
content_printed_header = False
buffer = ""
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if delta.content:
buffer += delta.content
while buffer:
if in_think:
# Look for closing </think> tag
end_idx = buffer.find("</think>")
if end_idx != -1:
print(buffer[:end_idx], end="", flush=True)
buffer = buffer[end_idx + len("</think>"):]
in_think = False
else:
# Still in thinking, print what we have
print(buffer, end="", flush=True)
buffer = ""
else:
# Look for opening <think> tag
start_idx = buffer.find("<think>")
if start_idx != -1:
# Print any content before <think>
before = buffer[:start_idx]
if before:
if not content_printed_header:
print("=============== Content =================", flush=True)
content_printed_header = True
print(before, end="", flush=True)
buffer = buffer[start_idx + len("<think>"):]
in_think = True
if not think_printed_header:
print("=============== Thinking =================", flush=True)
think_printed_header = True
else:
# No <think> tag, print as content
if not content_printed_header and think_printed_header:
print("\n=============== Content =================", flush=True)
content_printed_header = True
print(buffer, end="", flush=True)
buffer = ""
print()
```
**Output Example:**
```text Output
=============== Thinking =================
The user asks: "Solve this problem step by step: What is 15% of 240?" Straightforward. Provide solution: 15% = 15/100 = 0.15. Multiply 240 * 0.15 = 36. Show steps. So answer: 36. Provide explanation.
But also ensure we follow any policy? No issues. Just straightforward.
I'll provide a step-by-step solution.
Also could show fraction: 15% = 15/100 = 3/20, multiply 240 * 3/20 = (240/20)*3 = 12*3 = 36.
Yes. Provide final answer. Also show verification: 10% of 240 is 24, 5% is 12, total 36.
All good.
=============== Content =================
**Step‑by‑step solution**
1. **Convert the percent to a decimal (or a fraction).**
15% = 15/100 = 0.15 = 3/20
2. **Multiply the original number (240) by this decimal/fraction.**
Using the decimal:
240 × 0.15 = 36
Or using the fraction:
240 × 3/20 = (240/20) × 3 = 12 × 3 = 36
3. **Result:**
15% of 240 = **36**
*Check:*
- 10% of 240 = 24
- 5% of 240 = 12
- Adding them: 24 + 12 = 36, which matches the calculation.
```
**Note:** The `minimax-append-think` reasoning parser embeds the thinking process in `<think>...</think>` tags within the `content` field. The code above parses these tags in real-time to display thinking and content separately.
#### 4.2.2 Tool Calling
MiniMax-M2.7 supports tool calling capabilities. Enable the tool call parser:
```bash Command
sglang serve \
--model-path MiniMaxAI/MiniMax-M2.7 \
--tp 4 \
--tool-call-parser minimax-m2 \
--reasoning-parser minimax-append-think \
--trust-remote-code \
--mem-fraction-static 0.85
```
**Python Example:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Non-streaming request
response = client.chat.completions.create(
model="MiniMaxAI/MiniMax-M2.7",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools
)
message = response.choices[0].message
# Check for tool calls
if message.tool_calls:
for tool_call in message.tool_calls:
print(f"Tool Call: {tool_call.function.name}")
print(f" Arguments: {tool_call.function.arguments}")
else:
print(message.content)
```
**Output Example**:
```text Output
Tool Call: get_weather
Arguments: {"location": "Beijing"}
```
**Handling Tool Call Results:**
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="MiniMaxAI/MiniMax-M2.7",
messages=messages
)
print(final_response.choices[0].message.content)
```
**Output Example:**
```text Output
The weather in Beijing is currently 22°C and sunny.
```
## 5. Benchmark
This section uses **industry-standard configurations** for comparable benchmark results.
**Test Environment**:
- Hardware: 2× NVIDIA GB300 (275GB per die)
- Docker Image: `lmsysorg/sglang:v0.5.10.post1-cu130`
- Model: MiniMax-M2.7 (FP8)
- Tensor Parallelism: 2
- SGLang version: 0.5.10.post1
### 5.1 Accuracy Benchmark
**Evaluation Tool**: [NVIDIA NeMo-Skills](https://github.com/NVIDIA-NeMo/Skills)
**Evaluation Settings**: temperature=0.6, top_p=0.95, 8 seeds, max_tokens=120,000, `parse_reasoning=True`
#### 5.1.1 GPQA Diamond
- Dataset: [GPQA Diamond](https://huggingface.co/datasets/Idavidrein/gpqa) (198 questions)
- Prompt: `eval/aai/mcq-4choices` (4-choice multiple choice, matching [Artificial Analysis methodology](https://artificialanalysis.ai/methodology/intelligence-benchmarking))
- Evaluation command:
```bash Command
ns prepare_data gpqa
ns eval \
--cluster=local \
--server_type=openai \
--model=MiniMaxAI/MiniMax-M2.7 \
--server_address=http://localhost:30000/v1 \
--output_dir=./m2.7-eval/ \
--benchmarks=gpqa:8 \
++prompt_config=eval/aai/mcq-4choices \
++inference.tokens_to_generate=120000 \
++inference.temperature=0.6 \
++inference.top_p=0.95 \
++parse_reasoning=True
```
- Test Results:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Evaluation Mode</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Accuracy</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>No Answer</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@1 (avg-of-8)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>84.91%</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>3.54%</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**majority@8**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>**88.89%**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.00%</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@8</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>96.46%</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.00%</td>
</tr>
</tbody>
</table>
#### 5.1.2 AIME 2025
- Dataset: AIME 2025 (30 problems)
- Prompt: `generic/math` (boxed answer format)
- Evaluation command:
```bash Command
ns prepare_data aime25
ns eval \
--cluster=local \
--server_type=openai \
--model=MiniMaxAI/MiniMax-M2.7 \
--server_address=http://localhost:30000/v1 \
--output_dir=./m2.7-eval/ \
--benchmarks=aime25:8 \
++inference.tokens_to_generate=120000 \
++inference.temperature=0.6 \
++inference.top_p=0.95 \
++parse_reasoning=True
```
- Test Results:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Evaluation Mode</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Accuracy</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>No Answer</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@1 (avg-of-8)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>92.50% ± 5.56%</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>2.92%</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**majority@8**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>**97.08%**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.00%</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@8</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>100.00%</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.00%</td>
</tr>
</tbody>
</table>
#### 5.1.3 MMLU-Pro
- Dataset: [MMLU-Pro](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) (12,032 questions, 10-choice)
- Prompt: `eval/aai/mcq-10choices` (10-choice multiple choice)
- Evaluation command:
```bash Command
ns prepare_data mmlu-pro
ns eval \
--cluster=local \
--server_type=openai \
--model=MiniMaxAI/MiniMax-M2.7 \
--server_address=http://localhost:30000/v1 \
--output_dir=./m2.7-eval/ \
--benchmarks=mmlu-pro \
++prompt_config=eval/aai/mcq-10choices \
++inference.tokens_to_generate=32768 \
++inference.temperature=0.0 \
++parse_reasoning=True
```
- Test Results:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Evaluation Mode</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Accuracy</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>No Answer</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@1 (greedy)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>69.41%</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>18.75%</td>
</tr>
</tbody>
</table>
> **Note**: The high no-answer rate is due to the 32K token limit being insufficient for M2.7's extended thinking on some questions. A rerun with 120K tokens is expected to improve accuracy significantly.
#### 5.1.4 GSM8K Benchmark
- Benchmark Method: 8-shot Chain-of-Thought, evaluated via OpenAI-compatible API
- Test Results:
```text Output
GSM8K Results (8-shot CoT)
Model: MiniMaxAI/MiniMax-M2.7
Total: 1319
Correct: 1218
Accuracy: 92.34%
```
### 5.2 Speed Benchmark
#### 5.2.1 Low Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model MiniMaxAI/MiniMax-M2.7 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 34.33
Total input tokens: 6101
Total generated tokens: 4220
Request throughput (req/s): 0.29
Input token throughput (tok/s): 177.71
Output token throughput (tok/s): 122.92
Total token throughput (tok/s): 300.63
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 3431.21
Median E2E Latency (ms): 2742.57
---------------Time to First Token----------------
Mean TTFT (ms): 50.28
Median TTFT (ms): 53.85
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 8.02
Median TPOT (ms): 8.01
---------------Inter-Token Latency----------------
Mean ITL (ms): 8.03
Median ITL (ms): 8.02
==================================================
```
#### 5.2.2 High Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model MiniMaxAI/MiniMax-M2.7 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 100.20
Total input tokens: 249831
Total generated tokens: 252662
Request throughput (req/s): 4.99
Input token throughput (tok/s): 2493.41
Output token throughput (tok/s): 2521.66
Total token throughput (tok/s): 5015.07
Concurrency: 90.19
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 18072.69
Median E2E Latency (ms): 17761.84
---------------Time to First Token----------------
Mean TTFT (ms): 247.94
Median TTFT (ms): 92.05
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 35.75
Median TPOT (ms): 36.67
---------------Inter-Token Latency----------------
Mean ITL (ms): 35.34
Median ITL (ms): 30.55
==================================================
```
docs_new/cookbook/autoregressive/MiniMax/MiniMax-M2.mdx
+541
View File
@@ -0,0 +1,541 @@
---
title: MiniMax-M2
metatags:
description: "Deploy MiniMax-M2 with SGLang - community contribution guide for MiniMax M2 model deployment."
---
import { MiniMaxM2Deployment } from '/src/snippets/autoregressive/minimax-m2-deployment.jsx';
## 1. Model Introduction
[MiniMax-M2](https://huggingface.co/MiniMaxAI/MiniMax-M2) is a compact, fast, and cost-effective MoE model (230 billion total parameters with 10 billion active parameters) built for elite performance in coding and agentic tasks, all while maintaining powerful general intelligence.
This generation delivers comprehensive upgrades across the board:
- **Superior Intelligence**: MiniMax-M2 demonstrates highly competitive general intelligence across mathematics, science, instruction following, coding, and agentic tool use in [Artificial Analysis](https://artificialanalysis.ai/). Its composite score ranks #1 among open-source models globally.
- **Advanced Coding**: Engineered for end-to-end developer workflows, MiniMax-M2 excels at multi-file edits, coding-run-fix loops, and test-validated repairs. Strong performance on Terminal-Bench and (Multi-)SWE-Bench–style tasks demonstrates practical effectiveness in terminals, IDEs, and CI across languages.
- **Agent Performance**: MiniMax-M2 plans and executes complex, long-horizon toolchains across shell, browser, retrieval, and code runners. In BrowseComp-style evaluations, it consistently locates hard-to-surface sources, maintains evidence traceable, and gracefully recovers from flaky steps.
- **Efficient Design**: With 10 billion activated parameters (230 billion in total), MiniMax-M2 delivers lower latency, lower cost, and higher throughput for interactive agents and batched sampling—perfectly aligned with the shift toward highly deployable models that still shine on coding and agentic tasks.
For more details, please refer to the [official Minimax GitHub Repository](https://github.com/MiniMax-AI).
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions. The AMD environment is currently available in SGLang via Docker image install.
### 2.1 AMD Docker
#### 2.1.1 Launch docker
```shell Command
docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x
```
```shell Command
docker run -d -it --ipc=host --network=host --privileged \
--cap-add=CAP_SYS_ADMIN \
--device=/dev/kfd --device=/dev/dri --device=/dev/mem \
--group-add video --cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
-v /:/work \
-e SHELL=/bin/bash \
--name Minimax \
lmsysorg/sglang:v0.5.9-rocm720-mi30x \
/bin/bash
```
#### 2.1.2 Make modifications inside the docker
```shell Command
mv /sgl-workspace/sglang/python/sglang/srt/models/transformers.py \
/sgl-workspace/sglang/python/sglang/srt/models/hf_transformers_model.py
```
#### 2.1.3 Fix torch compile
Comment out the following line: @torch.compile(dynamic=True, backend=get_compiler_backend()) in /sgl-workspace/sglang/python/sglang/srt/models/minimax_m2.py
```shell Command
#@torch.compile(dynamic=True, backend=get_compiler_backend())
```
## 3. Model Deployment
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
<MiniMaxM2Deployment />
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
Server Command:
```shell Command
sglang serve \
--model-path MiniMaxAI/MiniMax-M2 \
--tp-size 4 \
--reasoning-parser minimax-append-think \
--trust-remote-code \
--mem-fraction-static 0.85
```
Test Code:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="MiniMaxAI/MiniMax-M2",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.6,
max_tokens=2048,
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
Output Example:
```text Output
<think>First, the user asks: "What is 15% of 240?" This is a straightforward percentage calculation. I need to solve it step by step as per the instruction.
The problem is: What is 15% of 240?
To find a percentage of a number, I multiply the number by the percentage divided by 100. So, 15% is 15/100, which simplifies to 0.15.
Therefore, 15% of 240 is 240 times 0.15.
Let me calculate that: 240 × 0.15.
I can break it down: 240 × 0.15 = 240 × (15/100) = (240 × 15) / 100.
Now, 240 × 15. 200 × 15 = 3000, and 40 × 15 = 600, so total 3000 + 600 = 3600.
Then, divide by 100: 3600 / 100 = 36.
So, 15% of 240 is 36.
I should confirm this with another method. For example, 10% of 240 is 24, and 5% is half of that, which is 12. Then 15% is 10% + 5% = 24 + 12 = 36. Same answer.
Or, using fractions: 15% = 3/20, so 240 × 3/20 = (240 / 20) × 3 = 12 × 3 = 36.
All methods confirm it's 36.
The user said "solve this problem step by step," so I should present the steps clearly.
Step 1: Understand that "15% of 240" means 15 per hundred of 240.
Step 2: Convert percentage to decimal: 15% = 15/100 = 0.15.
Step 3: Multiply the number by the decimal: 240 × 0.15.
Step 4: Calculate the multiplication: 240 × 0.15 = 36.
Step 5: Therefore, 15% of 240 is 36.
I should also mention that percentage means per hundred, so it's straightforward.
Finally, I need to box the answer as per the instruction.
So, the final answer is 36.
</think>
To find 15% of 240, follow these steps:
1. **Understand the percentage**: "15%" means 15 per hundred, or 15/100.
2. **Convert to a decimal**: 15/100 = 0.15.
3. **Multiply by the number**: 240 × 0.15.
4. **Calculate the result**:
- 240 × 0.15 = 36.
Alternatively, you can break it down:
- 10% of 240 is 24 (since 240 ÷ 10 = 24).
- 5% of 240 is half of 10%, which is 12.
- Therefore, 15% is 10% + 5% = 24 + 12 = 36.
Both methods confirm the result.
**Answer**: 36
```
### 4.2.2 Tool Calling
Server Command:
```shell Command
sglang serve \
--model-path MiniMaxAI/MiniMax-M2 \
--tp-size 4 \
--tool-call-parser minimax-m2 \
--trust-remote-code \
--mem-fraction-static 0.85
```
Test Code:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="MiniMaxAI/MiniMax-M2",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Accumulate tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================\n", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {
'name': None,
'arguments': ''
}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]['name'] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
# Print content
if delta.content:
print(delta.content, end="", flush=True)
# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"🔧 Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
print()
```
Output Example:
```text Output
Alright, the user is asking about the weather in Beijing. This is a straightforward request that I can help with using the get_weather tool that's available to me.
Let me think about what I need to do here. The user wants to know the current weather conditions in Beijing, which is the capital city of China. To provide this information, I need to use the get_weather tool that's been provided to me.
Looking at the tool's parameters, I can see it requires:
1. location - which is required and should be a string representing the city name
2. unit - which is optional and can be either "celsius" or "fahrenheit"
For the location parameter, I'll use "Beijing" since that's what the user asked about.
For the unit parameter, the user didn't specify their preference between celsius and fahrenheit. Since Beijing is in China, which primarily uses celsius, and celsius is the more standard unit internationally, I'll default to celsius. If the user wants the temperature in fahrenheit instead, they can ask in a follow-up message and I can provide that information.
So I need to make a tool call to get_weather with the following parameters:
- location: "Beijing"
- unit: "celsius"
This should return the current weather information for Beijing, which I can then share with the user. I'll format my response using the required XML tags for tool calls as specified in my instructions.
</think>
🔧 Tool Call: get_weather
Arguments: {"location": "Beijing", "unit": "celsius"}
```
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment**:
- Hardware: AMD MI300X GPU(4x)
- Model: MiniMax-M2
- Tensor Parallelism: 4
- sglang version: 0.5.7
**Model Deployment**:
```bash Command
sglang serve \
--model-path MiniMaxAI/MiniMax-M2 \
--tp-size 4 \
--trust-remote-code \
--mem-fraction-static 0.85
```
### 5.1.1 Low Concurrency (Latency-Optimized)
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model MiniMaxAI/MiniMax-M2 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 138.91
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4220
Request throughput (req/s): 0.07
Input token throughput (tok/s): 43.92
Output token throughput (tok/s): 30.38
Peak output token throughput (tok/s): 46.00
Peak concurrent requests: 2
Total token throughput (tok/s): 74.30
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 13887.62
Median E2E Latency (ms): 10377.26
---------------Time to First Token----------------
Mean TTFT (ms): 4528.94
Median TTFT (ms): 385.23
P99 TTFT (ms): 38338.51
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 22.21
Median TPOT (ms): 22.24
P99 TPOT (ms): 22.25
---------------Inter-Token Latency----------------
Mean ITL (ms): 22.23
Median ITL (ms): 22.24
P95 ITL (ms): 22.35
P99 ITL (ms): 22.41
Max ITL (ms): 23.64
==================================================
```
### 5.1.2 Medium Concurrency (Balanced)
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model MiniMaxAI/MiniMax-M2 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 81.07
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 40805
Total generated tokens (retokenized): 40803
Request throughput (req/s): 0.99
Input token throughput (tok/s): 489.29
Output token throughput (tok/s): 503.32
Peak output token throughput (tok/s): 704.00
Peak concurrent requests: 19
Total token throughput (tok/s): 992.61
Concurrency: 13.74
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 13925.95
Median E2E Latency (ms): 14348.75
---------------Time to First Token----------------
Mean TTFT (ms): 532.32
Median TTFT (ms): 147.69
P99 TTFT (ms): 1978.48
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 27.49
Median TPOT (ms): 26.56
P99 TPOT (ms): 46.52
---------------Inter-Token Latency----------------
Mean ITL (ms): 26.31
Median ITL (ms): 23.47
P95 ITL (ms): 24.37
P99 ITL (ms): 125.10
Max ITL (ms): 1192.51
==================================================
```
### 5.1.3 High Concurrency (Throughput-Optimized)
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model MiniMaxAI/MiniMax-M2 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 153.71
Total input tokens: 249831
Total input text tokens: 249831
Total input vision tokens: 0
Total generated tokens: 252662
Total generated tokens (retokenized): 250982
Request throughput (req/s): 3.25
Input token throughput (tok/s): 1625.33
Output token throughput (tok/s): 1643.75
Peak output token throughput (tok/s): 2597.00
Peak concurrent requests: 107
Total token throughput (tok/s): 3269.09
Concurrency: 91.14
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 28017.24
Median E2E Latency (ms): 26865.28
---------------Time to First Token----------------
Mean TTFT (ms): 387.41
Median TTFT (ms): 183.90
P99 TTFT (ms): 1192.44
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 55.23
Median TPOT (ms): 57.84
P99 TPOT (ms): 70.23
---------------Inter-Token Latency----------------
Mean ITL (ms): 54.79
Median ITL (ms): 39.01
P95 ITL (ms): 143.10
P99 ITL (ms): 150.46
Max ITL (ms): 986.14
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 GSM8K Benchmark
- **Server Command**:
```shell Command
sglang serve \
--model-path MiniMaxAI/MiniMax-M2 \
--tp-size 4 \
--trust-remote-code \
--mem-fraction-static 0.85
```
- **Benchmark Command**:
```shell Command
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
```
- **Result**:
- MiniMax-M2
```text Output
Accuracy: 0.950
Invalid: 0.000
Latency: 15.120 s
Output throughput: 1306.711 token/s
```
docs_new/cookbook/autoregressive/Mistral/Devstral-2.mdx
+518
View File
@@ -0,0 +1,518 @@
---
title: Devstral 2 (Mistral)
metatags:
description: "Deploy Devstral 2 agentic coding models with SGLang - optimized for tool use, codebase exploration, and multi-file edits with 256K context."
---
## 1. Model Introduction
**Devstral 2** is an agentic LLM family for software engineering tasks. It is designed for agentic workflows such as tool use, codebase exploration, and multi-file edits, and achieves strong performance on **SWE-bench**.
The **Devstral 2 Instruct** checkpoints are instruction-tuned **FP8** models, making them a good fit for chat, tool-using agents, and instruction-following SWE workloads.
**Key Features:**
- **Agentic coding**: Optimized for tool-driven coding and software engineering agents
- **Improved performance**: A step up compared to earlier Devstral models
- **Better generalization**: More robust across diverse prompts and coding environments
- **Long context**: Up to a **256K** context window
**Use Cases:**
AI code assistants, agentic coding, and software engineering tasks that require deep codebase understanding and tool integration.
For enterprises requiring specialized capabilities (increased context, domain-specific knowledge, etc.), please reach out to Mistral.
**Models:**
- **Collection**: [mistralai/devstral-2 (Hugging Face)](https://huggingface.co/collections/mistralai/devstral-2)
- **FP8 Instruct**:
- **[mistralai/Devstral-2-123B-Instruct-2512](https://huggingface.co/mistralai/Devstral-2-123B-Instruct-2512)**
- **[mistralai/Devstral-Small-2-24B-Instruct-2512](https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512)**
---
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
<Warning title="Transformers version requirement">
Devstral 2 requires a recent `transformers`. Please verify `transformers >= 5.0.0.rc`:
```shell Command
python -c "import transformers; print(transformers.__version__)"
```
If your version is lower, upgrade:
```shell Command
pip install -U --pre "transformers>=5.0.0rc0"
```
</Warning>
---
## 3. Model Deployment
### 3.1 Basic configuration
**Interactive Command Generator**: Use the configuration selector below to generate a launch command for Devstral Small 2 (24B) or Devstral 2 (123B).
<Note>
The TP size is set to the minimum required for the selected model size.
</Note>
import { Devstral2Deployment } from "/src/snippets/autoregressive/devstral-2-deployment.jsx";
<Devstral2Deployment />
### 3.2 Configuration tips
- **Context length vs memory**: Devstral 2 advertises a long context window; if you are memory-constrained, start by lowering `--context-length` (for example `32768`) and increase once things are stable.
- **FP8 checkpoints**: Both Devstral Small 2 and Devstral 2 are published as **FP8** weights. If you hit kernel / dtype issues, try a newer SGLang build and recent CUDA drivers.
---
## 4. Model Invocation
### 4.1 Basic Usage (OpenAI-Compatible API)
SGLang exposes an OpenAI-compatible endpoint. Example:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY",
)
resp = client.chat.completions.create(
model="mistralai/Devstral-Small-2-24B-Instruct-2512",
messages=[
{"role": "system", "content": "You are a helpful coding assistant."},
{"role": "user", "content": "Write a Python function that retries a request with exponential backoff."},
],
temperature=0.2,
max_tokens=512,
)
print(resp.choices[0].message.content)
```
**Output Example:**
```text Output
Here's a Python function that implements exponential backoff for retrying a request. This function uses the `requests` library to make HTTP requests and includes error handling for common HTTP and connection errors.
```python
import time
import requests
from requests.exceptions import RequestException
def retry_with_exponential_backoff(
url,
max_retries=3,
initial_delay=1,
backoff_factor=2,
method="GET",
**kwargs
):
"""
Retry a request with exponential backoff.
Parameters:
- url: The URL to request.
- max_retries: Maximum number of retry attempts (default: 3).
- initial_delay: Initial delay in seconds (default: 1).
- backoff_factor: Multiplier for the delay between retries (default: 2).
- method: HTTP method to use (default: "GET").
- **kwargs: Additional arguments to pass to the request function (e.g., headers, data, etc.).
Returns:
- Response object if the request succeeds.
- Raises an exception if all retries fail.
"""
retry_count = 0
delay = initial_delay
while retry_count < max_retries:
try:
response = requests.request(method, url, **kwargs)
# Check if the response status code indicates success
if response.status_code < 400:
return response
else:
raise RequestException(f"HTTP {response.status_code}: {response.text}")
except RequestException as e:
if retry_count == max_retries - 1:
raise Exception(f"All retries failed. Last error: {e}")
print(f"Attempt {retry_count + 1} failed. Retrying in {delay} seconds...")
time.sleep(delay)
...
```
### 4.2 Tool calling (optional)
Devstral 2 supports tool calling capabilities. Enable the tool call parser:
```shell Command
python -m sglang.launch_server \
--model mistralai/Devstral-2-123B-Instruct-2512 \
--tp 2 \
--tool-call-parser mistral
```
**Python Example (with Thinking Process):**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="mistralai/Devstral-2-123B-Instruct-2512",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Accumulate tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================\n", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {
'name': None,
'arguments': ''
}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]['name'] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
# Print content
if delta.content:
print(delta.content, end="", flush=True)
# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"🔧 Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
print()
```
**Output Example:**
```text Output
🔧 Tool Call: get_weather
Arguments: {"location": "Beijing"}
```
## AMD GPU Support
## 1. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 1.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 1.2 Advanced Usage
```shell Command
python3 -m sglang.launch_server \
--model-path mistralai/Devstral-2-123B-Instruct-2512 \
--tp 8 \
--trust-remote-code \
--port 8888
```
## 2.Benchmark
### 5.1 Benchmark Commands
**Scenario 1: Chat (1K/1K) - Most Important**
- **Model Deployment**
```bash Command
python3 -m sglang.launch_server \
--model-path mistralai/Devstral-2-123B-Instruct-2512 \
--tp 8 \
--trust-remote-code \
--port 8888
```
- Low Concurrency (Latency-Optimized)
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model mistralai/Devstral-2-123B-Instruct-2512 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf \
--port 8888
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 94.30
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4206
Request throughput (req/s): 0.11
Input token throughput (tok/s): 64.70
Output token throughput (tok/s): 44.75
Peak output token throughput (tok/s): 82.00
Peak concurrent requests: 2
Total token throughput (tok/s): 109.44
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 9427.59
Median E2E Latency (ms): 5637.23
---------------Time to First Token----------------
Mean TTFT (ms): 4253.85
Median TTFT (ms): 116.95
P99 TTFT (ms): 37764.48
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 12.28
Median TPOT (ms): 12.29
P99 TPOT (ms): 12.30
---------------Inter-Token Latency----------------
Mean ITL (ms): 12.29
Median ITL (ms): 12.29
P95 ITL (ms): 12.38
P99 ITL (ms): 12.42
Max ITL (ms): 12.90
==================================================
```
- Medium Concurrency (Balanced)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model mistralai/Devstral-2-123B-Instruct-2512 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf \
--port 8888
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 52.11
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 40805
Total generated tokens (retokenized): 40761
Request throughput (req/s): 1.54
Input token throughput (tok/s): 761.31
Output token throughput (tok/s): 783.13
Peak output token throughput (tok/s): 1120.00
Peak concurrent requests: 20
Total token throughput (tok/s): 1544.44
Concurrency: 13.60
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 8856.19
Median E2E Latency (ms): 9314.71
---------------Time to First Token----------------
Mean TTFT (ms): 398.80
Median TTFT (ms): 127.81
P99 TTFT (ms): 1500.32
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 17.32
Median TPOT (ms): 16.90
P99 TPOT (ms): 32.78
---------------Inter-Token Latency----------------
Mean ITL (ms): 16.61
Median ITL (ms): 14.26
P95 ITL (ms): 15.07
P99 ITL (ms): 114.46
Max ITL (ms): 1224.45
==================================================
```
- High Concurrency (Throughput-Optimized)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model mistralai/Devstral-2-123B-Instruct-2512 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100 \
--request-rate inf \
--port 8888
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 116.08
Total input tokens: 249831
Total input text tokens: 249831
Total input vision tokens: 0
Total generated tokens: 252662
Total generated tokens (retokenized): 252523
Request throughput (req/s): 4.31
Input token throughput (tok/s): 2152.21
Output token throughput (tok/s): 2176.60
Peak output token throughput (tok/s): 3600.00
Peak concurrent requests: 107
Total token throughput (tok/s): 4328.81
Concurrency: 92.42
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 21456.71
Median E2E Latency (ms): 20126.82
---------------Time to First Token----------------
Mean TTFT (ms): 291.60
Median TTFT (ms): 199.24
P99 TTFT (ms): 866.02
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 42.42
Median TPOT (ms): 45.18
P99 TPOT (ms): 53.32
---------------Inter-Token Latency----------------
Mean ITL (ms): 41.97
Median ITL (ms): 27.59
P95 ITL (ms): 130.43
P99 ITL (ms): 137.87
Max ITL (ms): 616.73
==================================================
```
#### 5.2 Understanding the Results
**Key Metrics:**
- **Request Throughput (req/s)**: Number of requests processed per second
- **Output Token Throughput (tok/s)**: Total tokens generated per second
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
**Why These Configurations Matter:**
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
- **Variable Concurrency**: Captures the Pareto frontier - the optimal trade-off between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
**Interpreting Results:**
- Compare your results against baseline numbers for your hardware
- Higher throughput at same latency = better performance
- Lower TTFT = more responsive user experience
- Lower TPOT = faster generation speed
### 5.3 Accuracy Benchmark
Document model accuracy on standard benchmarks:
#### 5.3.1 GSM8K Benchmark
- Benchmark Command
```bash Command
python3 benchmark/gsm8k/bench_sglang.py \
--num-shots 8 \
--num-questions 1316 \
--parallel 1316 \
--port 8888
```
**Test Results:**
```text Output
Accuracy: 0.922
Invalid: 0.000
Latency: 35.800 s
Output throughput: 4507.697 token/s
```
docs_new/cookbook/autoregressive/Mistral/Ministral-3.mdx
+288
View File
@@ -0,0 +1,288 @@
---
title: Ministral-3
metatags:
description: "Deploy Mistral 3 with SGLang - deployment configurations and usage patterns for Mistral's latest model."
---
import { Ministral3Deployment } from '/src/snippets/autoregressive/ministral-3-deployment.jsx';
## 1. Model Introduction
The largest model in the Ministral 3 family, Ministral 3 14B offers frontier capabilities and performance comparable to its larger Mistral Small 3.2 24B counterpart. A powerful and efficient language model with vision capabilities.
The Ministral 3 14B Instruct model offers the following capabilities:
Vision: Enables the model to analyze images and provide insights based on visual content, in addition to text.
Multilingual: Supports dozens of languages, including English, French, Spanish, German, Italian, Portuguese, Dutch, Chinese, Japanese, Korean, Arabic.
System Prompt: Maintains strong adherence and support for system prompts.
Agentic: Offers best-in-class agentic capabilities with native function calling and JSON outputting.
Edge-Optimized: Delivers best-in-class performance at a small scale, deployable anywhere.
Apache 2.0 License: Open-source license allowing usage and modification for both commercial and non-commercial purposes.
Large Context Window: Supports a 256k context window.
For further details, please refer to the [official documentation](https://github.com/mistralai)
## 2. SGLang Installation
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
<Ministral3Deployment />
### 3.2 Configuration Tips
**Context length vs memory**: Ministral-3 advertises a long context window; if you are memory-constrained, start by lowering --context-length (for example 32768) and increase once things are stable.
**Pre-installation steps**: Adding the following steps after launching the docker
```shell Command
pip install mistral-common --upgrade
pip install transformers==5.0.0.rc0
```
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
### 4.2 Advanced Usage
#### 4.2.1 Launch the docker
```shell Command
docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x
```
```shell Command
docker run -d -it --ipc=host --network=host --privileged \
--cap-add=CAP_SYS_ADMIN \
--device=/dev/kfd --device=/dev/dri --device=/dev/mem \
--group-add video --cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
-v /:/work \
-e SHELL=/bin/bash \
--name Ministral \
lmsysorg/sglang:v0.5.9-rocm720-mi30x \
/bin/bash
```
#### 4.2.2 Launch the server
```shell Command
sglang serve \
--model-path mistralai/Ministral-3-14B-Instruct-2512 \
--tp 1 \
--trust-remote-code
```
## 5. Benchmark
This section uses **industry-standard configurations** for comparable benchmark results.
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: MI300X GPU (8x)
- Model: mistralai/Ministral-3-14B-Instruct-2512
- Tensor Parallelism: 1
- SGLang Version: 0.5.7
- Model Deployment Command:
```bash Command
sglang serve \
--model-path mistralai/Ministral-3-14B-Instruct-2512 \
--tp 1 \
--trust-remote-code
```
##### Low Concurrency
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model mistralai/Ministral-3-14B-Instruct-2512 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 65.08
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4218
Request throughput (req/s): 0.15
Input token throughput (tok/s): 93.75
Output token throughput (tok/s): 64.84
Peak output token throughput (tok/s): 151.00
Peak concurrent requests: 2
Total token throughput (tok/s): 158.59
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 6505.51
Median E2E Latency (ms): 3037.37
---------------Time to First Token----------------
Mean TTFT (ms): 3709.33
Median TTFT (ms): 53.72
P99 TTFT (ms): 33320.77
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 6.63
Median TPOT (ms): 6.64
P99 TPOT (ms): 6.66
---------------Inter-Token Latency----------------
Mean ITL (ms): 6.64
Median ITL (ms): 6.65
P95 ITL (ms): 6.75
P99 ITL (ms): 6.82
Max ITL (ms): 8.45
==================================================
```
##### Medium Concurrency
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model mistralai/Ministral-3-14B-Instruct-2512 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 31.20
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 40805
Total generated tokens (retokenized): 40783
Request throughput (req/s): 2.56
Input token throughput (tok/s): 1271.38
Output token throughput (tok/s): 1307.82
Peak output token throughput (tok/s): 1760.00
Peak concurrent requests: 22
Total token throughput (tok/s): 2579.20
Concurrency: 13.72
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 5351.07
Median E2E Latency (ms): 5626.45
---------------Time to First Token----------------
Mean TTFT (ms): 280.87
Median TTFT (ms): 68.16
P99 TTFT (ms): 1194.79
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 10.47
Median TPOT (ms): 10.10
P99 TPOT (ms): 20.00
---------------Inter-Token Latency----------------
Mean ITL (ms): 9.96
Median ITL (ms): 9.10
P95 ITL (ms): 9.87
P99 ITL (ms): 51.39
Max ITL (ms): 888.63
==================================================
```
##### High Concurrency
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model mistralai/Ministral-3-14B-Instruct-2512 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 88.75
Total input tokens: 249831
Total input text tokens: 249831
Total input vision tokens: 0
Total generated tokens: 252662
Total generated tokens (retokenized): 252547
Request throughput (req/s): 5.63
Input token throughput (tok/s): 2815.01
Output token throughput (tok/s): 2846.91
Peak output token throughput (tok/s): 4271.00
Peak concurrent requests: 110
Total token throughput (tok/s): 5661.93
Concurrency: 93.04
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 16514.45
Median E2E Latency (ms): 15834.45
---------------Time to First Token----------------
Mean TTFT (ms): 148.57
Median TTFT (ms): 99.15
P99 TTFT (ms): 455.86
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 32.93
Median TPOT (ms): 34.73
P99 TPOT (ms): 38.05
---------------Inter-Token Latency----------------
Mean ITL (ms): 32.45
Median ITL (ms): 27.30
P95 ITL (ms): 71.73
P99 ITL (ms): 73.45
Max ITL (ms): 328.10
==================================================
```
### 5.2 Accuracy Benchmark
Document model accuracy on standard benchmarks:
#### 5.2.1 GSM8K Benchmark
- Benchmark Command
```bash Command
python3 benchmark/gsm8k/bench_sglang.py \
--num-shots 8 \
--num-questions 1316 \
--parallel 1316
```
**Test Results:**
```text Output
Accuracy: 0.959
Invalid: 0.000
Latency: 29.185 s
Output throughput: 4854.672 token/s
```
docs_new/cookbook/autoregressive/Mistral/Mistral-Small-4.mdx
+393
View File
@@ -0,0 +1,393 @@
---
title: Mistral Small 4
metatags:
description: "Deploy Mistral Small 4 with SGLang - unified hybrid model combining instruct, reasoning, and agentic capabilities with multimodal support."
---
import { MistralSmall4Deployment } from '/src/snippets/autoregressive/mistral-small-4-deployment.jsx';
## 1. Model Introduction
**Mistral Small 4** is a powerful hybrid model from Mistral AI that unifies the capabilities of three different model families — **Instruct**, **Reasoning** (formerly called Magistral), and **Agentic (formerly called Devstral)** — into a single, unified model.
With its multimodal capabilities, efficient MoE architecture, and flexible mode switching, Mistral Small 4 is a versatile general-purpose model for virtually any task. In a latency-optimized setup, it achieves a 40% reduction in end-to-end completion time; in a throughput-optimized setup, it delivers 3× more requests per second compared to Mistral Small 3.
**Key Features:**
- **Hybrid Reasoning**: Switch between instant reply mode and deep reasoning/thinking mode — reasoning effort is configurable per request
- **Vision**: Accepts both text and image inputs, providing insights based on visual content
- **Function Calling**: Native tool calling and JSON output support with best-in-class agentic capabilities
- **Multilingual**: Supports dozens of languages including English, French, Spanish, German, Chinese, Japanese, Korean, Arabic, and more
- **Context Window**: 256K context window
- **Efficient MoE**: 119B total parameters, 128 experts, 4 active per token (6.5B activated parameters)
- **Apache 2.0 License**: Open-source, usable and modifiable for commercial and non-commercial purposes
- Reasoning effort supported are only **"none" and "high"**
**Architecture:**
- Same general architecture as Mistral 3
- MoE: 128 experts, 4 active per token
- 119B total parameters, 6.5B activated per token
- Multimodal input: text + image
**Models:**
- **[mistralai/Mistral-Small-4-119B-2603](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603)** (FP8)
- **[mistralai/Mistral-Small-4-119B-2603-NVFP4](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-NVFP4)**
- **[mistralai/Leanstral-2603](https://huggingface.co/mistralai/Leanstral-2603)** — same architecture, use the same launch commands as Mistral-Small-4-119B-2603
- **[mistralai/Mistral-Small-4-119B-2603-eagle](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-eagle)** — EAGLE speculative decoding weights for faster inference
---
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
<Info>
Mistral Small 4 support landed in [sgl-project/sglang#20708](https://github.com/sgl-project/sglang/pull/20708) and has been merged into `main`. A model-specific Docker image is no longer required. Use the standard SGLang installation methods from the [official installation guide](../../../docs/get-started/installation).
</Info>
---
## 3. Model Deployment
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to generate a launch command for Mistral Small 4.
<MistralSmall4Deployment />
### 3.2 Configuration Tips
- **Tensor Parallelism**: Mistral Small 4 FP8 (~119 GB) requires tp=2 on Hopper (H100/H200), tp=1 on Blackwell (B200/B300). NVFP4 (~60 GB, Blackwell only) runs with tp=1.
- **Reasoning effort**: Reasoning depth is configurable per request via `reasoning_effort` (`"none"`, `"high"`). No restart required — toggle per call.
- **Context length vs memory**: The model has a 256K context window. If you are memory-constrained, lower `--context-length` (e.g. `32768`) and increase once things are stable.
- **Tool calling**: Enable `--tool-call-parser mistral` to activate native function calling support.
- **Reasoning parser**: Enable `--reasoning-parser mistral` to separate `reasoning_content` from the main response content.
- **Speculative decoding (EAGLE)**: Enable with `--speculative-algorithm EAGLE --speculative-draft-model-path mistralai/Mistral-Small-4-119B-2603-eagle` using the [EAGLE weights](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-eagle) for lower latency.
---
## 4. Model Invocation
### 4.1 Thinking Mode
Mistral Small 4 is a hybrid reasoning model. By default, it does not produce a default reasoning response. Use `--reasoning_effort high` to toggle reasoning on.
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY",
)
response = client.chat.completions.create(
model="mistralai/Mistral-Small-4-119B-2603",
messages=[
{"role": "user", "content": "Solve step by step: what is 17 × 23 + 144 / 12?"},
],
extra_body={"reasoning_effort": "high"},
)
print("Reasoning:", response.choices[0].message.reasoning_content)
print("Answer:", response.choices[0].message.content)
```
**Output:**
```text Output
Reasoning: First, I'll break down the problem into two parts: the multiplication and
the division. According to the order of operations (PEMDAS/BODMAS), multiplication and
division are performed from left to right before addition.
17 × 23 = 17 × (20 + 3) = (17 × 20) + (17 × 3) = 340 + 51 = 391
144 / 12 = 12
Finally, add the results: 391 + 12 = 403
Answer: The solution to the problem is as follows:
1. First, perform the multiplication: 17 × 23.
- 17 × 20 = 340
- 17 × 3 = 51
- 340 + 51 = 391
2. Then, perform the division: 144 / 12 = 12.
3. Finally, add the results:
- 391 + 12 = 403
**Answer:** \boxed{403}
```
### 4.2 Instruct Mode (Reasoning Off)
To skip the reasoning trace and get a fast direct response, set `reasoning_effort` to `"none"`:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY",
)
response = client.chat.completions.create(
model="mistralai/Mistral-Small-4-119B-2603",
messages=[
{"role": "user", "content": "Write a Python function to reverse a string."},
],
extra_body={"reasoning_effort": "none"},
)
print(response.choices[0].message.content)
```
**Output:**
````text Output
# Python Function to Reverse a String
Here are several ways to write a Python function to reverse a string:
## Method 1: Using String Slicing (Most Pythonic)
```python
def reverse_string(s):
"""Reverse a string using slicing."""
return s[::-1]
```
## Method 2: Using a Loop
```python Example
def reverse_string(s):
"""Reverse a string using a loop."""
reversed_str = ""
for char in s:
reversed_str = char + reversed_str
return reversed_str
```
## Method 3: Using reversed() function
```python Example
def reverse_string(s):
"""Reverse a string using reversed() function."""
return ''.join(reversed(s))
```
The first method using string slicing (`s[::-1]`) is generally the most efficient and
recommended approach in Python.
Example usage:
```python Example
original = "Hello, World!"
reversed_str = reverse_string(original)
print(reversed_str) # Output: "!dlroW ,olleH"
```
````
### 4.3 Streaming with Reasoning
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY",
)
stream = client.chat.completions.create(
model="mistralai/Mistral-Small-4-119B-2603",
messages=[
{"role": "user", "content": "Explain the difference between async and threading in Python."},
],
extra_body={"reasoning_effort": "high"},
stream=True,
)
print("=== Reasoning ===")
for chunk in stream:
delta = chunk.choices[0].delta
if hasattr(delta, "reasoning_content") and delta.reasoning_content:
print(delta.reasoning_content, end="", flush=True)
elif delta.content:
print("\n=== Response ===")
print(delta.content, end="", flush=True)
print()
```
**Output:**
```text Output
=== Reasoning ===
Okay, the user is asking about the difference between async and threading in Python.
I need to break this down clearly, covering the key aspects of both, like their
purposes, performance characteristics, and use cases...
=== Response ===
In Python, **`async`/`asyncio`** and **`threading`** are two different concurrency
models, each suited for specific use cases. Here's a breakdown of their key differences:
### 1. Model of Concurrency
- **Threading**: Based on preemptive multitasking using OS threads.
- **Async** (`asyncio`): Based on cooperative multitasking. Tasks voluntarily yield...
```
### 4.4 Tool Calling
Mistral Small 4 supports native function calling. Enable with `--tool-call-parser mistral`:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY",
)
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a city",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string", "description": "City name"},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
},
"required": ["location"],
},
},
}
]
response = client.chat.completions.create(
model="mistralai/Mistral-Small-4-119B-2603",
messages=[{"role": "user", "content": "What's the weather in Paris?"}],
tools=tools,
tool_choice="auto",
)
tool_calls = response.choices[0].message.tool_calls
for tc in tool_calls:
print(f"Tool: {tc.function.name}")
print(f"Args: {tc.function.arguments}")
```
**Output:**
```text Output
Tool: get_weather
Args: {"location": "Paris"}
```
### 4.5 Vision (Image Input)
Mistral Small 4 accepts image inputs alongside text:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY",
)
response = client.chat.completions.create(
model="mistralai/Mistral-Small-4-119B-2603",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Describe what you see in this image."},
{
"type": "image_url",
"image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png"},
},
],
}
],
)
print(response.choices[0].message.content)
```
**Output:**
```text Output
The image is a copyright symbol, represented by a stylized version of the lowercase
letter "c" inside a circle. The "c" is depicted in a white or light-colored font, and
the circle is orange. The design is simple yet striking, using oval and elliptical
shapes to create a distinct symbol which signifies copyright protection.
```
---
## 5. Benchmarks
### 5.1 Accuracy Benchmarks
#### GSM8K
```bash Command
python3 benchmark/gsm8k/bench_sglang.py --port 30000
```
**Results:**
```text Output
TODO
```
#### MMLU
```bash Command
python3 benchmark/mmlu/bench_sglang.py --port 30000
```
**Results:**
```text Output
TODO
```
### 5.2 Speed Benchmarks
#### Latency (Low Concurrency)
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--num-prompts 10 \
--max-concurrency 1 \
--random-input-len 1024 \
--random-output-len 512 \
--port 30000
```
**Results:**
```text Output
TODO
```
#### Throughput (High Concurrency)
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--num-prompts 1000 \
--max-concurrency 100 \
--random-input-len 1024 \
--random-output-len 512 \
--port 30000
```
**Results:**
```text Output
TODO
```
docs_new/cookbook/autoregressive/Moonshotai/Kimi-K2.5.mdx
+1244
View File
File diff suppressed because it is too large Load Diff
docs_new/cookbook/autoregressive/Moonshotai/Kimi-K2.6.mdx
+1309
View File
File diff suppressed because it is too large Load Diff
docs_new/cookbook/autoregressive/Moonshotai/Kimi-K2.mdx
+520
View File
@@ -0,0 +1,520 @@
---
title: Kimi-K2
metatags:
description: "Deploy Kimi-K2 MoE model with SGLang - 1T total parameters, 32B active, step-by-step reasoning and tool calling capabilities."
---
import { KimiK2Deployment } from '/src/snippets/autoregressive/kimi-k2-deployment.jsx';
## 1. Model Introduction
[Kimi-K2](https://moonshotai.github.io/Kimi-K2/) is a state-of-the-art MoE language model by Moonshot AI with 32B activated parameters and 1T total parameters.
**Model Variants:**
- **[Kimi-K2-Instruct](https://huggingface.co/moonshotai/Kimi-K2-Instruct)**: Post-trained model optimized for general-purpose chat and agentic tasks. Compatible with vLLM, SGLang, KTransformers, and TensorRT-LLM.
- **[Kimi-K2-Thinking](https://huggingface.co/moonshotai/Kimi-K2-Thinking)**: Advanced thinking model with step-by-step reasoning and tool calling. Native INT4 quantization with 256k context window. Ideal for complex reasoning and multi-step tool use.
- **ROCm Support**: Compatible with AMD MI300X GPUs via SGLang (verified).
For details, see [official documentation](https://github.com/MoonshotAI/Kimi-K2) and [technical report](https://www.arxiv.org/abs/2507.20534).
## 2. SGLang Installation
Refer to the [official SGLang installation guide](../../../docs/get-started/installation).
## 3. Model Deployment
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and capabilities.
<KimiK2Deployment />
### 3.2 Configuration Tips
- **Memory**: Requires 8 GPUs with ≥140GB each (H200/B200). Use `--context-length 128000` to conserve memory.
- **Expert Parallelism (EP)**: Use `--ep` for better MoE throughput. See [EP docs](../../../docs/advanced_features/expert_parallelism).
- **Data Parallel (DP)**: Enable with `--dp 4 --enable-dp-attention` for production throughput.
- **KV Cache**: Use `--kv-cache-dtype fp8_e4m3` to reduce memory by 50% (CUDA 11.8+).
- **Reasoning Parser**: Add `--reasoning-parser kimi_k2` for Kimi-K2-Thinking to separate thinking and content.
- **Tool Call Parser**: Add `--tool-call-parser kimi_k2` for structured tool calls.
- **AMD GPU**: Set `SGLANG_ROCM_FUSED_DECODE_MLA=0` before launching AMD GPU.
## 4. Model Invocation
### 4.1 Basic Usage
See [Basic API Usage](../../../docs/get-started/quickstart).
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
Enable reasoning parser for Kimi-K2-Thinking:
```shell Command
python -m sglang.launch_server \
--model moonshotai/Kimi-K2-Thinking \
--reasoning-parser kimi_k2 \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
**Example:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="moonshotai/Kimi-K2-Thinking",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.6,
max_tokens=2048,
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
The user asks: "What is 15% of 240?" This is a straightforward percentage calculation problem. I need to solve it step by step.
Step 1: Understand what "percent" means.
- "Percent" means "per hundred". So 15% means 15 per 100, or 15/100, or 0.15.
Step 2: Convert the percentage to a decimal.
- 15% = 15 / 100 = 0.15
Step 3: Multiply the decimal by the number.
- 0.15 * 240
Step 4: Perform the multiplication.
- 0.15 * 240 = (15/100) * 240
- = 15 * 240 / 100
- = 3600 / 100
- = 36
Alternatively, I can calculate it directly:
- 0.15 * 240
- 15 * 240 = 3600
- 3600 / 100 = 36
Or, break it down:
- 10% of 240 = 24
- 5% of 240 = half of 10% = 12
- 15% of 240 = 10% + 5% = 24 + 12 = 36
I should present the solution clearly with steps. The most standard method is converting to decimal and multiplying.
Let me structure the answer:
1. Convert the percentage to a decimal.
2. Multiply the decimal by the number.
3. Show the calculation.
4. State the final answer.
This is simple and easy to follow.
=============== Content =================
Here is the step-by-step solution:
**Step 1: Convert the percentage to a decimal**
15% means 15 per 100, which is 15 ÷ 100 = **0.15**
**Step 2: Multiply the decimal by the number**
0.15 × 240
**Step 3: Calculate the result**
0.15 × 240 = **36**
**Answer:** 15% of 240 is **36**.
```
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
#### 4.2.2 Tool Calling
Kimi-K2-Instruct and Kimi-K2-Thinking support tool calling capabilities. Enable the tool call parser during deployment:
**Deployment Command:**
```shell Command
python -m sglang.launch_server \
--model moonshotai/Kimi-K2-Instruct \
--tool-call-parser kimi_k2 \
--tp 8 \
--trust-remote-code \
--host 0.0.0.0 \
--port 8000
```
**Python Example (with Thinking Process):**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="moonshotai/Kimi-K2-Thinking",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Accumulate tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================\n", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {
'name': None,
'arguments': ''
}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]['name'] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
# Print content
if delta.content:
print(delta.content, end="", flush=True)
# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"🔧 Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
print()
```
**Output Example:**
```text Output
=============== Thinking =================
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. Beijing is a major city in China, so I should be able to get weather data for it. The location parameter is required, but the unit parameter is optional. Since the user didn't specify a temperature unit, I can just provide the location and let the function use its default. I'll check the weather in Beijing for you.
=============== Content =================
🔧 Tool Call: get_weather
Arguments: {"location":"Beijing"}
```
**Note:**
- The reasoning parser shows how the model decides to use a tool
- Tool calls are clearly marked with the function name and arguments
- You can then execute the function and send the result back to continue the conversation
**Handling Tool Call Results:**
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="moonshotai/Kimi-K2-Thinking",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: NVIDIA B200 GPU (8x)
- Model: Kimi-K2-Instruct
- sglang version: 0.5.6.post1
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.
#### 5.1.1 Latency-Sensitive Benchmark
- Model Deployment Command:
```shell Command
python3 -m sglang.launch_server \
--model-path moonshotai/Kimi-K2-Instruct \
--tp 8 \
--dp 4 \
--enable-dp-attention \
--trust-remote-code \
--host 0.0.0.0 \
--port 8000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 8000 \
--model moonshotai/Kimi-K2-Instruct\
--num-prompts 10 \
--max-concurrency 1
```
- **Test Results**:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 44.93
Total input tokens: 1951
Total input text tokens: 1951
Total input vision tokens: 0
Total generated tokens: 2755
Total generated tokens (retokenized): 2748
Request throughput (req/s): 0.22
Input token throughput (tok/s): 43.42
Output token throughput (tok/s): 61.32
Peak output token throughput (tok/s): 64.00
Peak concurrent requests: 3
Total token throughput (tok/s): 104.74
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 4489.56
Median E2E Latency (ms): 4994.53
---------------Time to First Token----------------
Mean TTFT (ms): 141.22
Median TTFT (ms): 158.28
P99 TTFT (ms): 166.90
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 18.40
Median TPOT (ms): 15.63
P99 TPOT (ms): 39.88
---------------Inter-Token Latency----------------
Mean ITL (ms): 15.78
Median ITL (ms): 15.76
P95 ITL (ms): 16.36
P99 ITL (ms): 16.59
Max ITL (ms): 19.94
==================================================
```
#### 5.1.2 Throughput-Sensitive Benchmark
- Model Deployment Command:
```shell Command
python3 -m sglang.launch_server \
--model-path moonshotai/Kimi-K2-Instruct \
--tp 8 \
--dp 4 \
--ep 4 \
--enable-dp-attention \
--trust-remote-code \
--host 0.0.0.0 \
--port 8000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 8000 \
--model moonshotai/Kimi-K2-Instruct\
--num-prompts 1000 \
--max-concurrency 100
```
- **Test Results**:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 174.11
Total input tokens: 296642
Total input text tokens: 296642
Total input vision tokens: 0
Total generated tokens: 193831
Total generated tokens (retokenized): 168687
Request throughput (req/s): 5.74
Input token throughput (tok/s): 1703.73
Output token throughput (tok/s): 1113.25
Peak output token throughput (tok/s): 2383.00
Peak concurrent requests: 112
Total token throughput (tok/s): 2816.97
Concurrency: 89.60
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 15601.09
Median E2E Latency (ms): 10780.52
---------------Time to First Token----------------
Mean TTFT (ms): 457.42
Median TTFT (ms): 221.62
P99 TTFT (ms): 2475.32
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 97.23
Median TPOT (ms): 85.61
P99 TPOT (ms): 435.95
---------------Inter-Token Latency----------------
Mean ITL (ms): 78.61
Median ITL (ms): 43.66
P95 ITL (ms): 169.53
P99 ITL (ms): 260.91
Max ITL (ms): 1703.21
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 GSM8K Benchmark
- Server Command
```shell Command
python3 -m sglang.launch_server \
--model-path moonshotai/Kimi-K2-Instruct \
--tp 8 \
--dp 4 \
--trust-remote-code \
--host 0.0.0.0 \
--port 8000
```
- Benchmark Command
```shell Command
python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000
```
- **Result**:
```text Output
Accuracy: 0.960
Invalid: 0.000
Latency: 15.956 s
Output throughput: 1231.699 token/s
```
docs_new/cookbook/autoregressive/Moonshotai/Kimi-Linear.mdx
+297
View File
@@ -0,0 +1,297 @@
---
title: Kimi-Linear
metatags:
description: "Deploy Kimi-Linear with SGLang - community contribution guide for Moonshot AI's Kimi-Linear model deployment."
---
import { KimiLinearDeployment } from '/src/snippets/autoregressive/kimi-linear-deployment.jsx';
## AMD GPU Support
## 1. Model Introduction
Kimi Linear is a hybrid linear attention architecture that outperforms traditional full attention methods across various contexts, including short, long, and reinforcement learning (RL) scaling regimes. At its core is Kimi Delta Attention (KDA)—a refined version of Gated DeltaNet that introduces a more efficient gating mechanism to optimize the use of finite-state RNN memory.
This generation delivers comprehensive upgrades across the board:
Kimi Delta Attention (KDA): A linear attention mechanism that refines the gated delta rule with finegrained gating.
Hybrid Architecture: A 3:1 KDA-to-global MLA ratio reduces memory usage while maintaining or surpassing the quality of full attention.
Superior Performance: Outperforms full attention in a variety of tasks, including long-context and RL-style benchmarks on 1.4T token training runs with fair comparisons.
High Throughput: Achieves up to 6× faster decoding and significantly reduces time per output token (TPOT).
For more details, please refer to the [official Kimi Linear GitHub Repository]: https://github.com/MoonshotAI/Kimi-Linear
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
<KimiLinearDeployment />
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
### 4.2 Advanced Usage
#### 4.2.1 Launch the docker
```shell Command
docker pull lmsysorg/sglang:v0.5.7-rocm700-mi30x
```
```shell Command
docker run -d -it --ipc=host --network=host --privileged \
--cap-add=CAP_SYS_ADMIN \
--device=/dev/kfd --device=/dev/dri --device=/dev/mem \
--group-add video --cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
-v /:/work \
-e SHELL=/bin/bash \
--name Kimi-linear \
lmsysorg/sglang:v0.5.7-rocm700-mi30x \
/bin/bash
```
#### 4.2.2 pre-installation steps inside the docker
```shell Command
pip install sentencepiece tiktoken
```
#### 4.2.3 Launch the server
```shell Command
export SGLANG_ROCM_FUSED_DECODE_MLA=0
SGLANG_ROCM_FUSED_DECODE_MLA=0 python3 -m sglang.launch_server \
--model-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
--tokenizer-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
--tp 4 \
--trust-remote-code
```
## 5. Benchmark
### 5.1 Speed Benchmark
Test Environment:
Hardware: AMD MI300X GPU
Model: Kimi-Linear-48B-A3B-Instruct
Tensor Parallelism: 4
sglang version: 0.5.7
- **Model Deployment**
```bash Command
SGLANG_ROCM_FUSED_DECODE_MLA=0 python3 -m sglang.launch_server \
--model-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
--tokenizer-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
--tp 4 \
--trust-remote-code
```
### 5.1.1 Low Concurrency (Latency-Optimized)
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model moonshotai/Kimi-Linear-48B-A3B-Instruct \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 23.86
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4001
Request throughput (req/s): 0.42
Input token throughput (tok/s): 255.70
Output token throughput (tok/s): 176.86
Peak output token throughput (tok/s): 190.00
Peak concurrent requests: 2
Total token throughput (tok/s): 432.56
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 2383.93
Median E2E Latency (ms): 1911.63
---------------Time to First Token----------------
Mean TTFT (ms): 141.33
Median TTFT (ms): 126.27
P99 TTFT (ms): 294.76
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 5.32
Median TPOT (ms): 5.33
P99 TPOT (ms): 5.36
---------------Inter-Token Latency----------------
Mean ITL (ms): 5.33
Median ITL (ms): 5.32
P95 ITL (ms): 5.44
P99 ITL (ms): 5.58
Max ITL (ms): 11.46
==================================================
```
### 5.1.2 Medium Concurrency (Balanced)
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model moonshotai/Kimi-Linear-48B-A3B-Instruct \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 31.38
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 40805
Total generated tokens (retokenized): 39667
Request throughput (req/s): 2.55
Input token throughput (tok/s): 1264.13
Output token throughput (tok/s): 1300.37
Peak output token throughput (tok/s): 1801.00
Peak concurrent requests: 21
Total token throughput (tok/s): 2564.50
Concurrency: 14.13
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 5543.18
Median E2E Latency (ms): 5755.31
---------------Time to First Token----------------
Mean TTFT (ms): 175.25
Median TTFT (ms): 137.87
P99 TTFT (ms): 292.92
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 10.75
Median TPOT (ms): 10.87
P99 TPOT (ms): 16.74
---------------Inter-Token Latency----------------
Mean ITL (ms): 10.54
Median ITL (ms): 7.95
P95 ITL (ms): 13.68
P99 ITL (ms): 116.80
Max ITL (ms): 299.89
==================================================
```
### 5.1.3 High Concurrency (Throughput-Optimized)
- Benchmark Command:
```bash Command
python3 -m sglang.bench_serving \
--backend sglang \
--model moonshotai/Kimi-Linear-48B-A3B-Instruct \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100 \
--request-rate inf
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 79.71
Total input tokens: 249831
Total input text tokens: 249831
Total input vision tokens: 0
Total generated tokens: 252662
Total generated tokens (retokenized): 228448
Request throughput (req/s): 6.27
Input token throughput (tok/s): 3134.20
Output token throughput (tok/s): 3169.72
Peak output token throughput (tok/s): 6109.00
Peak concurrent requests: 110
Total token throughput (tok/s): 6303.92
Concurrency: 94.80
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 15113.92
Median E2E Latency (ms): 13851.52
---------------Time to First Token----------------
Mean TTFT (ms): 564.46
Median TTFT (ms): 226.04
P99 TTFT (ms): 2683.14
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 29.63
Median TPOT (ms): 31.28
P99 TPOT (ms): 38.84
---------------Inter-Token Latency----------------
Mean ITL (ms): 28.85
Median ITL (ms): 16.29
P95 ITL (ms): 123.42
P99 ITL (ms): 157.80
Max ITL (ms): 2481.11
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 GSM8K Benchmark
- Server Command
```shell Command
SGLANG_ROCM_FUSED_DECODE_MLA=0 python3 -m sglang.launch_server \
--model-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
--tokenizer-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
--tp 4 \
--trust-remote-code
```
- Benchmark Command
```shell Command
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
```
- **Result**:
```text Output
Accuracy: 0.705
Invalid: 0.000
Latency: 11.855 s
Output throughput: 3224.982 token/s
```
docs_new/cookbook/autoregressive/NVIDIA/Nemotron3-Nano.mdx
+375
View File
@@ -0,0 +1,375 @@
---
title: Nemotron3-Nano
metatags:
description: "Deploy NVIDIA Nemotron3-Nano 30B hybrid LLM with SGLang - MoE, Mamba2, and attention layers with BF16/FP8 precision options."
---
import { Nemotron3NanoDeployment } from '/src/snippets/autoregressive/nemotron3-nano-deployment.jsx';
## 1. Model Introduction
`NVIDIA Nemotron3-Nano` is a 30B-parameter hybrid LLM that mixes Mixture-of-Experts (MoE) feed-forward layers, Mamba2 sequence-modeling layers, and standard self-attention layers in a single stack rather than classic “attention + MLP” transformer blocks.
The BF16 variant (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16`) is designed as a high-fidelity reference model. For optimized inference performance on modern NVIDIA GPUs, the FP8 variant (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8`) and the NVFP4 variant (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4`) are supported.
At a high level:
- **Hybrid layer stack (Mamba2 + MoE + attention):** The network is composed of interleaved layers that are *either* Mamba2, *or* MoE feed-forward, *or* attention-only.
- **Non-uniform layer ordering:** The order and mix of these specialized layers is not a simple, rigid pattern, enabling the model to trade off sequence modeling, routing capacity, and expressivity across depth.
- **Deployment-friendly precision:** Use BF16 for accuracy-sensitive and evaluation workloads; use FP8 for latency- and throughput-critical serving on recent NVIDIA GPUs.
## 2. SGLang Installation
Refer to the [official SGLang installation guide](../../../docs/get-started/installation), or install nightly wheel through:
```bash Command
uv pip install sglang==0.5.6.post3.dev1278+gad1b4e472 --extra-index-url https://sgl-project.github.io/whl/nightly/
```
## 3. Model Deployment
This section provides a progressive guide from quick deployment to performance tuning.
### 3.1 Basic Configuration
**Interactive Command Generator**: select hardware, model variant, and common knobs to generate a launch command.
<Nemotron3NanoDeployment />
### 3.2 Configuration Tips
- **Attention backend**:
**H200**: Use flash attention 3 backend by default.
**B200**: Use flashinfer backend by default.
- **TP support**:
To set tp size, use `--tp <1|2|4|8>`.
- **FP8 KV cache**:
To enable fp8 kv cache, please append `--kv-cache-dtype fp8_e4m3`.
## 4. Model Invocation
### 4.1 Basic Usage (OpenAI-Compatible API)
SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY",
)
resp = client.chat.completions.create(
model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Summarize what MoE models are in 5 bullets."},
],
temperature=0.7,
max_tokens=256,
)
print(resp.choices[0].message.content)
```
Streaming chat completion
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY",
)
stream = client.chat.completions.create(
model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
messages=[
{"role": "system", "content": "You are a helpful AI assistant."},
{"role": "user", "content": "What are the first 5 prime numbers?"}
],
temperature=0.7,
max_tokens=1024,
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta
if delta and delta.content:
print(delta.content, end="", flush=True)
```
### 4.2 Reasoning
To enable reasoning, `--reasoning-parser nemotron_3` should be appended to the launching command. The model supports two modes - Reasoning ON (default) vs OFF. This can be toggled by setting enable_thinking to False, as shown below.
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY",
)
# Reasoning on (default)
print("Reasoning on")
resp = client.chat.completions.create(
model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Write a haiku about GPUs."}
],
temperature=0.7,
max_tokens=512,
)
print(resp.choices[0].message.reasoning_content)
# Reasoning off
print("Reasoning off")
resp = client.chat.completions.create(
model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Write a haiku about GPUs."}
],
temperature=0.6,
max_tokens=256,
extra_body={"chat_template_kwargs": {"enable_thinking": False}}
)
print(resp.choices[0].message.reasoning_content)
```
### 4.3 Tool calling
To enable reasoning, `--tool-call-parser qwen3_coder` should be appended to the launching command. Call functions using the OpenAI Tools schema and inspect returned tool_calls.
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY",
)
# Tool calling via OpenAI tools schema
TOOLS = [
{
"type": "function",
"function": {
"name": "calculate_tip",
"parameters": {
"type": "object",
"properties": {
"bill_total": {
"type": "integer",
"description": "The total amount of the bill"
},
"tip_percentage": {
"type": "integer",
"description": "The percentage of tip to be applied"
}
},
"required": ["bill_total", "tip_percentage"]
}
}
}
]
completion = client.chat.completions.create(
model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
messages=[
{"role": "system", "content": ""},
{"role": "user", "content": "My bill is $50. What will be the amount for 15% tip?"}
],
tools=TOOLS,
temperature=0.6,
top_p=0.95,
max_tokens=512,
stream=False
)
print(completion.choices[0].message.reasoning_content)
print(completion.choices[0].message.tool_calls)
```
---
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: NVIDIA B200 GPU
**FP8 variant**
- Model Deployment Command:
```shell Command
python3 -m sglang.launch_server \
--model-path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \
--trust-remote-code \
--max-running-requests 1024 \
--host 0.0.0.0 \
--port 30000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \
--dataset-name random \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 4096 \
--max-concurrency 256
```
- **Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 256
Successful requests: 4096
Benchmark duration (s): 183.18
Total input tokens: 2081726
Total input text tokens: 2081726
Total input vision tokens: 0
Total generated tokens: 2116125
Total generated tokens (retokenized): 1076256
Request throughput (req/s): 22.36
Input token throughput (tok/s): 11364.25
Output token throughput (tok/s): 11552.04
Peak output token throughput (tok/s): 24692.00
Peak concurrent requests: 294
Total token throughput (tok/s): 22916.30
Concurrency: 251.19
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 11233.74
Median E2E Latency (ms): 11142.97
---------------Time to First Token----------------
Mean TTFT (ms): 172.99
Median TTFT (ms): 116.57
P99 TTFT (ms): 1193.68
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 21.74
Median TPOT (ms): 21.14
P99 TPOT (ms): 41.12
---------------Inter-Token Latency----------------
Mean ITL (ms): 21.45
Median ITL (ms): 9.06
P95 ITL (ms): 62.59
P99 ITL (ms): 110.83
Max ITL (ms): 5368.19
==================================================
```
**BF16 variant**
- Model Deployment Command:
```shell Command
python3 -m sglang.launch_server \
--model-path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \
--trust-remote-code \
--max-running-requests 1024 \
--host 0.0.0.0 \
--port 30000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \
--dataset-name random \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 4096 \
--max-concurrency 256
```
- **Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 256
Successful requests: 4096
Benchmark duration (s): 360.22
Total input tokens: 2081726
Total input text tokens: 2081726
Total input vision tokens: 0
Total generated tokens: 2087288
Total generated tokens (retokenized): 1940652
Request throughput (req/s): 11.37
Input token throughput (tok/s): 5779.10
Output token throughput (tok/s): 5794.55
Peak output token throughput (tok/s): 9169.00
Peak concurrent requests: 276
Total token throughput (tok/s): 11573.65
Concurrency: 249.76
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 21965.10
Median E2E Latency (ms): 21706.35
---------------Time to First Token----------------
Mean TTFT (ms): 211.54
Median TTFT (ms): 93.06
P99 TTFT (ms): 2637.66
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 43.27
Median TPOT (ms): 43.04
P99 TPOT (ms): 61.15
---------------Inter-Token Latency----------------
Mean ITL (ms): 42.77
Median ITL (ms): 28.46
P95 ITL (ms): 71.85
P99 ITL (ms): 113.20
Max ITL (ms): 5237.28
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 GSM8K Benchmark
**Environment**
- Hardware: NVIDIA B200 GPU
- Model: BF16 checkpoint
**Launch Model**
```bash Command
python3 -m sglang.launch_server \
--model-path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \
--trust-remote-code \
--reasoning-parser nemotron_3
```
**Run Benchmark with lm-eval**
```bash Command
pip install lm-eval[api]==0.4.9.2
lm_eval --model local-completions --tasks gsm8k --model_args "model=nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16,base_url=http://127.0.0.1:30000/v1/completions,num_concurrent=4,max_retries=3,tokenized_requests=False,max_lengths=16384" --gen_kwargs '{"chat_template_kwargs":{"thinking":true}}' --batch_size 256
```
**Test Results:**
```text Output
|Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.5603|± |0.0137|
| | |strict-match | 5|exact_match|↑ |0.8453|± |0.0100|
```
docs_new/cookbook/autoregressive/NVIDIA/Nemotron3-Super.mdx
+571
View File
@@ -0,0 +1,571 @@
---
title: NVIDIA Nemotron3-Super
metatags:
description: "Deploy NVIDIA Nemotron3-Super with SGLang - 120B hybrid MoE model (12B active) with 1M context window optimized for multi-agent systems and tool use."
---
import { Nemotron3SuperDeployment } from '/src/snippets/autoregressive/nemotron3-super-deployment.jsx';
## 1. Model Introduction
`NVIDIA Nemotron3-Super` is a leading open model in the Nemotron 3 family, built for running many collaborating agents together. It is optimized for agentic systems that chain planning, reasoning, and tool use workloads that generate far more tokens than single turn chat and require strong reasoning at every step.
Nemotron 3 Super is a 120B parameter hybrid MoE model that activates only 12B parameters per forward pass, delivering strong accuracy for coding, tool calling, and instruction following at a fraction of the cost. It also supports a 1M token context window so agents can keep conversation history and plan state in view across long workflows.
Architecture and key features:
- **Hybrid Transformer-Mamba Architecture (MoE):** Combines Mixture of Experts with a hybrid Transformer-Mamba architecture, enabling efficient routing and sequence modeling in a single stack.
- **Highest throughput efficiency in its size category:** Delivers up to 5x higher throughput compared to the previous Nemotron Super model (Llama Nemotron Super 1.5).
- **Multi-Token Prediction (MTP):** By predicting several future tokens simultaneously in a single forward pass, MTP drastically accelerates the generation of long-form text.
- **Thinking Budget support:** Supports Thinking Budget for optimal accuracy with minimum reasoning token generation.
## 2. SGLang Installation
SGLang from the main branch is required for Nemotron3-Super. You can install from source and with a nightly docker.
```bash Command
# Install from source
uv pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
# Or use Docker
docker pull lmsysorg/sglang:nightly-dev-20260310-0fd9a57d
```
For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/installation).
## 3. Model Deployment
This section provides a progressive guide from quick deployment to performance tuning.
### 3.1 Basic Configuration
**Interactive Command Generator**: select hardware, tensor parallelism, and common knobs to generate a launch command.
<Nemotron3SuperDeployment />
### 3.2 Configuration Tips
- **Attention backend**:
**H200**: Use flash attention 3 backend by default.
**B200**: Use flashinfer backend by default.
- **TP support**:
To set tp size, use `--tp <2|4|8>`.
- **FP8 KV cache**:
To enable fp8 kv cache, please append `--kv-cache-dtype fp8_e4m3`.
## 4. Model Invocation
```shell Command
python3 -m sglang.launch_server \
--model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
--host 0.0.0.0 \
--port 5000 \
--trust-remote-code \
--tp 4 \
--tool-call-parser qwen3_coder \
--reasoning-parser nemotron_3
```
### 4.1 Basic Usage (OpenAI-Compatible API)
SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:5000/v1",
api_key="EMPTY",
)
resp = client.chat.completions.create(
model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16",
messages=[
{"role": "system", "content": "You are a helpful AI assistant."},
{"role": "user", "content": "Give me 3 bullet points about SGLang."},
],
temperature=0.6,
max_tokens=1024,
)
print("Reasoning:", resp.choices[0].message.reasoning_content, "\nContent:", resp.choices[0].message.content)
print("\n")
```
Output:
```text Output
Reasoning: Okay, the user is asking for 3 bullet points about SGLang. Let me recall what I know about SGLang. It's a framework for serving large language models, right? Developed by the team at UC Berkeley and others.
First, I should verify the key features. SGLang is known for its high-performance serving capabilities, especially with features like Radix Attention and chunked prefill. Those are important points to mention...(more tokens)
Content: - SGLang introduces **Radix Attention**, an innovative attention mechanism that significantly reduces KV cache memory usage and improves computational efficiency during LLM serving by reusing intermediate states across tokens.
- It features **chunked prefill** for handling long prompts efficiently, breaking input sequences into manageable chunks to minimize latency and memory pressure while maintaining high throughput.
- Designed for **high-performance LLM serving**, SGLang achieves superior throughput and lower latency compared to traditional systems (like vLLM or TensorRT-LLM) through optimized kernel fusion, dynamic batching, and seamless integration with Hugging Face Transformers.
```
Streaming chat completion:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:5000/v1",
api_key="EMPTY",
)
stream = client.chat.completions.create(
model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16",
messages=[
{"role": "system", "content": "You are a helpful AI assistant."},
{"role": "user", "content": "What are the first 5 prime numbers?"}
],
temperature=0.7,
max_tokens=1024,
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta
if delta and delta.content:
print(delta.content, end="", flush=True)
```
Output:
```text Output
The first 5 prime numbers are:
**2, 3, 5, 7, 11**.
### Explanation:
- A **prime number** is a natural number greater than 1 that has no positive divisors other than 1 and itself.
- **2** is the smallest and only even prime number.
- **3** is prime (divisible only by 1 and 3).
- **4** is not prime (divisible by 2).
- **5** is prime.
- **6** is not prime (divisible by 2 and 3).
- **7** is prime.
- **8, 9, 10** are not prime.
- **11** is prime (the fifth in the sequence).
Note: **1 is not considered a prime number** by definition, as it has only one positive divisor.
This list is universally accepted in mathematics. Let me know if you'd like to explore more primes or related concepts! 😊
```
### 4.2 Reasoning
The model supports two modes — Reasoning ON (default) vs OFF. This can be toggled by setting `enable_thinking` to `False`, as shown below.
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:5000/v1",
api_key="EMPTY",
)
# Reasoning on (default)
print("Reasoning on")
resp = client.chat.completions.create(
model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Write a haiku about GPUs. Please make thinking process short."}
],
temperature=1,
max_tokens=1024,
)
print(f"Reasoning: \n{resp.choices[0].message.reasoning_content[:200]}... \nContent: \n{resp.choices[0].message.content[:200]}...")
print("\n")
# Reasoning off
print("Reasoning off")
resp = client.chat.completions.create(
model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Give me 3 facts about SGLang."}
],
temperature=0,
max_tokens=256,
extra_body={"chat_template_kwargs": {"enable_thinking": False}}
)
print(f"Content: \n{resp.choices[0].message.reasoning_content[:200]}...")
```
Output:
```text Output
Reasoning on
Reasoning:
We need to output a haiku about GPUs, with short thinking process. Probably we just need to produce the haiku. No extra commentary needed. Provide a haiku: 5-7-5 syllable lines about GPUs.
Let's deci...
Content:
Silicon hearts beat
Paint vivid worlds with bright light
GPU dreams rise...
Reasoning off
Content:
Certainly! Here are three accurate and informative facts about **SGLang**:
1. **SGLang is a high-performance serving system for large language models (LLMs)**
Developed by researchers at UC Berk...
```
### 4.3 Tool Calling
Call functions using the OpenAI Tools schema and inspect returned `tool_calls`.
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:5000/v1",
api_key="EMPTY",
)
# Tool calling via OpenAI tools schema
TOOLS = [
{
"type": "function",
"function": {
"name": "calculate_tip",
"parameters": {
"type": "object",
"properties": {
"bill_total": {
"type": "integer",
"description": "The total amount of the bill"
},
"tip_percentage": {
"type": "integer",
"description": "The percentage of tip to be applied"
}
},
"required": ["bill_total", "tip_percentage"]
}
}
}
]
completion = client.chat.completions.create(
model="nemotron",
messages=[
{"role": "system", "content": ""},
{"role": "user", "content": "My bill is $50. What will be the amount for 15% tip?"}
],
tools=TOOLS,
temperature=0.6,
top_p=0.95,
max_tokens=512,
stream=False
)
print(completion.choices[0].message.reasoning_content)
print(completion.choices[0].message.tool_calls)
```
Output:
```text Output
The user wants to calculate a 15% tip on a $50 bill. I have a function called calculate_tip that takes bill_total and tip_percentage as parameters. The bill_total is $50, and tip_percentage is 15. I need to call the function with these values. Let me do that.
[ChatCompletionMessageFunctionToolCall(id='call_ced9a83a3baa448e9d587aaf', function=Function(arguments='{"bill_total": 50, "tip_percentage": 15}', name='calculate_tip'), type='function', index=0)]
```
### 4.4 Controlling Reasoning Budget
The `reasoning_budget` parameter allows you to limit the length of the model's reasoning trace. When the reasoning output reaches the specified token budget, the model will attempt to gracefully end the reasoning at the next newline character.
If no newline is encountered within 500 tokens after reaching the budget threshold, the reasoning trace will be forcibly terminated at `reasoning_budget + 500` tokens.
```python Example
from typing import Any, Dict, List
import openai
from transformers import AutoTokenizer
class ThinkingBudgetClient:
def __init__(self, base_url: str, api_key: str, tokenizer_name_or_path: str):
self.base_url = base_url
self.api_key = api_key
self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_name_or_path)
self.client = openai.OpenAI(base_url=self.base_url, api_key=self.api_key)
def chat_completion(
self,
model: str,
messages: List[Dict[str, Any]],
reasoning_budget: int = 512,
max_tokens: int = 1024,
**kwargs,
) -> Dict[str, Any]:
assert (
max_tokens > reasoning_budget
), f"reasoning_budget must be smaller than max_tokens. Given {max_tokens=} and {reasoning_budget=}"
# 1. first call chat completion to get reasoning content
response = self.client.chat.completions.create(
model=model,
messages=messages,
max_tokens=reasoning_budget,
**kwargs
)
reasoning_content = response.choices[0].message.reasoning_content or ""
if "</think>" not in reasoning_content:
# reasoning content is too long, closed with a period (.)
reasoning_content = f"{reasoning_content}.\n</think>\n\n"
reasoning_tokens_used = len(
self.tokenizer.encode(reasoning_content, add_special_tokens=False)
)
remaining_tokens = max_tokens - reasoning_tokens_used
assert (
remaining_tokens > 0
), f"remaining tokens must be positive. Given {remaining_tokens=}. Increase max_tokens or lower reasoning_budget."
# 2. append reasoning content to messages and call completion
messages.append({"role": "assistant", "content": reasoning_content})
prompt = self.tokenizer.apply_chat_template(
messages,
tokenize=False,
continue_final_message=True,
)
response = self.client.completions.create(
model=model,
prompt=prompt,
max_tokens=remaining_tokens,
**kwargs
)
response_data = {
"reasoning_content": reasoning_content.strip().strip("</think>").strip(),
"content": response.choices[0].text,
"finish_reason": response.choices[0].finish_reason,
}
return response_data
```
Usage example with `reasoning_budget=128`:
```python Example
SERVED_MODEL_NAME = "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16"
# Client
client = ThinkingBudgetClient(
base_url="http://127.0.0.1:5000/v1",
api_key="null",
tokenizer_name_or_path=SERVED_MODEL_NAME
)
resp = client.chat_completion(
model=SERVED_MODEL_NAME,
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Write a haiku about GPUs."}
],
temperature=1,
max_tokens=512,
reasoning_budget=128
)
print("Reasoning:", resp["reasoning_content"], "\nContent:", resp["content"])
```
Output:
```text Output
Reasoning: Okay, the user wants a haiku about GPUs. Let me recall what a haiku is: a traditional Japanese poem with three lines, 5-7-5 syllable structure. So I need to make sure the syllable count is exact.
First, I should think about what makes GPUs interesting. They're used for graphics rendering, parallel processing, AI, gaming, etc. Maybe focus on their speed, power, or how they handle many tasks at once.
Let me brainstorm some words and phrases related to GPUs: silicon, cores, transistors, parallel, rendering, pixels, frames per second, CUDA, tensor.
Content:
Silicon minds awaken,
Thousands of cores hum in unison—
Lightning paints the void.
```
---
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: H200 (4x)
- Model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
- Tensor Parallelism: 4
- SGLang Version: main branch
- Model Deployment Command:
```shell Command
python3 -m sglang.launch_server \
--model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
--trust-remote-code \
--tp 4 \
--max-running-requests 1024 \
--host 0.0.0.0 \
--port 5000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 5000 \
--model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
--dataset-name random \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 4096 \
--max-concurrency 256
```
- **Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 256
Successful requests: 4096
Benchmark duration (s): 623.49
Total input tokens: 2081726
Total input text tokens: 2081726
Total generated tokens: 2087288
Total generated tokens (retokenized): 2044666
Request throughput (req/s): 6.57
Input token throughput (tok/s): 3338.85
Output token throughput (tok/s): 3347.77
Peak output token throughput (tok/s): 6349.00
Peak concurrent requests: 270
Total token throughput (tok/s): 6686.62
Concurrency: 250.35
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 38108.46
Median E2E Latency (ms): 37186.80
P90 E2E Latency (ms): 69325.24
P99 E2E Latency (ms): 77776.90
---------------Time to First Token----------------
Mean TTFT (ms): 436.49
Median TTFT (ms): 114.90
P99 TTFT (ms): 6938.11
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 75.02
Median TPOT (ms): 76.02
P99 TPOT (ms): 92.27
---------------Inter-Token Latency----------------
Mean ITL (ms): 74.07
Median ITL (ms): 38.45
P95 ITL (ms): 230.42
P99 ITL (ms): 242.70
Max ITL (ms): 7181.72
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 GSM8K Benchmark
**Environment**
- Hardware: H200 (4x)
- Model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
- Tensor Parallelism: 4
- SGLang Version: main branch
**Launch Model**
```bash Command
python3 -m sglang.launch_server \
--model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
--trust-remote-code \
--tp 4 \
--reasoning-parser nemotron_3
```
**Run Benchmark**
```bash Command
python3 benchmark/gsm8k/bench_sglang.py --port 5000
```
**Test Results:**
```text Output
Accuracy: 0.950
Invalid: 0.000
Latency: 21.442 s
Output throughput: 996.815 token/s
```
#### 5.2.2 MMLU Benchmark
**Run Benchmark**
```bash Command
python3 benchmark/mmlu/bench_sglang.py --port 5000
```
**Test Results:**
```text Output
subject: abstract_algebra, #q:100, acc: 0.730
subject: anatomy, #q:135, acc: 0.830
subject: astronomy, #q:152, acc: 0.934
subject: business_ethics, #q:100, acc: 0.830
subject: clinical_knowledge, #q:265, acc: 0.879
subject: college_biology, #q:144, acc: 0.931
subject: college_chemistry, #q:100, acc: 0.620
subject: college_computer_science, #q:100, acc: 0.840
subject: college_mathematics, #q:100, acc: 0.820
subject: college_medicine, #q:173, acc: 0.821
subject: college_physics, #q:102, acc: 0.794
subject: computer_security, #q:100, acc: 0.880
subject: conceptual_physics, #q:235, acc: 0.919
subject: econometrics, #q:114, acc: 0.746
subject: electrical_engineering, #q:145, acc: 0.828
subject: elementary_mathematics, #q:378, acc: 0.926
subject: formal_logic, #q:126, acc: 0.857
subject: global_facts, #q:100, acc: 0.570
subject: high_school_biology, #q:310, acc: 0.952
subject: high_school_chemistry, #q:203, acc: 0.828
subject: high_school_computer_science, #q:100, acc: 0.940
subject: high_school_european_history, #q:165, acc: 0.861
subject: high_school_geography, #q:198, acc: 0.939
subject: high_school_government_and_politics, #q:193, acc: 0.990
subject: high_school_macroeconomics, #q:390, acc: 0.928
subject: high_school_mathematics, #q:270, acc: 0.700
subject: high_school_microeconomics, #q:238, acc: 0.966
subject: high_school_physics, #q:151, acc: 0.834
subject: high_school_psychology, #q:545, acc: 0.960
subject: high_school_statistics, #q:216, acc: 0.852
subject: high_school_us_history, #q:204, acc: 0.926
subject: high_school_world_history, #q:237, acc: 0.937
subject: human_aging, #q:223, acc: 0.879
subject: human_sexuality, #q:131, acc: 0.939
subject: international_law, #q:121, acc: 0.934
subject: jurisprudence, #q:108, acc: 0.898
subject: logical_fallacies, #q:163, acc: 0.914
subject: machine_learning, #q:112, acc: 0.821
subject: management, #q:103, acc: 0.903
subject: marketing, #q:234, acc: 0.944
subject: medical_genetics, #q:100, acc: 0.980
subject: miscellaneous, #q:783, acc: 0.945
subject: moral_disputes, #q:346, acc: 0.861
subject: moral_scenarios, #q:895, acc: 0.542
subject: nutrition, #q:306, acc: 0.902
subject: philosophy, #q:311, acc: 0.884
subject: prehistory, #q:324, acc: 0.920
subject: professional_accounting, #q:282, acc: 0.805
subject: professional_law, #q:1534, acc: 0.681
subject: professional_medicine, #q:272, acc: 0.923
subject: professional_psychology, #q:612, acc: 0.889
subject: public_relations, #q:110, acc: 0.800
subject: security_studies, #q:245, acc: 0.837
subject: sociology, #q:201, acc: 0.960
subject: us_foreign_policy, #q:100, acc: 0.920
subject: virology, #q:166, acc: 0.590
subject: world_religions, #q:171, acc: 0.906
Total latency: 150.267
Average accuracy: 0.841
```
docs_new/cookbook/autoregressive/OpenAI/GPT-OSS.mdx
+572
View File
@@ -0,0 +1,572 @@
---
title: GPT-OSS
metatags:
description: "Deploy GPT-OSS (20B/120B) with SGLang - configurable reasoning, full chain-of-thought, MXFP4 quantization for single GPU deployment."
---
## 1.Model Introduction
[GPT-OSS](https://huggingface.co/openai/gpt-oss-20b) is an advanced large language model developed by OpenAI designed for power reasoning, agentic tasks, and versatile developer use cases. It has versions with two model sizes.
- **gpt-oss-120b** — for production, general purpose, high reasoning use cases that fit into a single 80GB GPU (like NVIDIA H100 80GB or AMD MI300X 192GB) (117B parameters with 5.1B active parameters)
- **gpt-oss-20b** — for lower latency, and local or specialized use cases (21B parameters with 3.6B active parameters)
GPT-OSS introduces several groundbreaking innovations:
- **Configurable reasoning effort**: Easily adjust the reasoning effort (low, medium, high) based on your specific use case and latency needs.
- **Full chain-of-thought**: Gain complete access to the model’s reasoning process, facilitating easier debugging and increased trust in outputs. It’s not intended to be shown to end users.
- **Fine-tunable**: Fully customize models to your specific use case through parameter fine-tuning.
- **Agentic capabilities**: Use the models’ native capabilities for function calling, web browsing, Python code execution, and Structured Outputs.
- **MXFP4 quantization**: The models were post-trained with MXFP4 quantization of the MoE weights, making gpt-oss-120b run on a single 80GB GPU (like NVIDIA H100 80GB or AMD MI300X 192GB) and the gpt-oss-20b model run within 16GB of memory. All evals were performed with the same MXFP4 quantization.
## 2.SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3.Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
The GPT-OSS series comes in two sizes. Recommended starting configurations vary depending on hardware.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.
import { GPTOSSDeployment } from "/src/snippets/autoregressive/gpt-oss-deployment.jsx";
<GPTOSSDeployment />
### 3.2 Configuration Tips
For more detailed configuration tips, please refer to [GPS-OSS Usage](../../../docs/basic_usage/gpt_oss).
## 4.Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
GPT-OSS supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
```shell Command
python -m sglang.launch_server \
--model openai/gpt-oss-120b \
--reasoning-parser gpt-oss \
--tp 8
```
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="openai/gpt-oss-120b",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
The user asks: "Solve this problem step by step: What is 15% of 240?" So we need to provide step-by-step solution. Compute 15% of 240: 0.15 * 240 = 36. Provide steps: convert percent to decimal, multiply, maybe use fraction. Provide answer.
=============== Content =================
**Step‑by‑step solution**
1. **Understand what “percent” means**
“15 %” means 15 out of every 100 parts, i.e. the fraction \(\displaystyle \frac{15}{100}\).
2. **Convert the percent to a decimal (or fraction)**
\[
\frac{15}{100}=0.15
\]
3. **Set up the multiplication**
To find 15 % of 240 we multiply 240 by the decimal 0.15:
\[
240 \times 0.15
\]
4. **Do the multiplication**
One convenient way is to break it into two easier parts:
\[
240 \times 0.15 = 240 \times \left(\frac{15}{100}\right)
= \frac{240 \times 15}{100}
\]
- First compute \(240 \times 15\):
\[
240 \times 15 = 240 \times (10 + 5) = 2400 + 1200 = 3600
\]
- Then divide by 100:
\[
\frac{3600}{100} = 36
\]
5. **Write the result**
\[
15\% \text{ of } 240 = 36
\]
---
**Answer:** \(36\)
```
#### 4.2.2 Tool Calling
GPT-OSS supports tool calling capabilities. Enable the tool call parser:
**Python Example (without Thinking Process):**
Start sglang server:
```shell Command
python -m sglang.launch_server \
--model openai/gpt-oss-120b \
--tool-call-parser gpt-oss \
--tp 8
```
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="openai/gpt-oss-120b",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
if tool_call.function:
print(f"🔧 Tool Call: {tool_call.function.name}")
print(f" Arguments: {tool_call.function.arguments}")
# Print content
if delta.content:
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
🔧 Tool Call: get_weather
Arguments: {"location": "Beijing", "unit": "celsius"}
```
**Python Example (with Thinking Process):**
Start sglang server:
```shell Command
python -m sglang.launch_server \
--model openai/gpt-oss-120b \
--reasoning-parser gpt-oss \
--tool-call-parser gpt-oss \
--tp 8
```
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="openai/gpt-oss-120b",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
if tool_call.function:
print(f"🔧 Tool Call: {tool_call.function.name}")
print(f" Arguments: {tool_call.function.arguments}")
# Print content
if delta.content:
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
User asks: "What's the weather in Beijing?" We need to get current weather. Use function get_weather with location "Beijing". No unit specified; default? Probably use default (maybe Celsius). We can specify unit as "celsius". We'll call function.
=============== Content =================
🔧 Tool Call: get_weather
Arguments: {"location": "Beijing", "unit": "celsius"}
```
**Note:**
- The reasoning parser shows how the model decides to use a tool
- Tool calls are clearly marked with the function name and arguments
- You can then execute the function and send the result back to continue the conversation
**Handling Tool Call Results:**
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="openai/gpt-oss-120b",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The current weather in Beijing is 22 °C and sunny. Let me know if you’d like a forecast for the next few days or any other details!"
```
## 5.Benchmark
### 5.1 Speed Benchmark
- Hardware: NVIDIA B200 GPU (8x)
- Tensor Parallelism: 8
- Model: openai/gpt-oss-120b
- sglang version: 0.5.6
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.
#### 5.1.1 Latency-Sensitive Benchmark
- Server Command:
```shell Command
python -m sglang.launch_server \
--model openai/gpt-oss-120b \
--tp 8
```
- Test Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--num-prompt 100 \
--max-concurrency 1
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 100
Benchmark duration (s): 52.35
Total input tokens: 33178
Total input text tokens: 33178
Total input vision tokens: 0
Total generated tokens: 21251
Total generated tokens (retokenized): 20868
Request throughput (req/s): 1.91
Input token throughput (tok/s): 633.76
Output token throughput (tok/s): 405.93
Peak output token throughput (tok/s): 433.00
Peak concurrent requests: 8
Total token throughput (tok/s): 1039.69
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 523.30
Median E2E Latency (ms): 389.91
---------------Time to First Token----------------
Mean TTFT (ms): 33.71
Median TTFT (ms): 31.79
P99 TTFT (ms): 108.98
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 2.31
Median TPOT (ms): 2.31
P99 TPOT (ms): 2.39
---------------Inter-Token Latency----------------
Mean ITL (ms): 2.31
Median ITL (ms): 2.31
P95 ITL (ms): 2.35
P99 ITL (ms): 2.38
Max ITL (ms): 3.54
==================================================
```
#### 5.1.2 Throughput-Sensitive Benchmark
- Server Command:
```shell Command
python -m sglang.launch_server \
--model openai/gpt-oss-120b \
--tp 8
```
- Test Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--num-prompt 1000 \
--max-concurrency 100
```
**Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 24.76
Total input tokens: 297156
Total input text tokens: 297156
Total input vision tokens: 0
Total generated tokens: 192432
Total generated tokens (retokenized): 187145
Request throughput (req/s): 40.39
Input token throughput (tok/s): 12003.57
Output token throughput (tok/s): 7773.26
Peak output token throughput (tok/s): 13780.00
Peak concurrent requests: 156
Total token throughput (tok/s): 19776.83
Concurrency: 89.23
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 2208.97
Median E2E Latency (ms): 1591.11
---------------Time to First Token----------------
Mean TTFT (ms): 102.94
Median TTFT (ms): 31.53
P99 TTFT (ms): 674.32
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 14.31
Median TPOT (ms): 11.00
P99 TPOT (ms): 91.28
---------------Inter-Token Latency----------------
Mean ITL (ms): 11.00
Median ITL (ms): 5.75
P95 ITL (ms): 25.35
P99 ITL (ms): 43.18
Max ITL (ms): 621.42
==================================================
```
### 5.2 Accuracy Benchmark
### 5.2.1 GSM8K Benchmark
- **Benchmark Command:**
```shell Command
python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000
```
- **Results**:
- GPT-OSS-120b
```text Output
Accuracy: 0.880
Invalid: 0.005
Latency: 5.262 s
Output throughput: 12143.675 token/s
```
- GPT-OSS-20b
```text Output
Accuracy: 0.535
Invalid: 0.165
Latency: 4.157 s
Output throughput: 19589.165 token/s
```
docs_new/cookbook/autoregressive/Qwen/Qwen2.5-VL.mdx
+392
View File
@@ -0,0 +1,392 @@
---
title: Qwen2.5-VL
metatags:
description: "Deploy Qwen2.5-VL vision-language models with SGLang on AMD MI300X - available in 3B to 72B sizes with enhanced visual understanding."
---
import { Qwen25VLDeployment } from '/src/snippets/autoregressive/qwen25-vl-deployment.jsx';
## 1. Model Introduction
**[Qwen2.5-VL](https://huggingface.co/collections/Qwen/qwen25-vl)** is a vision-language model series from the Qwen team, offering significant improvements over its predecessor in understanding, reasoning, and multi-modal processing.
**Key Features:**
- **Understand things visually**: Proficient in recognizing common objects such as flowers, birds, fish, and insects, and it is highly capable of analyzing texts, charts, icons, graphics, and layouts within images.
- **More Agentic**: Play as a visual agent that can reason and dynamically direct tools, which is capable of computer use and phone use.
- **Understanding long videos and capturing events**: Supports comprehending videos of over 1 hour, and this time it has a new ability of capturing event by pinpointing the relevant video segments.
- **Capable of visual localization in different formats**: Accurately localize objects in an image by generating bounding boxes or points, and it can provide stable JSON outputs for coordinates and attributes.
- **Generating structured outputs**: Supports structured outputs of the contents, benefiting usages in finance, commerce, etc for data like scans of invoices, forms, tables, etc.
- **Dynamic Resolution and Frame Rate Training for Video Understanding**: Extend dynamic resolution to the temporal dimension by adopting dynamic FPS sampling, enabling the model to comprehend videos at various sampling rates. Accordingly, we update mRoPE in the time dimension with IDs and absolute time alignment, enabling the model to learn temporal sequence and speed, and ultimately acquire the ability to pinpoint specific moments.
- **Multiple Sizes**: Available in 3B, 7B, 32B, and 72B variants to suit different deployment needs.
- **ROCm Support**: Compatible with AMD MI300X, MI325X and MI355X GPUs via SGLang (verified).
For more details, please refer to the [official Qwen2.5-VL GitHub Repository](https://github.com/QwenLM/Qwen3-VL).
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for AMD MI300X, MI325X and MI355X hardware platforms and different use cases.
### 3.1 Basic Configuration
The Qwen2.5-VL series offers models in various sizes. The following configurations have been verified on AMD MI300X, MI325X and MI355X GPUs.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model size.
<Qwen25VLDeployment />
### 3.2 Configuration Tips
* **Memory Management**: For the 72B model on MI300X/MI325X/MI355X, we have verified successful deployment with `--context-length 128000`. Smaller context lengths can be used to reduce memory usage if needed.
* **Multi-GPU Deployment**: Use Tensor Parallelism (`--tp`) to scale across multiple GPUs. For example, use `--tp 8` for the 72B model and `--tp 2` for the 32B model on MI300X/MI325X/MI355X.
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
### 4.2 Advanced Usage
#### 4.2.1 Multi-Modal Inputs
Qwen2.5-VL supports image inputs. Here's a basic example with single image input:
```python Example
import time
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:30000/v1",
timeout=3600
)
messages = [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
}
},
{
"type": "text",
"text": "Read all the text in the image."
}
]
}
]
start = time.time()
response = client.chat.completions.create(
model="Qwen/Qwen2.5-VL-7B-Instruct",
messages=messages,
max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```
**Example Output:**
```text Output
Response costs: 2.31s
Generated text: Auntie Anne's
CINNAMON SUGAR
1 x 17,000
SUB TOTAL
17,000
GRAND TOTAL
17,000
CASH IDR
20,000
CHANGE DUE
3,000
```
**Multi-Image Input Example:**
Qwen2.5-VL can process multiple images in a single request for comparison or analysis:
```python Example
import time
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:30000/v1",
timeout=3600
)
messages = [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
}
},
{
"type": "image_url",
"image_url": {
"url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
}
},
{
"type": "text",
"text": "Compare these two images and describe the differences in 100 words or less."
}
]
}
]
start = time.time()
response = client.chat.completions.create(
model="Qwen/Qwen2.5-VL-7B-Instruct",
messages=messages,
max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```
**Example Output:**
```text Output
Response costs: 13.79s
Generated text: The first image shows a single red taxi driving on a street with a few other taxis in the background. The second image shows a large number of taxis parked in a lot, with some appearing to be in various states of repair. The first image has a single taxi with a visible license plate, while the second image has multiple taxis with different license plates. The first image has a clear view of the street and surrounding area, while the second image is taken from an elevated perspective, showing a wider view of the parking lot and the surrounding area.
```
**Note:**
- You can also provide local file paths using `file://` protocol.
- For larger images, you may need more memory, adjust `--mem-fraction-static` accordingly.
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: AMD MI300X GPU (8x)
- Model: Qwen2.5-VL-72B-Instruct
- Tensor Parallelism: 8
- SGLang Version: 0.5.6
We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. To simulate real-world usage, you can specify different input and output lengths for each request. For example, each request can have 128 input tokens, two 720p images, and 1024 output tokens.
#### 5.1.1 Latency-Sensitive Benchmark
- Model Deployment Command:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen2.5-VL-72B-Instruct \
--tp 8 \
--host 0.0.0.0 \
--port 30000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang-oai-chat \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen2.5-VL-72B-Instruct \
--dataset-name image \
--image-count 2 \
--image-resolution 720p \
--random-input-len 128 \
--random-output-len 1024 \
--num-prompts 10 \
--max-concurrency 1
```
#### 5.1.2 Throughput-Sensitive Benchmark
- Model Deployment Command:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen2.5-VL-72B-Instruct \
--tp 8 \
--host 0.0.0.0 \
--port 30000
```
- Result:
```text Output
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 37.99
Total input tokens: 24781
Total input text tokens: 821
Total input vision tokens: 23960
Total generated tokens: 4220
Total generated tokens (retokenized): 2365
Request throughput (req/s): 0.26
Input token throughput (tok/s): 652.26
Output token throughput (tok/s): 111.07
Peak output token throughput (tok/s): 128.00
Peak concurrent requests: 2
Total token throughput (tok/s): 763.34
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 3797.61
Median E2E Latency (ms): 3140.90
P90 E2E Latency (ms): 6545.54
P99 E2E Latency (ms): 7939.56
---------------Time to First Token----------------
Mean TTFT (ms): 504.45
Median TTFT (ms): 510.93
P99 TTFT (ms): 521.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 7.82
Median TPOT (ms): 7.82
P99 TPOT (ms): 7.84
---------------Inter-Token Latency----------------
Mean ITL (ms): 10.07
Median ITL (ms): 7.90
P95 ITL (ms): 15.79
P99 ITL (ms): 15.93
Max ITL (ms): 23.60
==================================================
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang-oai-chat \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen2.5-VL-72B-Instruct \
--dataset-name image \
--image-count 2 \
--image-resolution 720p \
--random-input-len 128 \
--random-output-len 1024 \
--num-prompts 1000 \
--max-concurrency 100
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 454.68
Total input tokens: 2481865
Total input text tokens: 85865
Total input vision tokens: 2396000
Total generated tokens: 510855
Total generated tokens (retokenized): 296466
Request throughput (req/s): 2.20
Input token throughput (tok/s): 5458.50
Output token throughput (tok/s): 1123.55
Peak output token throughput (tok/s): 5004.00
Peak concurrent requests: 106
Total token throughput (tok/s): 6582.05
Concurrency: 98.63
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 44844.92
Median E2E Latency (ms): 42866.15
P90 E2E Latency (ms): 82798.20
P99 E2E Latency (ms): 106306.30
---------------Time to First Token----------------
Mean TTFT (ms): 4507.79
Median TTFT (ms): 1180.83
P99 TTFT (ms): 39975.22
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 80.26
Median TPOT (ms): 82.38
P99 TPOT (ms): 152.89
---------------Inter-Token Latency----------------
Mean ITL (ms): 100.66
Median ITL (ms): 13.26
P95 ITL (ms): 428.45
P99 ITL (ms): 1393.35
Max ITL (ms): 31943.26
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 MMMU Benchmark
You can evaluate the model's accuracy using the MMMU dataset:
- Benchmark Command:
```shell Command
python3 benchmark/mmmu/bench_sglang.py \
--port 30000 \
--concurrency 64
```
```text Output
Benchmark time: 97.75084622902796
answers saved to: ./answer_sglang.json
Evaluating...
answers saved to: ./answer_sglang.json
{'Accounting': {'acc': 0.633, 'num': 30},
'Agriculture': {'acc': 0.5, 'num': 30},
'Architecture_and_Engineering': {'acc': 0.367, 'num': 30},
'Art': {'acc': 0.767, 'num': 30},
'Art_Theory': {'acc': 0.9, 'num': 30},
'Basic_Medical_Science': {'acc': 0.7, 'num': 30},
'Biology': {'acc': 0.467, 'num': 30},
'Chemistry': {'acc': 0.433, 'num': 30},
'Clinical_Medicine': {'acc': 0.733, 'num': 30},
'Computer_Science': {'acc': 0.567, 'num': 30},
'Design': {'acc': 0.833, 'num': 30},
'Diagnostics_and_Laboratory_Medicine': {'acc': 0.467, 'num': 30},
'Economics': {'acc': 0.767, 'num': 30},
'Electronics': {'acc': 0.433, 'num': 30},
'Energy_and_Power': {'acc': 0.467, 'num': 30},
'Finance': {'acc': 0.533, 'num': 30},
'Geography': {'acc': 0.633, 'num': 30},
'History': {'acc': 0.7, 'num': 30},
'Literature': {'acc': 0.867, 'num': 30},
'Manage': {'acc': 0.633, 'num': 30},
'Marketing': {'acc': 0.733, 'num': 30},
'Materials': {'acc': 0.333, 'num': 30},
'Math': {'acc': 0.533, 'num': 30},
'Mechanical_Engineering': {'acc': 0.433, 'num': 30},
'Music': {'acc': 0.367, 'num': 30},
'Overall': {'acc': 0.62, 'num': 900},
'Overall-Art and Design': {'acc': 0.717, 'num': 120},
'Overall-Business': {'acc': 0.66, 'num': 150},
'Overall-Health and Medicine': {'acc': 0.693, 'num': 150},
'Overall-Humanities and Social Science': {'acc': 0.775, 'num': 120},
'Overall-Science': {'acc': 0.553, 'num': 150},
'Overall-Tech and Engineering': {'acc': 0.443, 'num': 210},
'Pharmacy': {'acc': 0.833, 'num': 30},
'Physics': {'acc': 0.7, 'num': 30},
'Psychology': {'acc': 0.767, 'num': 30},
'Public_Health': {'acc': 0.733, 'num': 30},
'Sociology': {'acc': 0.767, 'num': 30}}
eval out saved to ./val_sglang.json
Overall accuracy: 0.62
```
docs_new/cookbook/autoregressive/Qwen/Qwen3-Coder-Next.mdx
+902
View File
@@ -0,0 +1,902 @@
---
title: Qwen3-Coder-Next
metatags:
description: "Deploy Qwen3-Coder-Next code-focused models with SGLang on AMD MI300X - available in 3B to 80B sizes with enhanced code understanding."
---
import { Qwen3CoderNextDeployment } from '/src/snippets/autoregressive/qwen3-coder-next-deployment.jsx';
## 1. Model Introduction
[Qwen3-Coder-Next](https://huggingface.co/Qwen/Qwen3-Coder-Next) is a cost-efficient code-focused language model from the Qwen team (Alibaba). With 80B total parameters but only 3B activated parameters, it achieves performance comparable to models with 10–20x more active parameters through its innovative hybrid architecture.
**Key Features:**
- **Hybrid Architecture**: Uses a 48-layer hybrid layout combining Gated DeltaNet and Gated Attention with Mixture-of-Experts (512 total experts, 10 activated, 1 shared), enabling exceptional efficiency.
- **Tool Calling Support**: Advanced agentic capabilities with native support for function calling and tool use via the `qwen3_coder` parser.
- **Extended Context Length**: Supports up to 256K tokens for processing large codebases and long documents.
- **Cost-Efficient Inference**: Only 3B parameters activated per token, making it ideal for local development and cost-effective deployment at scale.
- **IDE Integration**: Compatible with Claude Code, Qwen Code, Cline, and other IDE platforms.
For more details, please refer to the [Qwen3-Coder-Next model card](https://huggingface.co/Qwen/Qwen3-Coder-Next).
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
**Note:** Qwen3-Coder-Next requires SGLang v0.5.8 or later.
## 3. Model Deployment
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and deployment options.
<Qwen3CoderNextDeployment />
### 3.2 Configuration Tips
- **Context Length**: The model supports up to 256K tokens natively. If you encounter OOM issues, try `--context-length 32768`.
- **Tool Use**: To enable tool calling capabilities, use the `--tool-call-parser qwen3_coder` flag.
- **Sampling Parameters**: SGLang automatically applies the recommended sampling parameters from the model's `generation_config.json`. No manual configuration is needed.
- **Mamba Radix Cache**: Qwen3-Coder-Next's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-scheduler-strategy`:
- **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage.
- **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-scheduler-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64).
## 4. Model Invocation
**Deployment Command:**
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-Coder-Next \
--tp 2 \
--tool-call-parser qwen3_coder \
--host 0.0.0.0 \
--port 30000
```
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Advanced Usage
#### 4.2.1 Code Generation Example
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="Qwen/Qwen3-Coder-Next",
messages=[
{"role": "user", "content": "Write a Python function that implements binary search on a sorted list. Include type hints."}
],
max_tokens=2048
)
print(response.choices[0].message.content)
```
**Example Output:**
````text Output
Here's a Python function implementing binary search on a sorted list, with comprehensive type hints:
```python
from typing import Sequence, TypeVar, Optional
T = TypeVar('T')
def binary_search(sorted_list: Sequence[T], target: T) -> Optional[int]:
"""
Perform binary search on a sorted list to find the index of a target element.
Args:
sorted_list: A sequence (e.g., list, tuple) sorted in ascending order.
target: The element to search for in the list.
Returns:
The index of the target element if found, or None if not found.
Time Complexity: O(log n)
Space Complexity: O(1)
Note:
The function assumes the list is sorted in ascending order.
If the list contains duplicate elements, it returns the index of one of them.
"""
left = 0
right = len(sorted_list) - 1
while left <= right:
mid = (left + right) // 2
mid_val = sorted_list[mid]
if mid_val == target:
return mid
elif mid_val < target:
left = mid + 1
else:
right = mid - 1
return None
```
### Example usage:
```python
# Example 1: Finding an existing element
numbers = [1, 3, 5, 7, 9, 11]
print(binary_search(numbers, 7)) # Output: 3
# Example 2: Element not in the list
print(binary_search(numbers, 4)) # Output: None
# Example 3: Empty list
print(binary_search([], 5)) # Output: None
# Example 4: Single element
print(binary_search([1], 1)) # Output: 0
print(binary_search([1], 2)) # Output: None
```
### Key features:
- Uses `TypeVar` to support generic types (as long as comparison operations are defined)
- Returns `Optional[int]` to indicate either the index or no match found
- Uses `Sequence[T]` to accept any sequence type (list, tuple, etc.)
- Includes comprehensive docstring with time/space complexity
- Implements standard iterative binary search for O(1) space complexity
````
#### 4.2.2 Streaming Example
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="Qwen/Qwen3-Coder-Next",
messages=[
{"role": "user", "content": "Explain the difference between a stack and a queue in 3 sentences."}
],
max_tokens=512,
stream=True
)
for chunk in response:
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
print()
```
**Example Output:**
```text Output
A **stack** follows the **Last In, First Out (LIFO)** principle, meaning the last element added is the first one removed—operations like `push` (add) and `pop` (remove) occur at the same end, called the *top*. In contrast, a **queue** follows the **First In, First Out (FIFO)** principle, where elements are added at the *back* (enqueue) and removed from the *front* (dequeue), preserving the order of insertion. This structural difference makes stacks ideal for tasks like function call management and expression evaluation, while queues suit scheduling, buffering, and breadth-first traversal.
```
#### 4.2.3 Tool Calling Example
Qwen3-Coder-Next supports tool calling capabilities. Make sure `--tool-call-parser qwen3_coder` is included in the deployment command above.
**Python Example:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "execute_code",
"description": "Execute Python code and return the result",
"parameters": {
"type": "object",
"properties": {
"code": {
"type": "string",
"description": "The Python code to execute"
}
},
"required": ["code"]
}
}
}
]
response = client.chat.completions.create(
model="Qwen/Qwen3-Coder-Next",
messages=[
{"role": "user", "content": "Calculate the factorial of 10 using Python"}
],
tools=tools
)
# Check if the model wants to call a tool
if response.choices[0].message.tool_calls:
tool_call = response.choices[0].message.tool_calls[0]
print(f"Tool: {tool_call.function.name}")
print(f"Arguments: {tool_call.function.arguments}")
else:
print(response.choices[0].message.content)
```
**Example Output:**
```text Output
Tool: execute_code
Arguments: {"code": "import math\nmath.factorial(10)"}
```
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: NVIDIA B200 GPU (2x)
- Model: Qwen/Qwen3-Coder-Next
- Tensor Parallelism: 2
- sglang version: 0.5.8+
#### 5.1.1 Standard Scenario Benchmark
- Model Deployment Command:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-Coder-Next \
--tp 2 \
--host 0.0.0.0 \
--port 30000
```
##### 5.1.1.1 Low Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen3-Coder-Next \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1
```
- Result:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 27.86
Total input tokens: 6101
Total input text tokens: 6101
Total generated tokens: 4220
Total generated tokens (retokenized): 4218
Request throughput (req/s): 0.36
Input token throughput (tok/s): 219.00
Output token throughput (tok/s): 151.48
Peak output token throughput (tok/s): 166.00
Peak concurrent requests: 2
Total token throughput (tok/s): 370.48
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 2784.14
Median E2E Latency (ms): 2258.08
P90 E2E Latency (ms): 5044.43
P99 E2E Latency (ms): 6130.52
---------------Time to First Token----------------
Mean TTFT (ms): 161.68
Median TTFT (ms): 168.09
P99 TTFT (ms): 183.26
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 6.19
Median TPOT (ms): 6.23
P99 TPOT (ms): 6.32
---------------Inter-Token Latency----------------
Mean ITL (ms): 6.23
Median ITL (ms): 6.23
P95 ITL (ms): 6.51
P99 ITL (ms): 6.64
Max ITL (ms): 13.45
==================================================
```
##### 5.1.1.2 Medium Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen3-Coder-Next \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16
```
- Result:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 39.06
Total input tokens: 39668
Total input text tokens: 39668
Total generated tokens: 40805
Total generated tokens (retokenized): 40789
Request throughput (req/s): 2.05
Input token throughput (tok/s): 1015.62
Output token throughput (tok/s): 1044.73
Peak output token throughput (tok/s): 1664.00
Peak concurrent requests: 21
Total token throughput (tok/s): 2060.34
Concurrency: 14.16
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 6910.97
Median E2E Latency (ms): 7248.27
P90 E2E Latency (ms): 11612.63
P99 E2E Latency (ms): 13933.91
---------------Time to First Token----------------
Mean TTFT (ms): 183.48
Median TTFT (ms): 156.50
P99 TTFT (ms): 311.46
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 13.61
Median TPOT (ms): 13.59
P99 TPOT (ms): 21.11
---------------Inter-Token Latency----------------
Mean ITL (ms): 13.22
Median ITL (ms): 9.76
P95 ITL (ms): 10.43
P99 ITL (ms): 158.04
Max ITL (ms): 394.39
==================================================
```
##### 5.1.1.3 High Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen3-Coder-Next \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100
```
- Result:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 102.81
Total input tokens: 249831
Total input text tokens: 249831
Total generated tokens: 252662
Total generated tokens (retokenized): 252536
Request throughput (req/s): 4.86
Input token throughput (tok/s): 2429.99
Output token throughput (tok/s): 2457.53
Peak output token throughput (tok/s): 5299.00
Peak concurrent requests: 109
Total token throughput (tok/s): 4887.52
Concurrency: 94.28
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 19385.20
Median E2E Latency (ms): 17584.09
P90 E2E Latency (ms): 36762.15
P99 E2E Latency (ms): 42518.35
---------------Time to First Token----------------
Mean TTFT (ms): 270.62
Median TTFT (ms): 159.65
P99 TTFT (ms): 938.90
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 38.57
Median TPOT (ms): 41.78
P99 TPOT (ms): 53.28
---------------Inter-Token Latency----------------
Mean ITL (ms): 37.90
Median ITL (ms): 18.26
P95 ITL (ms): 167.82
P99 ITL (ms): 311.45
Max ITL (ms): 993.20
==================================================
```
#### 5.1.2 Reasoning Scenario Benchmark
- Model Deployment Command:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-Coder-Next \
--tp 2 \
--host 0.0.0.0 \
--port 30000
```
##### 5.1.2.1 Low Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen3-Coder-Next \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 10 \
--max-concurrency 1
```
- Result:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 285.02
Total input tokens: 6101
Total input text tokens: 6101
Total generated tokens: 44462
Total generated tokens (retokenized): 44432
Request throughput (req/s): 0.04
Input token throughput (tok/s): 21.41
Output token throughput (tok/s): 156.00
Peak output token throughput (tok/s): 173.00
Peak concurrent requests: 2
Total token throughput (tok/s): 177.40
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 28499.54
Median E2E Latency (ms): 30424.65
P90 E2E Latency (ms): 49132.26
P99 E2E Latency (ms): 51075.28
---------------Time to First Token----------------
Mean TTFT (ms): 95.51
Median TTFT (ms): 93.86
P99 TTFT (ms): 112.56
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 6.24
Median TPOT (ms): 6.30
P99 TPOT (ms): 6.60
---------------Inter-Token Latency----------------
Mean ITL (ms): 6.39
Median ITL (ms): 6.34
P95 ITL (ms): 7.16
P99 ITL (ms): 7.42
Max ITL (ms): 12.48
==================================================
```
##### 5.1.2.2 Medium Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen3-Coder-Next \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 80 \
--max-concurrency 16
```
- Result:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 237.77
Total input tokens: 39668
Total input text tokens: 39668
Total generated tokens: 318306
Total generated tokens (retokenized): 315646
Request throughput (req/s): 0.34
Input token throughput (tok/s): 166.83
Output token throughput (tok/s): 1338.72
Peak output token throughput (tok/s): 1727.00
Peak concurrent requests: 19
Total token throughput (tok/s): 1505.55
Concurrency: 13.88
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 41266.21
Median E2E Latency (ms): 41010.10
P90 E2E Latency (ms): 77574.22
P99 E2E Latency (ms): 82688.04
---------------Time to First Token----------------
Mean TTFT (ms): 140.73
Median TTFT (ms): 84.52
P99 TTFT (ms): 365.86
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 10.32
Median TPOT (ms): 10.38
P99 TPOT (ms): 10.87
---------------Inter-Token Latency----------------
Mean ITL (ms): 10.34
Median ITL (ms): 10.19
P95 ITL (ms): 10.75
P99 ITL (ms): 11.18
Max ITL (ms): 206.79
==================================================
```
##### 5.1.2.3 High Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen3-Coder-Next \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 320 \
--max-concurrency 64
```
- Result:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 384.82
Total input tokens: 158939
Total input text tokens: 158939
Total generated tokens: 1301025
Total generated tokens (retokenized): 1299908
Request throughput (req/s): 0.83
Input token throughput (tok/s): 413.02
Output token throughput (tok/s): 3380.83
Peak output token throughput (tok/s): 4317.00
Peak concurrent requests: 69
Total token throughput (tok/s): 3793.85
Concurrency: 56.42
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 67847.54
Median E2E Latency (ms): 70724.38
P90 E2E Latency (ms): 120888.83
P99 E2E Latency (ms): 133234.48
---------------Time to First Token----------------
Mean TTFT (ms): 212.24
Median TTFT (ms): 115.96
P99 TTFT (ms): 652.93
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 16.76
Median TPOT (ms): 16.99
P99 TPOT (ms): 18.18
---------------Inter-Token Latency----------------
Mean ITL (ms): 16.64
Median ITL (ms): 15.83
P95 ITL (ms): 31.64
P99 ITL (ms): 90.85
Max ITL (ms): 576.60
==================================================
```
#### 5.1.3 Summarization Scenario Benchmark
##### 5.1.3.1 Low Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen3-Coder-Next \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1
```
- Result:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 29.42
Total input tokens: 41941
Total input text tokens: 41941
Total generated tokens: 4220
Total generated tokens (retokenized): 4220
Request throughput (req/s): 0.34
Input token throughput (tok/s): 1425.35
Output token throughput (tok/s): 143.42
Peak output token throughput (tok/s): 169.00
Peak concurrent requests: 3
Total token throughput (tok/s): 1568.77
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 2941.19
Median E2E Latency (ms): 2411.84
P90 E2E Latency (ms): 5661.26
P99 E2E Latency (ms): 6497.45
---------------Time to First Token----------------
Mean TTFT (ms): 139.46
Median TTFT (ms): 160.33
P99 TTFT (ms): 184.30
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 6.56
Median TPOT (ms): 6.65
P99 TPOT (ms): 7.29
---------------Inter-Token Latency----------------
Mean ITL (ms): 6.65
Median ITL (ms): 6.68
P95 ITL (ms): 7.39
P99 ITL (ms): 7.51
Max ITL (ms): 16.34
==================================================
```
##### 5.1.3.2 Medium Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen3-Coder-Next \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16
```
- Result:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 41.62
Total input tokens: 300020
Total input text tokens: 300020
Total generated tokens: 41669
Total generated tokens (retokenized): 41664
Request throughput (req/s): 1.92
Input token throughput (tok/s): 7208.67
Output token throughput (tok/s): 1001.19
Peak output token throughput (tok/s): 1536.00
Peak concurrent requests: 21
Total token throughput (tok/s): 8209.86
Concurrency: 14.27
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 7421.29
Median E2E Latency (ms): 7985.77
P90 E2E Latency (ms): 12122.09
P99 E2E Latency (ms): 14595.05
---------------Time to First Token----------------
Mean TTFT (ms): 248.49
Median TTFT (ms): 179.25
P99 TTFT (ms): 915.90
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 14.13
Median TPOT (ms): 14.28
P99 TPOT (ms): 24.02
---------------Inter-Token Latency----------------
Mean ITL (ms): 13.80
Median ITL (ms): 10.46
P95 ITL (ms): 11.00
P99 ITL (ms): 173.14
Max ITL (ms): 823.32
==================================================
```
##### 5.1.3.3 High Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen3-Coder-Next \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 320 \
--max-concurrency 64
```
- Result:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 85.74
Total input tokens: 1273893
Total input text tokens: 1273893
Total generated tokens: 170000
Total generated tokens (retokenized): 169983
Request throughput (req/s): 3.73
Input token throughput (tok/s): 14858.12
Output token throughput (tok/s): 1982.80
Peak output token throughput (tok/s): 3734.00
Peak concurrent requests: 70
Total token throughput (tok/s): 16840.92
Concurrency: 59.75
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 16008.12
Median E2E Latency (ms): 15460.65
P90 E2E Latency (ms): 27705.81
P99 E2E Latency (ms): 32874.74
---------------Time to First Token----------------
Mean TTFT (ms): 476.99
Median TTFT (ms): 177.50
P99 TTFT (ms): 3014.39
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 29.81
Median TPOT (ms): 31.19
P99 TPOT (ms): 45.53
---------------Inter-Token Latency----------------
Mean ITL (ms): 29.29
Median ITL (ms): 15.75
P95 ITL (ms): 173.94
P99 ITL (ms): 202.00
Max ITL (ms): 2783.23
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 GSM8K Benchmark
- **Benchmark Command:**
```shell Command
python benchmark/gsm8k/bench_sglang.py --port 30000
```
- **Test Results:**
```text Output
Accuracy: 0.965
Invalid: 0.000
Latency: 26.407 s
Output throughput: 929.132 token/s
```
#### 5.2.2 MMLU Benchmark
- **Benchmark Command:**
```shell Command
cd benchmark/mmlu
bash download_data.sh
python3 bench_sglang.py --port 30000
```
- **Test Results:**
```text Output
subject: abstract_algebra, #q:100, acc: 0.780
subject: anatomy, #q:135, acc: 0.807
subject: astronomy, #q:152, acc: 0.921
subject: business_ethics, #q:100, acc: 0.820
subject: clinical_knowledge, #q:265, acc: 0.860
subject: college_biology, #q:144, acc: 0.944
subject: college_chemistry, #q:100, acc: 0.590
subject: college_computer_science, #q:100, acc: 0.820
subject: college_mathematics, #q:100, acc: 0.800
subject: college_medicine, #q:173, acc: 0.803
subject: college_physics, #q:102, acc: 0.775
subject: computer_security, #q:100, acc: 0.880
subject: conceptual_physics, #q:235, acc: 0.936
subject: econometrics, #q:114, acc: 0.807
subject: electrical_engineering, #q:145, acc: 0.834
subject: elementary_mathematics, #q:378, acc: 0.854
subject: formal_logic, #q:126, acc: 0.802
subject: global_facts, #q:100, acc: 0.610
subject: high_school_biology, #q:310, acc: 0.971
subject: high_school_chemistry, #q:203, acc: 0.803
subject: high_school_computer_science, #q:100, acc: 0.920
subject: high_school_european_history, #q:165, acc: 0.891
subject: high_school_geography, #q:198, acc: 0.929
subject: high_school_government_and_politics, #q:193, acc: 0.969
subject: high_school_macroeconomics, #q:390, acc: 0.903
subject: high_school_mathematics, #q:270, acc: 0.689
subject: high_school_microeconomics, #q:238, acc: 0.962
subject: high_school_physics, #q:151, acc: 0.854
subject: high_school_psychology, #q:545, acc: 0.947
subject: high_school_statistics, #q:216, acc: 0.815
subject: high_school_us_history, #q:204, acc: 0.907
subject: high_school_world_history, #q:237, acc: 0.937
subject: human_aging, #q:223, acc: 0.821
subject: human_sexuality, #q:131, acc: 0.840
subject: international_law, #q:121, acc: 0.934
subject: jurisprudence, #q:108, acc: 0.870
subject: logical_fallacies, #q:163, acc: 0.847
subject: machine_learning, #q:112, acc: 0.812
subject: management, #q:103, acc: 0.922
subject: marketing, #q:234, acc: 0.923
subject: medical_genetics, #q:100, acc: 0.970
subject: miscellaneous, #q:783, acc: 0.941
subject: moral_disputes, #q:346, acc: 0.850
subject: moral_scenarios, #q:895, acc: 0.726
subject: nutrition, #q:306, acc: 0.915
subject: philosophy, #q:311, acc: 0.859
subject: prehistory, #q:324, acc: 0.889
subject: professional_accounting, #q:282, acc: 0.723
subject: professional_law, #q:1534, acc: 0.648
subject: professional_medicine, #q:272, acc: 0.923
subject: professional_psychology, #q:612, acc: 0.845
subject: public_relations, #q:110, acc: 0.782
subject: security_studies, #q:245, acc: 0.796
subject: sociology, #q:201, acc: 0.925
subject: us_foreign_policy, #q:100, acc: 0.950
subject: virology, #q:166, acc: 0.572
subject: world_religions, #q:171, acc: 0.883
Total latency: 208.985
Average accuracy: 0.834
```
docs_new/cookbook/autoregressive/Qwen/Qwen3-Coder.mdx
+520
View File
@@ -0,0 +1,520 @@
---
title: Qwen3-Coder
metatags:
description: "Deploy Qwen3-Coder(480B, 30B) MoE coding model with SGLang on AMD MI300X (MI325X, MI355X)"
---
import { Qwen3CoderDeployment } from '/src/snippets/autoregressive/qwen3-coder-deployment.jsx';
## 1. Model Introduction
[Qwen3-Coder](https://huggingface.co/collections/Qwen/qwen3-coder) is the latest code-focused large language model series from the Qwen team. Built on the foundation of Qwen3, Qwen3-Coder delivers exceptional performance in code generation, understanding, and reasoning tasks.
**Key Features:**
- **State-of-the-art Coding Performance**: Achieves top-tier results on HumanEval, MBPP, LiveCodeBench, and other major coding benchmarks.
- **Tool Calling Support**: Native support for function calling and tool use, enabling seamless integration with external APIs and services.
- **Extended Context Length**: Supports up to 256K tokens for processing large codebases and long documents.
- **Multilingual Code Support**: Proficient in Python, JavaScript, TypeScript, Java, C++, Go, Rust, and many other programming languages.
- **MoE Architecture**: Efficient Mixture-of-Experts design for optimal performance-to-cost ratio.
- **ROCm Support**: Compatible with AMD MI300X, MI325X and MI355X GPUs via SGLang (verified).
- **NVIDIA GPU Support**: Compatible with NVIDIA GB200 and B200 GPUs via SGLang (verified).
For more details, please refer to the [official Qwen3-Coder GitHub Repository](https://github.com/QwenLM/Qwen3-Coder).
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations verified on AMD MI300X, MI325X, MI355X and NVIDIA B200, GB200 hardware platforms.
### 3.1 Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, and quantization method.
<Qwen3CoderDeployment />
### 3.2 Configuration Tips
**AMD (MI300X/MI325X/MI355X):**
* **Memory Management**: We have verified successful deployment on MI300X/MI325X/MI355X with `--context-length 8192`. Larger context lengths may be supported but require additional memory.
* **Expert Parallelism**: For 480B-A35B with FP8 quantization, `--ep 2` is required to satisfy the dimension alignment requirement.
* **Page Size**: `--page-size 32` is recommended for MoE models to optimize memory usage.
* **Environment Variable**: If you encounter aiter-related issues, try setting `SGLANG_USE_AITER=0`.
**NVIDIA (B200/GB200):**
* **MOE Runner Backend**: FP8 uses `--moe-runner-backend triton`, NVFP4 uses `--moe-runner-backend flashinfer_cutlass`.
* **NVFP4 Quantization**: Requires `--quantization modelopt_fp4` and uses a different model path (`nvidia/Qwen3-Coder-...`).
* **DP Attention**: NVFP4 configuration supports `--enable-dp-attention` for improved throughput.
**General:**
* **Tool Use**: To enable tool calling capabilities, add `--tool-call-parser qwen3_coder` to the launch command.
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Advanced Usage
#### 4.2.1 Code Generation Example
```python Example
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:30000/v1",
timeout=3600
)
messages = [
{
"role": "user",
"content": "Write a Python function that implements binary search on a sorted list. Include docstring and type hints."
}
]
response = client.chat.completions.create(
model="Qwen/Qwen3-Coder-480B-A35B-Instruct",
messages=messages,
max_tokens=2048,
temperature=0.7
)
print(response.choices[0].message.content)
```
**Example Output:**
````text Output
```python
from typing import List, Optional, TypeVar
T = TypeVar('T')
def binary_search(arr: List[T], target: T) -> Optional[int]:
"""
Perform binary search on a sorted list to find the index of a target element.
This function implements the binary search algorithm, which efficiently finds
a target value in a sorted array by repeatedly dividing the search interval
in half.
Args:
arr (List[T]): A sorted list of elements to search through.
target (T): The element to search for in the list.
Returns:
Optional[int]: The index of the target element if found, None otherwise.
Time Complexity:
O(log n) where n is the number of elements in the array.
Space Complexity:
O(1) - iterative implementation uses constant extra space.
Examples:
>>> binary_search([1, 2, 3, 4, 5], 3)
2
>>> binary_search([1, 2, 3, 4, 5], 6)
None
>>> binary_search(['a', 'b', 'c', 'd'], 'b')
1
>>> binary_search([], 1)
None
"""
if not arr:
return None
left: int = 0
right: int = len(arr) - 1
while left <= right:
mid: int = (left + right) // 2
if arr[mid] == target:
return mid
elif arr[mid] < target:
left = mid + 1
else:
right = mid - 1
return None
# Alternative recursive implementation
def binary_search_recursive(arr: List[T], target: T, left: int = 0, right: Optional[int] = None) -> Optional[int]:
"""
Perform binary search recursively on a sorted list to find the index of a target element.
Args:
arr (List[T]): A sorted list of elements to search through.
target (T): The element to search for in the list.
left (int): Left boundary of the search range (inclusive).
right (Optional[int]): Right boundary of the search range (inclusive).
Returns:
Optional[int]: The index of the target element if found, None otherwise.
Time Complexity:
O(log n) where n is the number of elements in the array.
Space Complexity:
O(log n) due to recursive call stack.
Examples:
>>> binary_search_recursive([1, 2, 3, 4, 5], 3)
2
>>> binary_search_recursive([1, 2, 3, 4, 5], 6)
None
"""
if not arr:
return None
if right is None:
right = len(arr) - 1
if left > right:
return None
mid: int = (left + right) // 2
if arr[mid] == target:
return mid
elif arr[mid] < target:
return binary_search_recursive(arr, target, mid + 1, right)
else:
return binary_search_recursive(arr, target, left, mid - 1)
```
This implementation provides:
1. **Main function** (`binary_search`): An iterative implementation that's more memory-efficient
2. **Alternative function** (`binary_search_recursive`): A recursive implementation for educational purposes
3. **Type hints**: Using generics (`TypeVar`) to work with any comparable type
4. **Comprehensive docstring**: Including description, parameters, return value, complexity analysis, and examples
5. **Edge case handling**: Empty lists, elements not found, etc.
6. **Clear variable names**: Self-documenting code
7. **Examples**: Doctest-style examples in the docstring
The function works with any sorted list of comparable elements (integers, strings, etc.) and returns the index of the target element if found, or `None` if not found.
````
#### 4.2.2 Tool Calling Example
Qwen3-Coder supports tool calling capabilities. Enable the tool call parser during deployment. The following example uses 30B-A3B model:
```shell Command
SGLANG_USE_AITER=0 python -m sglang.launch_server \
--model Qwen/Qwen3-Coder-30B-A3B-Instruct \
--tp 1 \
--context-length 8192 \
--page-size 32 \
--tool-call-parser qwen3_coder
```
**Python Example:**
```python Example
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:30000/v1",
timeout=3600
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "execute_code",
"description": "Execute Python code and return the result",
"parameters": {
"type": "object",
"properties": {
"code": {
"type": "string",
"description": "The Python code to execute"
}
},
"required": ["code"]
}
}
}
]
response = client.chat.completions.create(
model="Qwen/Qwen3-Coder-30B-A3B-Instruct",
messages=[
{"role": "user", "content": "Calculate the factorial of 10 using Python"}
],
tools=tools,
temperature=0.7
)
# Check if the model wants to call a tool
if response.choices[0].message.tool_calls:
tool_call = response.choices[0].message.tool_calls[0]
print(f"Tool: {tool_call.function.name}")
print(f"Arguments: {tool_call.function.arguments}")
else:
# Model may return tool call in content format
print(response.choices[0].message.content)
```
**Example Output:**
```text Output
Tool: execute_code
Arguments: {"code": "def factorial(n):\n if n == 0 or n == 1:\n return 1\n else:\n return n * factorial(n-1)\n\nresult = factorial(10)\nresult"}
```
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: AMD MI300X GPU (8x)
- Model: Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8
- Tensor Parallelism: 8
- Expert Parallelism: 2
- sglang version: 0.5.7
We use SGLang's built-in benchmarking tool to conduct performance evaluation with random dataset.
#### 5.1.1 Standard Scenario Benchmark
- Model Deployment Command:
```shell Command
SGLANG_USE_AITER=0 python -m sglang.launch_server \
--model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
--tp 8 \
--ep 2 \
--context-length 8192 \
--page-size 32 \
--trust-remote-code
```
##### 5.1.1.1 Low Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 73.79
Total input tokens: 6101
Total input text tokens: 6101
Total generated tokens: 4220
Total generated tokens (retokenized): 4104
Request throughput (req/s): 0.14
Input token throughput (tok/s): 82.68
Output token throughput (tok/s): 57.19
Peak output token throughput (tok/s): 59.00
Peak concurrent requests: 2
Total token throughput (tok/s): 139.86
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 7376.26
Median E2E Latency (ms): 5851.51
P90 E2E Latency (ms): 13351.89
P99 E2E Latency (ms): 16908.32
---------------Time to First Token----------------
Mean TTFT (ms): 191.93
Median TTFT (ms): 126.06
P99 TTFT (ms): 662.15
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 17.06
Median TPOT (ms): 17.07
P99 TPOT (ms): 17.08
---------------Inter-Token Latency----------------
Mean ITL (ms): 17.06
Median ITL (ms): 17.06
P95 ITL (ms): 17.14
P99 ITL (ms): 17.19
Max ITL (ms): 18.53
==================================================
```
##### 5.1.1.2 Medium Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 87.04
Total input tokens: 39668
Total input text tokens: 39668
Total generated tokens: 40805
Total generated tokens (retokenized): 40364
Request throughput (req/s): 0.92
Input token throughput (tok/s): 455.77
Output token throughput (tok/s): 468.83
Peak output token throughput (tok/s): 608.00
Peak concurrent requests: 20
Total token throughput (tok/s): 924.59
Concurrency: 13.76
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 14966.88
Median E2E Latency (ms): 15871.93
P90 E2E Latency (ms): 24983.41
P99 E2E Latency (ms): 29504.85
---------------Time to First Token----------------
Mean TTFT (ms): 388.94
Median TTFT (ms): 157.49
P99 TTFT (ms): 1318.63
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 29.41
Median TPOT (ms): 29.22
P99 TPOT (ms): 43.48
---------------Inter-Token Latency----------------
Mean ITL (ms): 28.64
Median ITL (ms): 26.42
P95 ITL (ms): 27.51
P99 ITL (ms): 131.63
Max ITL (ms): 995.11
==================================================
```
##### 5.1.1.3 High Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 320 \
--max-concurrency 64
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 177.82
Total input tokens: 158939
Total input text tokens: 158939
Total generated tokens: 170134
Total generated tokens (retokenized): 168387
Request throughput (req/s): 1.80
Input token throughput (tok/s): 893.84
Output token throughput (tok/s): 956.80
Peak output token throughput (tok/s): 1728.00
Peak concurrent requests: 70
Total token throughput (tok/s): 1850.64
Concurrency: 58.88
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 32716.53
Median E2E Latency (ms): 30896.37
P90 E2E Latency (ms): 65605.24
P99 E2E Latency (ms): 80970.63
---------------Time to First Token----------------
Mean TTFT (ms): 372.97
Median TTFT (ms): 181.67
P99 TTFT (ms): 529.01
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 62.98
Median TPOT (ms): 50.44
P99 TPOT (ms): 204.24
---------------Inter-Token Latency----------------
Mean ITL (ms): 60.95
Median ITL (ms): 37.87
P95 ITL (ms): 143.98
P99 ITL (ms): 148.02
Max ITL (ms): 36863.32
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 GSM8K Benchmark
- **Benchmark Command:**
```shell Command
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
```
##### AMD (MI300X/MI325X/MI355X)
- **Results**:
- Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8
```
Accuracy: 0.965
Invalid: 0.000
Latency: 23.084 s
Output throughput: 1148.425 token/s
```
##### NVIDIA (B200/GB200)
For deployment commands, see [Section 3.1](#31-configuration).
- Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 (tp=8, ep=2)
```
Accuracy: 0.950
Invalid: 0.000
Latency: 12.914 s
Output throughput: 2065.515 token/s
```
- nvidia/Qwen3-Coder-480B-A35B-Instruct-NVFP (NVFP4, tp=8, ep=1)
```
Accuracy: 0.970
Invalid: 0.000
Latency: 71.280 s
Output throughput: 390.080 token/s
```
docs_new/cookbook/autoregressive/Qwen/Qwen3-Next.mdx
+774
View File
@@ -0,0 +1,774 @@
---
title: Qwen3-Next
metatags:
description: "Deploy Qwen3-Next with SGLang - hybrid attention architecture supporting 262K context, 80B MoE with 3B active parameters, and multi-token prediction."
---
import { Qwen3NextDeployment } from '/src/snippets/autoregressive/qwen3-next-deployment.jsx';
## 1. Model Introduction
[Qwen3-Next](https://huggingface.co/collections/Qwen/qwen3-next) is an advanced large language model architecture developed by Alibaba's Qwen team, designed to enhance efficiency and performance in handling extensive contexts and large-scale parameters. It features advanced capabilities in reasoning, function calling, and multilingual understanding.
Qwen3-Next introduces several groundbreaking innovations:
- **Hybrid Attention Mechanism**: Replaces standard attention with a combination of **Gated DeltaNet** (linear attention) and **Full Attention**, enabling efficient processing of context lengths up to 262,144 tokens. This hybrid approach makes it ideal for analyzing lengthy documents such as entire books or contracts.
- **Highly Sparse Mixture-of-Experts (MoE)**: Features an 80-billion parameter architecture where only 3 billion parameters are active during inference. This design reduces computational costs by up to 90% while maintaining high performance, drastically reducing FLOPs per token without compromising model capacity.
- **Multi-Token Prediction (MTP)**: Enables generation of multiple tokens per inference step, significantly reducing latency and enhancing user experience in real-time applications. This innovation boosts both pretraining performance and inference speed.
- **Multilingual Support**: Natively supports 119 languages, facilitating seamless cross-lingual tasks and making it versatile for global applications.
- **Enterprise-Ready Deployment**: Released under the Apache 2.0 license, offering flexible deployment options including on-premises, virtual private cloud (VPC), and private cloud environments, ensuring security and compliance for enterprise use.
- **Advanced Reasoning & Stability**: Demonstrates clear improvement in reasoning performance with support for tool use during inference. Includes stability optimizations such as **zero-centered** and **weight-decayed layernorm** for robust pre-training and post-training.
For more details, please refer to the [official Qwen3-Next blog](https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd&from=research.latest-advancements-list).
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
The Qwen3-Next series comes in only one size but offers different thinking modes. Recommended starting configurations vary depending on hardware.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.
<Qwen3NextDeployment />
### 3.2 Configuration Tips
- `--max-mamba-cache-size`: Adjust `--max-mamba-cache-size` to increase mamba cache space and max running requests capability. It will decrease KV cache space as a trade-off. You can adjust it according to workload.
- `--mamba-ssm-dtype`: `bfloat16` or `float32`, use `bfloat16` to save mamba cache size and `float32` to get more accurate results. The default setting is `float32`.
- `--mamba-full-memory-ratio`: Adjust `--mamba-full-memory-ratio` to set the ratio of mamba state memory to full kv cache memory. The default setting is `0.9`.
- **Mamba Radix Cache**: Qwen3-Next's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-scheduler-strategy`:
- **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage.
- **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-scheduler-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64).
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
1. **Streaming with Thinking Process:**
Qwen3-Next-80B-A3B-Thinking only supports thinking mode. Enable the reasoning parser during deployment to separate the thinking and the content sections.
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-Next-80B-A3B-Thinking \
--reasoning-parser qwen3 \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="Qwen/Qwen3-Next-80B-A3B-Thinking",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
Okay, let's see. I need to find 15% of 240. Hmm, percentages. Right, "percent" means per hundred, so 15% is 15 per 100, or 15/100. To find a percentage of a number, I think you multiply the number by the percentage as a decimal. So first, maybe convert 15% to a decimal. To convert a percentage to a decimal, you divide by 100. So 15 divided by 100 is 0.15. Then, multiply that by 240. Let me check that. So 0.15 times 240. Let's calculate that. Maybe break it down. 10% of 240 is 24, because 10% is just moving the decimal one place left, so 240 becomes 24. Then 5% would be half of 10%, so half of 24 is 12. So 10% + 5% = 15%, so 24 + 12 = 36. Oh, that's another way to do it. Let me verify with the multiplication. 0.15 * 240. Let's do 240 * 0.1 = 24, 240 * 0.05 = 12, so 24 + 12 = 36. Yep, that works. Alternatively, 240 * 15 = 3600, then divide by 100, which is 36. Because 15% of 240 is (15/100)*240 = (15*240)/100. 15*240: 10*240=2400, 5*240=1200, so 2400+1200=3600. Then 3600/100=36. So that's 36. So the answer should be 36. Let me make sure. 15% of 240. If I take 240 and multiply by 0.15, 240*0.15. Let's compute 240*0.1=24, 240*0.05=12, so 24+12=36. Yep, that's right. So 15% of 240 is 36.
=============== Content =================
To find **15% of 240**, follow these steps:
---
### **Step 1: Understand what "percent" means**
- "Percent" means **per hundred**, so **15% = 15/100 = 0.15** in decimal form.
---
### **Step 2: Multiply the number by the decimal**
- To find 15% of 240, multiply:
$$
240 \times 0.15
$$
---
### **Step 3: Break it down for clarity (optional but helpful)**
- **10% of 240** = $ 240 \times 0.1 = 24 $
- **5% of 240** = $ 240 \times 0.05 = 12 $
- Add them together:
$$
24 + 12 = 36
$$
---
### **Step 4: Confirm with direct multiplication**
- $ 240 \times 0.15 = 36 $
---
### ✅ Final Answer:
$$
\boxed{36}
$$
```
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
2. **Turn off Thinking:**
Qwen3-Next-80B-A3B-Instruct only supports instruct (non-thinking) mode.
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-Next-80B-A3B-Instruct \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Turn off thinking process
response = client.chat.completions.create(
model="Qwen/Qwen3-Next-80B-A3B-Instruct",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
stream=True,
extra_body={"chat_template_kwargs": {"enable_thinking": False}}
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
To find **15% of 240**, follow these steps:
---
### **Step 1: Understand what percentage means**
"Percent" means "per hundred," so **15%** is the same as **15 per 100**, or the fraction:
$$
\frac{15}{100}
$$
---
### **Step 2: Multiply the fraction by the number**
To find 15% of 240, multiply:
$$
\frac{15}{100} \times 240
$$
---
### **Step 3: Simplify the multiplication**
You can simplify this in a couple of ways.
#### **Option A: Multiply first, then divide**
$$
15 \times 240 = 3600
$$
Then divide by 100:
$$
\frac{3600}{100} = 36
$$
#### **Option B: Simplify the fraction first**
$$
\frac{15}{100} = \frac{3}{20} \quad \text{(divided numerator and denominator by 5)}
$$
Now multiply:
$$
\frac{3}{20} \times 240 = \frac{3 \times 240}{20} = \frac{720}{20} = 36
$$
---
### **Step 4: Final Answer**
$$
\boxed{36}
$$
So, **15% of 240 is 36**.
```
#### 4.2.2 Tool Calling
Qwen/Qwen3-Next-80B-A3B-Instruct | Qwen/Qwen3-Next-80B-A3B-Thinking both support tool calling capabilities. Enable the tool call parser:
**Python Example (without Thinking Process):**
Start sglang server:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-Next-80B-A3B-Instruct \
--tool-call-parser qwen \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="Qwen/Qwen3-Next-80B-A3B-Instruct",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
if tool_call.function:
print(f"🔧 Tool Call: {tool_call.function.name}")
print(f" Arguments: {tool_call.function.arguments}")
# Print content
if delta.content:
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
<tool_call>
{"name": "get_weather", "arguments": {"location": "Beijing"}}
</tool_call>
```
**Python Example (with Thinking Process):**
Start sglang server:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-Next-80B-A3B-Thinking \
--reasoning-parser qwen3 \
--tool-call-parser qwen \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="Qwen/Qwen3-Next-80B-A3B-Thinking",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
if tool_call.function:
print(f"🔧 Tool Call: {tool_call.function.name}")
print(f" Arguments: {tool_call.function.arguments}")
# Print content
if delta.content:
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
Okay, the user is asking for the weather in Beijing. Let me check the available tools. There's a get_weather function that requires location and optionally unit. The location is needed, so I need to provide Beijing as the location. The unit is optional, but the user didn't specify Celsius or Fahrenheit. Since the default might be Celsius, but maybe I should check if the parameters require unit. Wait, the required field is only location, so unit is optional. So I can just call get_weather with location "Beijing" and not include the unit. Let me confirm the parameters. The parameters for get_weather have location as required, and unit is an enum with celsius or fahrenheit, but not required. So the correct call is to send location as Beijing, and omit unit. So the tool call should be {"name": "get_weather", "arguments": {"location": "Beijing"}}.
<tool_call>
{"name": "get_weather", "arguments": {"location": "Beijing"}}
</tool_call>
```
**Note:**
- The reasoning parser shows how the model decides to use a tool
- Tool calls are clearly marked with the function name and arguments
- You can then execute the function and send the result back to continue the conversation
**Handling Tool Call Results:**
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="Qwen/Qwen3-Next-80B-A3B-Thinking",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```
#### 4.2.3 Processing Ultra-Long Texts
Qwen3-Next natively supports context lengths of up to 262,144 tokens. For conversations where the total length (including both input and output) significantly exceeds this limit, we recommend using RoPE scaling techniques to handle long texts effectively. We have validated the model's performance on context lengths of up to 1 million tokens using the YaRN method.
**Qwen3-Next-80B-A3B-Instruct**
```shell Command
SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python -m sglang.launch_server --model Qwen/Qwen3-Next-80B-A3B-Instruct --tp 8 --host 0.0.0.0 --port 8000 --json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144}}' --context-length 1010000
```
**Qwen3-Next-80B-A3B-Thinking**
```shell Command
SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python -m sglang.launch_server --model Qwen/Qwen3-Next-80B-A3B-Thinking --reasoning-parser qwen3 --tp 8 --host 0.0.0.0 --port 8000 --json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144}}' --context-length 1010000
```
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: NVIDIA B200 GPU (8x)
- Tensor Parallelism: 8
- Model: Qwen/Qwen3-Next-80B-A3B-Instruct
- sglang version: 0.5.6
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.
#### 5.1.1 Latency-Sensitive Benchmark
- Server Command:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-Next-80B-A3B-Instruct \
--tp 8
```
- Test Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--num-prompt 100 \
--max-concurrency 1
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 100
Benchmark duration (s): 146.52
Total input tokens: 33839
Total input text tokens: 33839
Total input vision tokens: 0
Total generated tokens: 21640
Total generated tokens (retokenized): 21619
Request throughput (req/s): 0.68
Input token throughput (tok/s): 230.95
Output token throughput (tok/s): 147.70
Peak output token throughput (tok/s): 164.00
Peak concurrent requests: 6
Total token throughput (tok/s): 378.65
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 1464.81
Median E2E Latency (ms): 1077.48
---------------Time to First Token----------------
Mean TTFT (ms): 127.88
Median TTFT (ms): 132.88
P99 TTFT (ms): 212.85
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 6.19
Median TPOT (ms): 6.17
P99 TPOT (ms): 6.64
---------------Inter-Token Latency----------------
Mean ITL (ms): 6.21
Median ITL (ms): 6.16
P95 ITL (ms): 6.51
P99 ITL (ms): 6.71
Max ITL (ms): 10.07
==================================================
```
#### 5.1.2 Throughput-Sensitive Benchmark
- Server Command:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-Next-80B-A3B-Instruct \
--tp 8 \
```
- Test Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--num-prompt 1000 \
--max-concurrency 100
```
**Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 100.32
Total input tokens: 302118
Total input text tokens: 302118
Total input vision tokens: 0
Total generated tokens: 195775
Total generated tokens (retokenized): 195016
Request throughput (req/s): 9.97
Input token throughput (tok/s): 3011.69
Output token throughput (tok/s): 1951.60
Peak output token throughput (tok/s): 5909.00
Peak concurrent requests: 120
Total token throughput (tok/s): 4963.29
Concurrency: 93.05
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 9333.98
Median E2E Latency (ms): 6054.12
---------------Time to First Token----------------
Mean TTFT (ms): 161.77
Median TTFT (ms): 137.94
P99 TTFT (ms): 503.29
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 50.87
Median TPOT (ms): 50.28
P99 TPOT (ms): 122.87
---------------Inter-Token Latency----------------
Mean ITL (ms): 47.11
Median ITL (ms): 13.84
P95 ITL (ms): 195.33
P99 ITL (ms): 289.56
Max ITL (ms): 486.38
==================================================
```
### 5.2 Accuracy Benchmark
### 5.2.1 GSM8K Benchmark
- **Benchmark Command:**
```shell Command
python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000
```
- **Results**:
- Qwen3-Next-80B-A3B-Instruct
```
Accuracy: 0.960
Invalid: 0.000
Latency: 12.673 s
Output throughput: 2538.255 token/s
```
- Qwen3-Next-80B-A3B-Thinking
```
Accuracy: 0.935
Invalid: 0.000
Latency: 9.912 s
Output throughput: 3288.737 token/s
```
### 5.2.2 MMLU Benchmark
- **Benchmark Command:**
```shell Command
cd sglang
bash benchmark/mmlu/download_data.sh
python3 benchmark/mmlu/bench_sglang.py --nsub 10
```
- **Results**:
- Qwen3-Next-80B-A3B-Instruct
```
subject: abstract_algebra, #q:100, acc: 0.800
subject: anatomy, #q:135, acc: 0.807
subject: astronomy, #q:152, acc: 0.947
subject: business_ethics, #q:100, acc: 0.810
subject: clinical_knowledge, #q:265, acc: 0.894
subject: college_biology, #q:144, acc: 0.972
subject: college_chemistry, #q:100, acc: 0.680
subject: college_computer_science, #q:100, acc: 0.860
subject: college_mathematics, #q:100, acc: 0.780
subject: college_medicine, #q:173, acc: 0.861
Total latency: 10.098
Average accuracy: 0.856
```
- Qwen3-Next-80B-A3B-Thinking
```
subject: abstract_algebra, #q:100, acc: 0.780
subject: anatomy, #q:135, acc: 0.815
subject: astronomy, #q:152, acc: 0.941
subject: business_ethics, #q:100, acc: 0.870
subject: clinical_knowledge, #q:265, acc: 0.894
subject: college_biology, #q:144, acc: 0.965
subject: college_chemistry, #q:100, acc: 0.670
subject: college_computer_science, #q:100, acc: 0.840
subject: college_mathematics, #q:100, acc: 0.770
subject: college_medicine, #q:173, acc: 0.861
Total latency: 10.236
Average accuracy: 0.855
```
docs_new/cookbook/autoregressive/Qwen/Qwen3-VL.mdx
+777
View File
@@ -0,0 +1,777 @@
---
title: Qwen3-VL
metatags:
description: "Deploy Qwen3-VL vision-language models with SGLang - open model for text, 262K context, enhanced visual reasoning and agent capabilities."
---
## 1. Model Introduction
[Qwen3-VL series](https://github.com/QwenLM/Qwen3-VL) are the most powerful vision-language models in the Qwen series to date, featuring advanced capabilities in multi-modal understanding, reasoning, and agentic applications.
This generation delivers comprehensive upgrades across the board:
- **Superior text understanding & generation**: Qwen3-VL-235B-A22B-Instruct was ranked as the [#1 open model for text on lmarena.ai](https://x.com/arena/status/1973151703563460942)
- **Deeper visual perception & reasoning**: Enhanced image and video understanding capabilities.
- **Extended context length**: Supports up to 262K tokens for processing long documents and videos.
- **Enhanced spatial and video dynamics comprehension**: Better understanding of spatial relationships and temporal dynamics.
- **Stronger agent interaction capabilities**: Improved tool use and search-based agent performance.
- **Flexible deployment options**: Available in Dense and MoE architectures that scale from edge to cloud, with Instruct and reasoning-enhanced Thinking editions.
For more details, please refer to the [official Qwen3-VL GitHub Repository](https://github.com/QwenLM/Qwen3-VL).
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
The Qwen3-VL series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA and AMD GPUs. The recommended launch configurations vary by hardware and model size.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.
import { Qwen3VLDeployment } from "/src/snippets/autoregressive/qwen3-vl-deployment.jsx";
<Qwen3VLDeployment />
### 3.2 Configuration Tips
* **Multimodal attention backend** : Usually, `--mm-attention-backend` is default to `fa3` on H100/H200/A100 for better performance, but it is default to `triton_attn` on B200 for compatibility.
* **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size * number of images in current running requests.)
* **Memory Management** : Set lower `--context-length` to conserve memory. A value of `128000` is sufficient for most scenarios, down from the default 262K.
* **Expert Parallelism** : SGLang supports Expert Parallelism (EP) via `--ep`, allowing experts in MoE models to be deployed on separate GPUs for better throughput. One thing to note is that, for quantized models, you need to set `--ep` to a value that satisfies the requirement: `(moe_intermediate_size / moe_tp_size) % weight_block_size_n == 0, where moe_tp_size is equal to tp_size divided by ep_size.` Note that EP may perform worse in low concurrency scenarios due to additional communication overhead. Check out [Expert Parallelism Deployment](../../../docs/advanced_features/expert_parallelism) for more details.
* **Kernel Tuning** : For MoE Triton kernel tuning on your specific hardware, refer to [fused_moe_triton](https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton).
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
### 4.2 Advanced Usage
#### 4.2.1 Multi-Modal Inputs
Qwen3-VL supports both image and video inputs. Here's a basic example with image input:
```python Example
import time
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:8000/v1",
timeout=3600
)
messages = [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
}
},
{
"type": "text",
"text": "Read all the text in the image."
}
]
}
]
start = time.time()
response = client.chat.completions.create(
model="Qwen/Qwen3-VL-235B-A22B-Instruct",
messages=messages,
max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```
**Example Output:**
```text Output
Response costs: 3.37s
Generated text: Auntie Anne's
CINNAMON SUGAR
1 x 17,000 17,000
SUB TOTAL 17,000
GRAND TOTAL 17,000
CASH IDR 20,000
CHANGE DUE 3,000
```
**Multi-Image Input Example:**
Qwen3-VL can process multiple images in a single request for comparison or analysis:
```python Example
import time
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:8000/v1",
timeout=3600
)
messages = [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
}
},
{
"type": "image_url",
"image_url": {
"url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
}
},
{
"type": "text",
"text": "Compare these two images and describe the differences in 100 words or less. Focus on the key visual elements, colors, textures, and any notable contrasts between the two scenes. Be specific about what you see in each image."
}
]
}
]
start = time.time()
response = client.chat.completions.create(
model="Qwen/Qwen3-VL-235B-A22B-Instruct",
messages=messages,
max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```
**Example Output:**
```text Output
Response costs: 10.18s
Generated text: The two images present starkly different portrayals of Hong Kong’s iconic red taxis, contrasting a dynamic street-level moment with a static, large-scale gathering.
The first image is a close-up, eye-level shot capturing a single red Toyota Crown taxi (license plate RX 5004) in motion or paused at an urban intersection. Its glossy red paint gleams under daylight, reflecting the vibrant, cluttered backdrop of a Hong Kong street — neon signs, glass-fronted shops displaying sunglasses, and Chinese characters. The taxi’s chrome grille, clear headlights, and black trim provide visual contrast. A green “4 SEATS” sticker and a “的士 TAXI” sign on the side reinforce its identity. The composition is intimate, focusing on the vehicle’s details — the texture of its paint, the slight reflections on the windows, and the crispness of its license plate. Other red taxis flank it, suggesting a bustling city rhythm, but the central taxi dominates the frame, conveying movement and immediacy.
In contrast, the second image is an elevated, wide-angle shot of dozens of red taxis — along with a few green ones — parked in neat, grid-like rows on what appears to be a highway or staging area. The scene is static, almost ceremonial. Many taxis have their hoods open, suggesting maintenance, inspection, or protest. People are scattered among the vehicles, some inspecting engines, others conversing — adding a human, documentary element. The dominant color remains red, but the repetition creates a visual pattern rather than individual focus. The green taxis offer a subtle color contrast, hinting at different service zones (green for New Territories, red for urban areas). The setting is more utilitarian — concrete barriers, metal railings, and sparse vegetation — with an overpass looming in the background. The texture here is less about polished paint and more about the collective mass of vehicles, the asphalt, and the functional layout.
Key contrasts emerge: the first image is kinetic and personal, emphasizing the taxi as a working vehicle in the city’s daily flow; the second is static and collective, portraying the taxis as a fleet, possibly for logistical or political purposes. The lighting in both is bright daylight, but the first has richer color saturation and depth due to its proximity and urban backdrop, while the second feels flatter, more documentary in tone. The first image invites you into the city’s pulse; the second invites you to observe a system — organized, perhaps even paused — from a distance.
In essence, the first image celebrates the individual taxi in its natural habitat; the second reveals the scale and structure behind the fleet, transforming the familiar red icon into a symbol of coordination, maintenance, or collective action. Both are quintessentially Hong Kong, yet they offer vastly different narratives — one of motion and commerce, the other of assembly and purpose.
```
**Video Input Example:**
Qwen3-VL supports video understanding by processing video URLs:
```python Example
import time
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:8000/v1",
timeout=3600
)
messages = [
{
"role": "user",
"content": [
{
"type": "video_url",
"video_url": {
"url": "https://videos.pexels.com/video-files/4114797/4114797-uhd_3840_2160_25fps.mp4"
}
},
{
"type": "text",
"text": "Describe what happens in this video."
}
]
}
]
start = time.time()
response = client.chat.completions.create(
model="Qwen/Qwen3-VL-235B-A22B-Instruct",
messages=messages,
max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```
**Note:**
- For video processing, ensure you have sufficient context length configured (up to 262K tokens)
- Video processing may require more memory; adjust `--mem-fraction-static` accordingly
- You can also provide local file paths using `file://` protocol
**Example Output:**
```text Output
Response costs: 3.89s
Generated text: A person wearing blue gloves is using a microscope. They are adjusting the focus knob with one hand while holding a pipette with the other, suggesting they are preparing or examining a sample on the slide beneath the objective lens. The microscope's 40x objective lens is positioned over the slide, indicating a high-magnification observation. The person carefully manipulates the slide and the microscope controls, likely to achieve a clear view of the specimen.
```
#### 4.2.2 Reasoning Parser
Qwen3-VL-Thinking supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-VL-235B-A22B-Thinking \
--reasoning-parser qwen3 \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
**Streaming with Thinking Process:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="Qwen/Qwen3-VL-235B-A22B-Thinking",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
To solve this problem, I need to calculate 15% of 240.
Step 1: Convert 15% to decimal: 15% = 0.15
Step 2: Multiply 240 by 0.15
Step 3: 240 × 0.15 = 36
=============== Content =================
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
```
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
#### 4.2.3 Tool Calling
Qwen3-VL supports tool calling capabilities. Enable the tool call parser:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-VL-235B-A22B-Thinking \
--reasoning-parser qwen3 \
--tool-call-parser qwen \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
**Python Example (with Thinking Process):**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="Qwen/Qwen3-VL-235B-A22B-Thinking",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Accumulate tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================\n", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {
'name': None,
'arguments': ''
}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]['name'] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
# Print content
if delta.content:
print(delta.content, end="", flush=True)
# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"🔧 Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
print()
```
**Output Example:**
```text Output
=============== Thinking =================
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
I should call the function with location="Beijing".
=============== Content =================
🔧 Tool Call: get_weather
Arguments: {"location": "Beijing", "unit": "celsius"}
```
**Note:**
- The reasoning parser shows how the model decides to use a tool
- Tool calls are clearly marked with the function name and arguments
- You can then execute the function and send the result back to continue the conversation
**Handling Tool Call Results:**
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="Qwen/Qwen3-VL-235B-A22B-Thinking",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: NVIDIA B200 GPU (8x)
- Model: Qwen3-VL-235B-A22B-Instruct
- Tensor Parallelism: 8
- sglang version: 0.5.6
We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. To simulate real-world usage, you can specify different input and output lengths for each request. For example, each request can have 128 input tokens, two 720p images, and 1024 output tokens.
#### 5.1.1 Latency-Sensitive Benchmark
- Model Deployment Command:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang-oai-chat \
--host 127.0.0.1 \
--port 8000 \
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
--dataset-name image \
--image-count 2 \
--image-resolution 720p \
--random-input-len 128 \
--random-output-len 1024 \
--num-prompts 10 \
--max-concurrency 1
```
- **Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 45.97
Total input tokens: 18348
Total input text tokens: 708
Total input vision tokens: 17640
Total generated tokens: 4220
Total generated tokens (retokenized): 3423
Request throughput (req/s): 0.22
Input token throughput (tok/s): 399.17
Output token throughput (tok/s): 91.81
Peak output token throughput (tok/s): 96.00
Peak concurrent requests: 2
Total token throughput (tok/s): 490.98
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 4594.52
Median E2E Latency (ms): 3725.04
---------------Time to First Token----------------
Mean TTFT (ms): 193.35
Median TTFT (ms): 196.32
P99 TTFT (ms): 222.75
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 10.44
Median TPOT (ms): 10.44
P99 TPOT (ms): 10.47
---------------Inter-Token Latency----------------
Mean ITL (ms): 11.78
Median ITL (ms): 10.48
P95 ITL (ms): 21.01
P99 ITL (ms): 31.40
Max ITL (ms): 31.92
==================================================
```
**Optimized Results (with CUDA IPC Transport):**
For further TTFT optimization, enable CUDA IPC Transport for multimodal features by setting `SGLANG_USE_CUDA_IPC_TRANSPORT=1`. This significantly reduces TTFT by using CUDA IPC for transferring multimodal features.
- Model Deployment Command:
```shell Command
SGLANG_USE_CUDA_IPC_TRANSPORT=1 python -m sglang.launch_server \
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang-oai-chat \
--host 127.0.0.1 \
--port 8000 \
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
--dataset-name image \
--image-count 2 \
--image-resolution 720p \
--random-input-len 128 \
--random-output-len 1024 \
--num-prompts 100 \
--max-concurrency 1
```
- **Test Results:**
With `SGLANG_USE_CUDA_IPC_TRANSPORT=1`, TTFT improves significantly:
```text Output
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 100
Benchmark duration (s): 566.84
Total input tokens: 183667
Total input text tokens: 7267
Total input vision tokens: 176400
Total generated tokens: 52444
Total generated tokens (retokenized): 28702
Request throughput (req/s): 0.18
Input token throughput (tok/s): 324.02
Output token throughput (tok/s): 92.52
Peak output token throughput (tok/s): 96.00
Peak concurrent requests: 3
Total token throughput (tok/s): 416.54
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 5667.50
Median E2E Latency (ms): 5830.00
---------------Time to First Token----------------
Mean TTFT (ms): 191.16
Median TTFT (ms): 182.58
P99 TTFT (ms): 244.58
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 10.46
Median TPOT (ms): 10.46
P99 TPOT (ms): 10.48
---------------Inter-Token Latency----------------
Mean ITL (ms): 13.91
Median ITL (ms): 10.56
P95 ITL (ms): 21.35
P99 ITL (ms): 31.55
Max ITL (ms): 42.36
==================================================
```
#### 5.1.2 Throughput-Sensitive Benchmark
- Model Deployment Command:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang-oai-chat \
--host 127.0.0.1 \
--port 8000 \
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
--dataset-name image \
--image-count 2 \
--image-resolution 720p \
--random-input-len 128 \
--random-output-len 1024 \
--num-prompts 1000 \
--max-concurrency 100
```
- **Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 584.65
Total input tokens: 1839015
Total input text tokens: 75015
Total input vision tokens: 1764000
Total generated tokens: 510855
Total generated tokens (retokenized): 284284
Request throughput (req/s): 1.71
Input token throughput (tok/s): 3145.50
Output token throughput (tok/s): 873.78
Peak output token throughput (tok/s): 2855.00
Peak concurrent requests: 107
Total token throughput (tok/s): 4019.29
Concurrency: 98.35
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 57502.05
Median E2E Latency (ms): 54301.08
---------------Time to First Token----------------
Mean TTFT (ms): 5802.23
Median TTFT (ms): 1444.75
P99 TTFT (ms): 46675.92
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 100.22
Median TPOT (ms): 105.43
P99 TPOT (ms): 144.37
---------------Inter-Token Latency----------------
Mean ITL (ms): 134.20
Median ITL (ms): 25.57
P95 ITL (ms): 558.14
P99 ITL (ms): 1449.01
Max ITL (ms): 33453.23
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 MMMU Benchmark
You can evaluate the model's accuracy using the MMMU dataset with `lmms_eval`:
- Benchmark Command:
```shell Command
uv pip install lmms_eval
python3 -m lmms_eval \
--model openai_compatible \
--model_args "model=Qwen/Qwen3-VL-235B-A22B-Instruct,api_key=EMPTY,base_url=http://127.0.0.1:8000/v1/" \
--tasks mmmu_val \
--batch_size 128 \
--log_samples \
--log_samples_suffix "openai_compatible" \
--output_path ./logs \
--gen_kwargs "max_new_tokens=4096"
```
- **Test Results:**
```text Output
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "12%"}} />
<col style={{width: "11%"}} />
<col style={{width: "11%"}} />
<col style={{width: "11%"}} />
<col style={{width: "11%"}} />
<col style={{width: "11%"}} />
<col style={{width: "11%"}} />
<col style={{width: "11%"}} />
<col style={{width: "11%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Tasks</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Version</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Filter</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>n-shot</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Metric</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}></th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Value</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}></th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Stderr</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>mmmu_val</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>none</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>mmmu_acc</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>↑</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.6567</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>±</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
</tr>
</tbody>
</table>
```
docs_new/cookbook/autoregressive/Qwen/Qwen3.5.mdx
+1001
View File
File diff suppressed because it is too large Load Diff
docs_new/cookbook/autoregressive/Qwen/Qwen3.6.mdx
+471
View File
@@ -0,0 +1,471 @@
---
title: Qwen3.6
metatags:
description: "Deploy Qwen3.6 with SGLang - open-weight 35B MoE multimodal model with 3B active parameters, thinking preservation, tool calling, MTP, and long-context support."
tag: NEW
---
import { Qwen36Deployment } from '/src/snippets/autoregressive/qwen36-deployment.jsx';
## 1. Model Introduction
[Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) is the first open-weight variant of the Qwen3.6 series developed by Alibaba. Built on direct feedback from the community, Qwen3.6 prioritizes stability and real-world utility, delivering substantial upgrades in agentic coding and thinking preservation.
Qwen3.6 features a Gated Delta Networks combined with sparse Mixture-of-Experts architecture (35B total parameters, 3B activated), supporting multimodal inputs (text, image, video) and natively handles context lengths of up to 262,144 tokens, extensible to over 1M tokens.
**Key Features:**
- **Agentic Coding**: Handles frontend workflows and repository-level reasoning with greater fluency and precision
- **Thinking Preservation**: New option to retain reasoning context from historical messages, streamlining iterative development
- **Efficient Hybrid Architecture**: Gated Delta Networks + sparse MoE (35B total / 3B active) for high-throughput inference
- **Hybrid Reasoning**: Thinking mode enabled by default with step-by-step reasoning, can be disabled for direct responses
- **Tool Calling**: Built-in tool calling support with `qwen3_coder` parser
- **Multi-Token Prediction (MTP)**: Speculative decoding support for lower latency
- **Multimodal**: Unified vision-language model supporting text, image, and video inputs
**Available Models:**
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<thead>
<tr>
<th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>Model</th>
<th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>Weights</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3.6-35B-A3B (BF16)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3.6-35B-A3B (FP8)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[Qwen/Qwen3.6-35B-A3B-FP8](https://huggingface.co/Qwen/Qwen3.6-35B-A3B-FP8)</td>
</tr>
</tbody>
</table>
**License:** Apache 2.0
## 2. SGLang Installation
SGLang `>=0.5.10` is required for Qwen3.6. You can install from source or use a Docker image:
```bash Command
# Install from PyPI
uv pip install "sglang[all]"
# Or install from source
uv pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
# Or use Docker (NVIDIA GPUs)
docker pull lmsysorg/sglang:latest
```
For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/installation).
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and capabilities.
<Qwen36Deployment />
### 3.2 Configuration Tips
- Speculative decoding (MTP) can significantly reduce latency for interactive use cases.
- **Mamba Radix Cache**: Qwen3.6's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-scheduler-strategy`:
- **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage.
- **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-scheduler-strategy extra_buffer --page-size 64`. Requires FLA kernel backend (NVIDIA GPUs only). Trades higher mamba state memory for better throughput.
- The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload.
- Context length defaults to 262,144 tokens. If you encounter OOM errors, consider reducing it, but maintain at least 128K to preserve thinking capabilities.
- **CUDA IPC Transport**: Add `SGLANG_USE_CUDA_IPC_TRANSPORT=1` as an environment variable to use CUDA IPC for transferring multimodal features, significantly improving TTFT (Time To First Token). Note: this consumes additional memory proportional to image size, so you may need to lower `--mem-fraction-static` or `--max-running-requests`.
- **Multimodal Attention Backend**: Use `--mm-attention-backend fa3` on H100/H200 for better vision performance, or `--mm-attention-backend fa4` on B200.
- For processing large images or videos, you may need to lower `--mem-fraction-static` to leave room for image feature tensors.
- Hardware requirements:
- **BF16**: ~35B parameters require ~70GB of GPU memory for weights. TP=1 fits on all supported hardware.
- **FP8**: The FP8 quantized model requires ~35GB for weights. TP=1 fits on all supported hardware.
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<thead>
<tr>
<th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>Hardware</th>
<th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>Memory</th>
<th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>BF16 TP</th>
<th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>FP8 TP</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H100</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>80GB</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.05)"}}>H200</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>141GB</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>B200</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>183GB</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
</tr>
</tbody>
</table>
## 4. Model Invocation
Deploy Qwen3.6-35B-A3B with the following command (H200, all features enabled):
```shell Command
SGLANG_ENABLE_SPEC_V2=1 sglang serve \
--model-path Qwen/Qwen3.6-35B-A3B-FP8 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--mem-fraction-static 0.8 \
--host 0.0.0.0 \
--port 30000
```
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Vision Input
Qwen3.6 supports image and video inputs as a unified vision-language model.
**Image Input Example:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="Qwen/Qwen3.6-35B-A3B-FP8",
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/CI_Demo/mathv-1327.jpg"
}
},
{
"type": "text",
"text": "Describe this image in detail."
}
]
}
],
max_tokens=2048,
stream=True
)
thinking_started = False
has_thinking = False
has_answer = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
if delta.content:
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Video Input Example:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="Qwen/Qwen3.6-35B-A3B-FP8",
messages=[
{
"role": "user",
"content": [
{
"type": "video_url",
"video_url": {
"url": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/video/N1cdUjctpG8.mp4"
}
},
{
"type": "text",
"text": "Describe what happens in this video."
}
]
}
],
max_tokens=2048,
stream=True
)
thinking_started = False
has_thinking = False
has_answer = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
if delta.content:
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
### 4.3 Advanced Usage
#### 4.3.1 Reasoning Parser
Qwen3.6 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response.
To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time:
- **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed.
- **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process.
**Example 1: Thinking Mode (Default)**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="Qwen/Qwen3.6-35B-A3B-FP8",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
max_tokens=2048,
stream=True
)
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
if delta.content:
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Example 2: Instruct Mode (Thinking Off)**
To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="Qwen/Qwen3.6-35B-A3B-FP8",
messages=[
{"role": "user", "content": "What is 15% of 240?"}
],
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
max_tokens=2048,
stream=True
)
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if delta.content:
print(delta.content, end="", flush=True)
print()
```
#### 4.3.2 Thinking Preservation
Qwen3.6 has been trained to preserve and leverage thinking traces from historical messages. Enable this for agent scenarios where maintaining full reasoning context improves decision consistency:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="Qwen/Qwen3.6-35B-A3B-FP8",
messages=[
{"role": "user", "content": "Help me plan a web app architecture."}
],
extra_body={"chat_template_kwargs": {"preserve_thinking": True}},
max_tokens=2048,
stream=True
)
thinking_started = False
has_thinking = False
has_answer = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
if delta.content:
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
#### 4.3.3 Tool Calling
Qwen3.6 supports tool calling capabilities. Enable the tool call parser during deployment.
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
response = client.chat.completions.create(
model="Qwen/Qwen3.6-35B-A3B-FP8",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
stream=True
)
thinking_started = False
has_thinking = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
if hasattr(delta, 'tool_calls') and delta.tool_calls:
if has_thinking and thinking_started:
print("\n=============== Content =================", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
if tool_call.function:
print(f"Tool Call: {tool_call.function.name}")
print(f" Arguments: {tool_call.function.arguments}")
if delta.content:
print(delta.content, end="", flush=True)
print()
```
docs_new/cookbook/autoregressive/Qwen/Qwen3.mdx
+884
View File
@@ -0,0 +1,884 @@
---
title: Qwen3
metatags:
description: "Deploy Qwen3 series models with SGLang - featuring advanced reasoning, 256K context, and flexible Dense/MoE architectures for edge to cloud."
---
## 1. Model Introduction
[Qwen3 series](https://github.com/QwenLM/Qwen3) are the most powerful vision-language models in the Qwen series to date, featuring advanced capabilities in multi-modal understanding, reasoning, and agentic applications.
This generation delivers comprehensive upgrades across the board:
- **Stronger general intelligence**: Significant improvements in instruction following, logical reasoning, text comprehension, mathematics, science, coding, and tool usage.
- **Broader multilingual knowledge**: Substantial gains in long-tail knowledge coverage across multiple languages.
- **More helpful & aligned responses**: Markedly better alignment with user preferences in subjective and open-ended tasks, enabling higher-quality, more useful text generation.
- **Extended context length**: Enhanced capabilities in understanding and reasoning over 256K-token long contexts.
- **Stronger agent interaction capabilities**: Improved tool use and search-based agent performance.
- **Flexible deployment options**: Available in Dense and MoE architectures that scale from edge to cloud, with Instruct and reasoning-enhanced Thinking editions.
For more details, please refer to the [official Qwen3 GitHub Repository](https://github.com/QwenLM/Qwen3).
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
The Qwen3 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA and AMD GPUs. The recommended launch configurations vary by hardware and model size.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.
import { Qwen3Deployment } from "/src/snippets/autoregressive/qwen3-deployment.jsx";
<Qwen3Deployment />
### 3.2 Configuration Tips
- **Memory Management** : Set lower `--context-length` to conserve memory. A value of `128000` is sufficient for most scenarios, down from the default 262K.
- **Expert Parallelism** : SGLang supports Expert Parallelism (EP) via `--ep`, allowing experts in MoE models to be deployed on separate GPUs for better throughput. One thing to note is that, for quantized models, you need to set `--ep` to a value that satisfies the requirement: `(moe_intermediate_size / moe_tp_size) % weight_block_size_n == 0, where moe_tp_size is equal to tp_size divided by ep_size.` Note that EP may perform worse in low concurrency scenarios due to additional communication overhead. Check out [Expert Parallelism Deployment](../../../docs/advanced_features/expert_parallelism) for more details.
- **Kernel Tuning** : For MoE Triton kernel tuning on your specific hardware, refer to [fused_moe_triton](https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton).
- **Speculative Decoding**: Using Speculative Decoding for latency-sensitive scenarios.
- `--speculative-algorithm EAGLE3`: Speculative decoding algorithm
- `--speculative-num-steps 3`: Number of speculative verification rounds
- `--speculative-eagle-topk 1`: Top-k sampling for draft tokens
- `--speculative-num-draft-tokens 4`: Number of draft tokens per step
- `--speculative-draft-model-path`: The path of the draft model weights. This can be a local folder or a Hugging Face repo ID such as [`lmsys/SGLang-EAGLE3-Qwen3-235B-A22B-Instruct-2507-SpecForge-Meituan`](https://huggingface.co/lmsys/SGLang-EAGLE3-Qwen3-235B-A22B-Instruct-2507-SpecForge-Meituan).
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
Qwen3-235B-A22B supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-235B-A22B-Thinking-2507 \
--reasoning-parser qwen3 \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
**Streaming with Thinking Process:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="Qwen/Qwen3-235B-A22B-Thinking-2507",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
Okay, so I need to figure out what 15% of 240 is. Hmm, percentages can sometimes trip me up, but I think I remember some basics. Let me start by recalling that "percent" means "per hundred," so 15% is the same as 15 per 100, or 15/100. So, maybe I can convert 15% into a decimal first? Yeah, I think that's a common method.
...
So conclusion: The answer is 36.
=============== Content =================
To determine what 15% of 240 is, we can follow a systematic approach that involves converting the percentage to a decimal and then performing multiplication. Here's a step-by-step breakdown of the solution:
....
### Final Answer:
$$
\boxed{36}
$$
Thus, 15% of 240 is **36**.
```
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
#### 4.2.3 Tool Calling
Qwen3 supports tool calling capabilities. Enable the tool call parser:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-235B-A22B-Thinking-2507 \
--reasoning-parser qwen3 \
--tool-call-parser qwen25 \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
**Python Example (with Thinking Process):**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="Qwen/Qwen3-235B-A22B-Thinking-2507",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Accumulate tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================\n", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {
'name': None,
'arguments': ''
}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]['name'] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
# Print content
if delta.content:
print(delta.content, end="", flush=True)
# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"🔧 Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
print()
```
**Output Example:**
```text Output
=============== Thinking =================
Okay, the user is asking for the weather in Beijing. Let me check the tools available. There's a function called get_weather that takes location and unit parameters. The location is required, so I need to specify Beijing as the location. The unit is optional and can be either celsius or fahrenheit. Since the user didn't specify the unit, maybe I should default to a common one. In China, they usually use celsius, so I'll set unit to celsius. I'll call the get_weather function with location: Beijing and unit: celsius. That should get the current weather for them.
=============== Content =================
🔧 Tool Call: get_weather
Arguments: {"location": "Beijing", "unit": "celsius"}
```
**Note:**
- The reasoning parser shows how the model decides to use a tool
- Tool calls are clearly marked with the function name and arguments
- You can then execute the function and send the result back to continue the conversation
**Handling Tool Call Results:**
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="Qwen/Qwen3-235B-A22B-Thinking-2507",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The current weather in Beijing is **22°C** and **sunny**. A perfect day to enjoy outdoor activities! 🌞"
```
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: NVIDIA B200 GPU (8x)
- Model: Qwen3-235B-A22B-Instruct-2507
- Tensor Parallelism: 8
- sglang version: 0.5.6
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.
#### 5.1.1 Standard Scenario Benchmark
- Model Deployment Command:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
--tp 8
```
##### 5.1.1.1 Low Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 43.56
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 4210
Total generated tokens (retokenized): 4206
Request throughput (req/s): 0.23
Input token throughput (tok/s): 140.07
Output token throughput (tok/s): 96.65
Peak output token throughput (tok/s): 100.00
Peak concurrent requests: 2
Total token throughput (tok/s): 236.72
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 4353.63
Median E2E Latency (ms): 3475.79
---------------Time to First Token----------------
Mean TTFT (ms): 99.03
Median TTFT (ms): 92.18
P99 TTFT (ms): 166.05
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 10.12
Median TPOT (ms): 10.12
P99 TPOT (ms): 10.15
---------------Inter-Token Latency----------------
Mean ITL (ms): 10.13
Median ITL (ms): 10.12
P95 ITL (ms): 10.49
P99 ITL (ms): 10.70
Max ITL (ms): 13.45
==================================================
```
##### 5.1.1.2 Medium Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 48.95
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 40725
Total generated tokens (retokenized): 40716
Request throughput (req/s): 1.63
Input token throughput (tok/s): 810.44
Output token throughput (tok/s): 832.04
Peak output token throughput (tok/s): 1151.00
Peak concurrent requests: 21
Total token throughput (tok/s): 1642.48
Concurrency: 13.61
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 8326.72
Median E2E Latency (ms): 8827.86
---------------Time to First Token----------------
Mean TTFT (ms): 215.70
Median TTFT (ms): 88.82
P99 TTFT (ms): 727.08
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 16.36
Median TPOT (ms): 16.12
P99 TPOT (ms): 24.09
---------------Inter-Token Latency----------------
Mean ITL (ms): 15.96
Median ITL (ms): 14.52
P95 ITL (ms): 16.04
P99 ITL (ms): 67.69
Max ITL (ms): 457.52
==================================================
```
##### 5.1.1.3 High Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 92.07
Total input tokens: 249831
Total input text tokens: 249831
Total input vision tokens: 0
Total generated tokens: 252162
Total generated tokens (retokenized): 251124
Request throughput (req/s): 5.43
Input token throughput (tok/s): 2713.46
Output token throughput (tok/s): 2738.78
Peak output token throughput (tok/s): 4400.00
Peak concurrent requests: 110
Total token throughput (tok/s): 5452.24
Concurrency: 90.50
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 16665.09
Median E2E Latency (ms): 16060.10
---------------Time to First Token----------------
Mean TTFT (ms): 260.55
Median TTFT (ms): 122.68
P99 TTFT (ms): 863.11
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 32.94
Median TPOT (ms): 34.04
P99 TPOT (ms): 41.19
---------------Inter-Token Latency----------------
Mean ITL (ms): 32.59
Median ITL (ms): 23.54
P95 ITL (ms): 69.79
P99 ITL (ms): 119.09
Max ITL (ms): 577.70
==================================================
```
#### 5.1.2 Reasoning Scenario Benchmark
- Model Deployment Command:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
--tp 8
```
##### 5.1.2.1 Low Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 10 \
--max-concurrency 1
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 457.45
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 44452
Total generated tokens (retokenized): 44059
Request throughput (req/s): 0.02
Input token throughput (tok/s): 13.34
Output token throughput (tok/s): 97.17
Peak output token throughput (tok/s): 100.00
Peak concurrent requests: 2
Total token throughput (tok/s): 110.51
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 45742.42
Median E2E Latency (ms): 49266.87
---------------Time to First Token----------------
Mean TTFT (ms): 110.60
Median TTFT (ms): 109.36
P99 TTFT (ms): 167.43
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 10.23
Median TPOT (ms): 10.24
P99 TPOT (ms): 10.32
---------------Inter-Token Latency----------------
Mean ITL (ms): 10.27
Median ITL (ms): 10.26
P95 ITL (ms): 10.71
P99 ITL (ms): 10.97
Max ITL (ms): 15.79
==================================================
```
##### 5.1.2.2 Medium Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 80 \
--max-concurrency 16
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 340.17
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 318226
Total generated tokens (retokenized): 318104
Request throughput (req/s): 0.24
Input token throughput (tok/s): 116.61
Output token throughput (tok/s): 935.49
Peak output token throughput (tok/s): 1120.00
Peak concurrent requests: 19
Total token throughput (tok/s): 1052.10
Concurrency: 13.85
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 58885.30
Median E2E Latency (ms): 59238.70
---------------Time to First Token----------------
Mean TTFT (ms): 169.71
Median TTFT (ms): 101.61
P99 TTFT (ms): 455.71
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 14.82
Median TPOT (ms): 14.91
P99 TPOT (ms): 15.20
---------------Inter-Token Latency----------------
Mean ITL (ms): 14.76
Median ITL (ms): 14.63
P95 ITL (ms): 15.46
P99 ITL (ms): 16.62
Max ITL (ms): 104.94
==================================================
```
##### 5.1.2.3 High Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 320 \
--max-concurrency 64
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 544.83
Total input tokens: 158939
Total input text tokens: 158939
Total input vision tokens: 0
Total generated tokens: 1300705
Total generated tokens (retokenized): 1293015
Request throughput (req/s): 0.59
Input token throughput (tok/s): 291.72
Output token throughput (tok/s): 2387.34
Peak output token throughput (tok/s): 3008.00
Peak concurrent requests: 68
Total token throughput (tok/s): 2679.06
Concurrency: 56.35
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 95937.70
Median E2E Latency (ms): 99362.32
---------------Time to First Token----------------
Mean TTFT (ms): 265.03
Median TTFT (ms): 129.11
P99 TTFT (ms): 823.85
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 23.66
Median TPOT (ms): 24.07
P99 TPOT (ms): 24.97
---------------Inter-Token Latency----------------
Mean ITL (ms): 23.54
Median ITL (ms): 23.07
P95 ITL (ms): 25.92
P99 ITL (ms): 63.87
Max ITL (ms): 408.30
==================================================
```
#### 5.1.3 Summarization Scenario Benchmark
##### 5.1.3.1 Low Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 44.82
Total input tokens: 41941
Total input text tokens: 41941
Total input vision tokens: 0
Total generated tokens: 4210
Total generated tokens (retokenized): 4210
Request throughput (req/s): 0.22
Input token throughput (tok/s): 935.86
Output token throughput (tok/s): 93.94
Peak output token throughput (tok/s): 99.00
Peak concurrent requests: 2
Total token throughput (tok/s): 1029.80
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 4479.60
Median E2E Latency (ms): 3622.99
---------------Time to First Token----------------
Mean TTFT (ms): 139.90
Median TTFT (ms): 114.85
P99 TTFT (ms): 225.17
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 10.31
Median TPOT (ms): 10.33
P99 TPOT (ms): 10.51
---------------Inter-Token Latency----------------
Mean ITL (ms): 10.33
Median ITL (ms): 10.33
P95 ITL (ms): 10.73
P99 ITL (ms): 10.93
Max ITL (ms): 14.48
==================================================
```
##### 5.1.3.2 Medium Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 50.68
Total input tokens: 300020
Total input text tokens: 300020
Total input vision tokens: 0
Total generated tokens: 41589
Total generated tokens (retokenized): 41578
Request throughput (req/s): 1.58
Input token throughput (tok/s): 5920.41
Output token throughput (tok/s): 820.69
Peak output token throughput (tok/s): 1200.00
Peak concurrent requests: 20
Total token throughput (tok/s): 6741.10
Concurrency: 13.90
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 8805.54
Median E2E Latency (ms): 9368.79
---------------Time to First Token----------------
Mean TTFT (ms): 284.29
Median TTFT (ms): 168.48
P99 TTFT (ms): 1027.21
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 16.81
Median TPOT (ms): 16.66
P99 TPOT (ms): 27.18
---------------Inter-Token Latency----------------
Mean ITL (ms): 16.42
Median ITL (ms): 13.68
P95 ITL (ms): 17.23
P99 ITL (ms): 90.75
Max ITL (ms): 574.64
==================================================
```
##### 5.1.3.3 High Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 320 \
--max-concurrency 64
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 94.77
Total input tokens: 1273893
Total input text tokens: 1273893
Total input vision tokens: 0
Total generated tokens: 169680
Total generated tokens (retokenized): 169640
Request throughput (req/s): 3.38
Input token throughput (tok/s): 13441.86
Output token throughput (tok/s): 1790.43
Peak output token throughput (tok/s): 2687.00
Peak concurrent requests: 70
Total token throughput (tok/s): 15232.28
Concurrency: 58.63
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 17364.14
Median E2E Latency (ms): 17495.95
---------------Time to First Token----------------
Mean TTFT (ms): 238.22
Median TTFT (ms): 203.27
P99 TTFT (ms): 510.48
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 32.50
Median TPOT (ms): 34.27
P99 TPOT (ms): 40.59
---------------Inter-Token Latency----------------
Mean ITL (ms): 32.36
Median ITL (ms): 22.50
P95 ITL (ms): 97.81
P99 ITL (ms): 151.55
Max ITL (ms): 352.79
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 GSM8K Benchmark
- **Benchmark Command:**
```shell Command
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
```
- **Results**:
- Qwen/Qwen3-235B-A22B-Instruct-2507
```text Output
Accuracy: 0.945
Invalid: 0.000
Latency: 11.980 s
Output throughput: 2358.105 token/s
```
docs_new/cookbook/autoregressive/StepFun/Step3-VL-10B.mdx
+695
View File
@@ -0,0 +1,695 @@
---
title: Step3-VL-10B
metatags:
description: "Deploy Step3-VL-10B multimodal model with SGLang - compact 10B dense model with frontier-level vision understanding, complex reasoning, and tool calling capabilities."
tag: NEW
---
import { Step3VL10BDeployment } from '/src/snippets/autoregressive/step-3vl-10b-deployment.jsx';
## 1. Model Introduction
[Step3-VL-10B](https://huggingface.co/stepfun-ai/Step3-VL-10B) is a lightweight open-source multimodal model developed by StepFun, designed to redefine the trade-off between compact efficiency and frontier-level multimodal intelligence. Despite its compact 10B parameter footprint, Step3-VL-10B excels in visual perception, complex reasoning, and human-centric alignment.
Key highlights of Step3-VL-10B include:
- **STEM Reasoning**: Achieves 94.43% on AIME 2025 and 75.95% on MathVision (with PaCoRe), demonstrating exceptional complex reasoning capabilities that outperform models 10×–20× larger.
- **Visual Perception**: Records 92.05% on MMBench and 80.11% on MMMU, establishing strong general visual understanding and multimodal reasoning.
- **GUI & OCR**: Delivers state-of-the-art performance on ScreenSpot-V2 (92.61%), ScreenSpot-Pro (51.55%), and OCRBench (86.75%), optimized for agentic and document understanding tasks.
- **Spatial Understanding**: Demonstrates emergent spatial awareness with 66.79% on BLINK and 57.21% on All-Angles-Bench, establishing strong potential for embodied intelligence applications.
For more details, please refer to the [Step3-VL-10B model card on Hugging Face](https://huggingface.co/stepfun-ai/Step3-VL-10B).
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
Step3-VL-10B is a compact 10B dense model that can run on a single GPU. Recommended starting configurations vary depending on hardware.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and quantization method. SGLang supports serving Step3-VL-10B on NVIDIA B200, H200, H100, and AMD MI355X, MI325X, MI300X GPUs.
<Step3VL10BDeployment />
### 3.2 Configuration Tips
- **Single GPU Deployment**: Step3-VL-10B fits comfortably on a single GPU with BF16 precision, no tensor parallelism required.
- **Memory Management**: Set lower `--context-length` to conserve memory if needed. A value of `32768` is sufficient for most scenarios.
- **FP8 Quantization**: Use FP8 quantization to further reduce memory usage while maintaining quality.
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
### 4.2 Advanced Usage
#### 4.2.1 Multi-Modal Inputs
Step3-VL-10B supports image inputs. Here's a basic example with image input:
```python Example
import time
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:30000/v1",
timeout=3600
)
messages = [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
}
},
{
"type": "text",
"text": "Read all the text in the image."
}
]
}
]
start = time.time()
response = client.chat.completions.create(
model="stepfun-ai/Step3-VL-10B",
messages=messages,
max_tokens=2048,
extra_body={"top_k": -1}
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```
**Example output:**
```text Output
Response costs: 5.89s
Generated text: Auntie Anne's
CINNAMON SUGAR
1 × 17,000               17,000
SUB TOTAL                    17,000
GRAND TOTAL                 17,000
CASH IDR                    20,000
CHANGE DUE                 3,000
```
**Multi-Image Input Example:**
Step3-VL-10B can process multiple images in a single request for comparison or analysis:
```python Example
import time
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:30000/v1",
timeout=3600
)
messages = [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
}
},
{
"type": "image_url",
"image_url": {
"url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
}
},
{
"type": "text",
"text": "Compare these two images and describe the differences in 100 words or less."
}
]
}
]
start = time.time()
response = client.chat.completions.create(
model="stepfun-ai/Step3-VL-10B",
messages=messages,
max_tokens=2048,
extra_body={"top_k": -1}
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```
**Example Output:**
```text Output
Response costs: 3.24s
Generated text: First image: Single red Hong Kong taxi close - up, clear license plate (RX 5004), “4 SEATS” sticker, urban street with shops behind. Second image: Aerial view of many taxis (red, green) on a highway with a viaduct, some hoods open, dense arrangement. Differences: Scale (single vs many), perspective (close - up vs aerial), context (street shops vs highway), and taxi conditions (normal vs some open hoods).
```
#### 4.2.2 Reasoning Parser
Step3-VL-10B supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
```shell Command
python -m sglang.launch_server \
--model stepfun-ai/Step3-VL-10B \
--reasoning-parser deepseek-r1 \
--host 0.0.0.0 \
--port 30000 \
--trust-remote-code
```
**Streaming with Thinking Process:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="stepfun-ai/Step3-VL-10B",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
stream=True,
extra_body={"top_k": -1}
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Example Output:**
```text Output
=============== Thinking =================
Okay, let's see. The problem is asking for 15% of 240. Hmm, I need to remember how to calculate percentages. So, percentage means "per hundred," right? So, 15% is the same as 15 per 100, or 15/100. To find a percentage of a number, I think you convert the percentage to a decimal and then multiply it by the number. Let me check that.
First, 15% as a decimal. To convert a percentage to a decimal, you divide by 100. So 15 divided by 100 is 0.15. Yeah, that's right. So 15% is 0.15 in decimal form. Then, to find 15% of 24
0, I need to multiply 0.15 by 240. Let me do that calculation.
Let me write it out: 0.15 * 240. Let's compute that. Maybe break it down. 0.1 is 10%, and 0.05 is 5%, so 10% of 240 is 24, and 5% of 240 is 12. Then 10% + 5% is 15%, so 24 + 12 is 36. Oh,
that's a good way to check. So 15% is 10% plus 5%, which adds up to 36. Let me verify with the decimal method. 0.15 * 240. Let's multiply 240 by 0.1 first: 24. Then 240 by 0.05: 12. Adding them gives 36. Yep, that matches. Alternatively, 240 * 15 = 3600, and then divide by 100 (since it's per hundred), so 3600 / 100 = 36. That's another way. So 15% of 240 is 36. Let me make sure I didn't make a mistake. Let's check with another method. 10% of 240 is 24, 20% would be 48, so 15% is halfway between 10% and 20%, which is (24 + 48)/2 = 36. Yep, that works too
. So all methods point to 36. I think that's the answer.
=============== Content =================
To solve the problem "What is 15% of 240?" step by step:
---
### **Step 1: Understand the concept of percentage**
A percentage represents a portion of a whole. Specifically, "percent" means "per hundred." So, 15% means **15 out of 100**, or **15/100**.
---
### **Step 2: Convert the percentage to a decimal**
To use percentages in calculations, convert them to decimals by dividing by 100:
$$
15\% = \frac{15}{100} = 0.15
$$
---
### **Step 3: Multiply the decimal by the given number**
Now, multiply 0.15 (the decimal form of 15%) by 240:
$$
0.15 \times 240 = 36
$$
---
### **Alternative Verification Methods**
#### **Method A: Break into parts**
- 10% of 240 = $ 0.10 \times 240 = 24 $
- 5% of 240 = $ 0.05 \times 240 = 12 $
- Add them: $ 24 + 12 = 36 $
#### **Method B: Use direct multiplication**
- $ 15\% \text{ of } 240 = \frac{15}{100} \times 240 = \frac{3600}{100} = 36 $
#### **Method C: Estimate using known percentages**
- 20% of 240 = $ 0.20 \times 240 = 48 $
- 10% of 240 = $ 0.10 \times 240 = 24 $
- 15% is halfway between 10% and 20%: $ \frac{24 + 48}{2} = 36 $
---
### **Final Answer**
$$
\boxed{36}
$$
```
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
#### 4.2.3 Tool Calling
Step3-VL-10B supports tool calling capabilities. Enable the tool call parser:
```shell Command
python -m sglang.launch_server \
--model stepfun-ai/Step3-VL-10B \
--reasoning-parser deepseek-r1 \
--tool-call-parser hermes \
--host 0.0.0.0 \
--port 30000 \
--trust-remote-code
```
**Python Example (with Thinking Process):**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="stepfun-ai/Step3-VL-10B",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True,
extra_body={"top_k": -1}
)
# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Accumulate tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================\n", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {
'name': None,
'arguments': ''
}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]['name'] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
# Print content
if delta.content:
print(delta.content, end="", flush=True)
# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
print()
```
**Example Output:**
```text Output
=============== Thinking =================
The user is asking about the weather in Beijing. I have a function called "get_weather" that can provide weather information for a location. Let me check the parameters:
- location: required (string) - "Beijing"
- unit: optional (string, enum: ["celsius", "fahrenheit"]) - not specified by the user, so I won't include it
I should call the function with location="Beijing".
<tool_calls>
=============== Content =================
</tool_calls>Tool Call: get_weather
Arguments: {"location": "Beijing"}
```
**Handling Tool Call Results:**
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="stepfun-ai/Step3-VL-10B",
messages=messages,
temperature=0.7,
extra_body={"top_k": -1}
)
print(final_response.choices[0].message.content)
```
**Note:**
- The reasoning parser shows how the model decides to use a tool
- Tool calls are clearly marked with the function name and arguments
- You can then execute the function and send the result back to continue the conversation
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: NVIDIA B200 GPU (1x)
- Model: stepfun-ai/Step3-VL-10B
- Tensor Parallelism: 1
- sglang version: 0.5.8+
We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images.
#### 5.1.1 Latency-Sensitive Benchmark
- Model Deployment Command:
```shell Command
python -m sglang.launch_server \
--model stepfun-ai/Step3-VL-10B \
--host 0.0.0.0 \
--port 30000 \
--trust-remote-code
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang-oai-chat \
--host 127.0.0.1 \
--port 30000 \
--model stepfun-ai/Step3-VL-10B \
--dataset-name image \
--image-count 2 \
--image-resolution 720p \
--random-input-len 128 \
--random-output-len 1024 \
--num-prompts 10 \
--max-concurrency 1
```
- Result:
```text Output
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 30.85
Total input tokens: 14120
Total input text tokens: 720
Total input vision tokens: 13400
Total generated tokens: 4220
Total generated tokens (retokenized): 4217
Request throughput (req/s): 0.32
Input token throughput (tok/s): 457.71
Output token throughput (tok/s): 136.79
Peak output token throughput (tok/s): 240.00
Peak concurrent requests: 2
Total token throughput (tok/s): 594.50
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 3083.40
Median E2E Latency (ms): 2747.00
P90 E2E Latency (ms): 4574.50
P99 E2E Latency (ms): 5462.49
---------------Time to First Token----------------
Mean TTFT (ms): 1327.69
Median TTFT (ms): 1341.01
P99 TTFT (ms): 1486.11
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 4.16
Median TPOT (ms): 4.17
P99 TPOT (ms): 4.18
---------------Inter-Token Latency----------------
Mean ITL (ms): 4.17
Median ITL (ms): 4.18
P95 ITL (ms): 4.30
P99 ITL (ms): 4.38
Max ITL (ms): 8.24
==================================================
```
#### 5.1.2 Throughput-Sensitive Benchmark
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang-oai-chat \
--host 127.0.0.1 \
--port 30000 \
--model stepfun-ai/Step3-VL-10B \
--dataset-name image \
--image-count 2 \
--image-resolution 720p \
--random-input-len 128 \
--random-output-len 1024 \
--num-prompts 1000 \
--max-concurrency 100
```
- Result:
```text Output
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 976.52
Total input tokens: 1416949
Total input text tokens: 76949
Total input vision tokens: 1340000
Total generated tokens: 510855
Total generated tokens (retokenized): 510526
Request throughput (req/s): 1.02
Input token throughput (tok/s): 1451.02
Output token throughput (tok/s): 523.14
Peak output token throughput (tok/s): 20429.00
Peak concurrent requests: 103
Total token throughput (tok/s): 1974.16
Concurrency: 99.81
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 97463.22
Median E2E Latency (ms): 91872.75
P90 E2E Latency (ms): 118553.42
P99 E2E Latency (ms): 198445.56
---------------Time to First Token----------------
Mean TTFT (ms): 94379.07
Median TTFT (ms): 87163.09
P99 TTFT (ms): 194871.41
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 5.89
Median TPOT (ms): 5.72
P99 TPOT (ms): 23.58
---------------Inter-Token Latency----------------
Mean ITL (ms): 6.05
Median ITL (ms): 0.13
P95 ITL (ms): 0.56
P99 ITL (ms): 3.99
Max ITL (ms): 97551.06
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 MMMU Benchmark
You can evaluate the model's accuracy using the MMMU dataset:
- Model Deployment Command:
```shell Command
python -m sglang.launch_server \
--model stepfun-ai/Step3-VL-10B \
--host 0.0.0.0 \
--port 30000 \
--trust-remote-code
```
- Benchmark Command:
```shell Command
python3 benchmark/mmmu/bench_sglang.py \
--port 30000 \
--concurrency 64
```
- Result:
```text Output
Benchmark time: 934.6179109360091
answers saved to: ./answer_sglang.json
Evaluating...
answers saved to: ./answer_sglang.json
{'Accounting': {'acc': 0.667, 'num': 30},
'Agriculture': {'acc': 0.367, 'num': 30},
'Architecture_and_Engineering': {'acc': 0.4, 'num': 30},
'Art': {'acc': 0.467, 'num': 30},
'Art_Theory': {'acc': 0.5, 'num': 30},
'Basic_Medical_Science': {'acc': 0.367, 'num': 30},
'Biology': {'acc': 0.3, 'num': 30},
'Chemistry': {'acc': 0.467, 'num': 30},
'Clinical_Medicine': {'acc': 0.567, 'num': 30},
'Computer_Science': {'acc': 0.467, 'num': 30},
'Design': {'acc': 0.567, 'num': 30},
'Diagnostics_and_Laboratory_Medicine': {'acc': 0.3, 'num': 30},
'Economics': {'acc': 0.6, 'num': 30},
'Electronics': {'acc': 0.567, 'num': 30},
'Energy_and_Power': {'acc': 0.633, 'num': 30},
'Finance': {'acc': 0.733, 'num': 30},
'Geography': {'acc': 0.333, 'num': 30},
'History': {'acc': 0.533, 'num': 30},
'Literature': {'acc': 0.533, 'num': 30},
'Manage': {'acc': 0.6, 'num': 30},
'Marketing': {'acc': 0.767, 'num': 30},
'Materials': {'acc': 0.6, 'num': 30},
'Math': {'acc': 0.7, 'num': 30},
'Mechanical_Engineering': {'acc': 0.333, 'num': 30},
'Music': {'acc': 0.4, 'num': 30},
'Overall': {'acc': 0.523, 'num': 900},
'Overall-Art and Design': {'acc': 0.483, 'num': 120},
'Overall-Business': {'acc': 0.673, 'num': 150},
'Overall-Health and Medicine': {'acc': 0.513, 'num': 150},
'Overall-Humanities and Social Science': {'acc': 0.492, 'num': 120},
'Overall-Science': {'acc': 0.5, 'num': 150},
'Overall-Tech and Engineering': {'acc': 0.481, 'num': 210},
'Pharmacy': {'acc': 0.6, 'num': 30},
'Physics': {'acc': 0.7, 'num': 30},
'Psychology': {'acc': 0.467, 'num': 30},
'Public_Health': {'acc': 0.733, 'num': 30},
'Sociology': {'acc': 0.433, 'num': 30}}
eval out saved to ./val_sglang.json
Overall accuracy: 0.523
```
docs_new/cookbook/autoregressive/StepFun/Step3.5.mdx
+529
View File
@@ -0,0 +1,529 @@
---
title: Step-3.5
metatags:
description: "Deploy Step-3.5 reasoning engine with SGLang. "
tag: NEW
---
import { Step35Deployment } from '/src/snippets/autoregressive/step-35-deployment.jsx';
## 1. Model Introduction
[Step-3.5-Flash](https://huggingface.co/stepfun-ai/Step-3.5-Flash) is StepFun's production-grade reasoning engine built to decouple elite intelligence from heavy compute, and cuts attention cost for low-latency, cost-effective long-context inference—purpose-built for autonomous agents in real-world workflows. The model is available in multiple quantization formats optimized for different hardware platforms.
This generation delivers comprehensive upgrades across the board:
- **Hybrid Attention Architecture**: Interleaves Sliding Window Attention (SWA) and Global Attention (GA) with a 3:1 ratio and an aggressive 128-token window. This hybrid approach ensures consistent performance across massive datasets or long codebases while significantly reducing the computational overhead typical of standard long-context models.
- **Sparse Mixture-of-Experts**: Only 11B active parameters out of 196B parameters.
- **Multi-Layer Multi-Token Prediction (MTP)**: Equipped with a 3-way Multi-Token Prediction (MTP-3). This allows for complex, multi-step reasoning chains with immediate responsiveness.
## 2.SGLang Installation
Step-3.5-Flash is currently available in SGLang via Docker image install.
### Docker (NVIDIA)
```bash Command
# Pull the docker image
docker pull lmsysorg/sglang:dev-pr-18084
# Launch the container
docker run -it --gpus all \
--shm-size=32g \
--ipc=host \
--network=host \
lmsysorg/sglang:dev-pr-18084 bash
```
### Docker (AMD ROCm)
```bash Command
# For MI300X/MI325X
docker pull lmsysorg/sglang:v0.5.9-rocm700-mi30x
# For MI350X/MI355X
docker pull lmsysorg/sglang:v0.5.9-rocm700-mi35x
docker run -it \
--device=/dev/kfd --device=/dev/dri \
--shm-size=32g \
--ipc=host \
--network=host \
--group-add video --cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
lmsysorg/sglang:v0.5.9-rocm700-mi30x bash # or mi35x for MI350X/MI355X
```
## 3.Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
The Step-3.5-Flash series comes in only one sizes. Recommended starting configurations vary depending on hardware.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.
<Step35Deployment />
### 3.2 Configuration Tips
- **Memory**: Requires GPUs with high VRAM capacity. Supported platforms: H200 (4×, TP=4), MI300X/MI325X/MI350X/MI355X (4×, TP=4 EP=4).
- **AMD Docker Image**: Use `lmsysorg/sglang:v0.5.9-rocm700-mi30x` for MI300X/MI325X and `lmsysorg/sglang:v0.5.9-rocm700-mi35x` for MI350X/MI355X.
- **AMD Expert Parallelism Required**: On AMD GPUs, always use `--ep 4` with `--tp 4`. Both BF16 and FP8 models require expert parallelism. Without EP, the MoE intermediate dimension is split across GPUs (N=320), which triggers an AITER CK GEMM incompatibility. With EP=4, each GPU handles 72 full experts (N=1280), which works correctly with cuda graph enabled.
- **AITER JIT Compilation**: First inference on AMD may take 30-40 seconds for AITER kernel JIT compilation. Subsequent requests use cached kernels.
## 4.Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
Step-3.5-Flash only supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
```shell Command
sglang serve \
--model-path stepfun-ai/Step-3.5-Flash \
--tp 4 \
--ep 4 \
--reasoning-parser step3p5
```
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="stepfun-ai/Step-3.5-Flash",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
We are asked: "What is 15% of 240?" We need to solve step by step.
Step 1: Understand that "15% of 240" means we need to calculate 15 percent of 240. In mathematical terms, it is (15/100) * 240.
Step 2: Simplify the calculation. We can compute 15% of 240 by first finding 10% of 240 and then 5% of 240, and adding them. Alternatively, we can multiply directly.
Method 1:
10% of 240 = 240 * 0.10 = 24.
5% is half of 10%, so 5% of 240 = 24 / 2 = 12.
Then 15% = 10% + 5% = 24 + 12 = 36.
Method 2: Direct multiplication: 15% = 15/100 = 0.15, so 0.15 * 240 = 36.
We can also compute fractionally: (15/100)*240 = (15*240)/100. 15*240 = 3600, divided by 100 gives 36.
Thus, the answer is 36.
We'll present the solution step by step.
=============== Content =================
To find 15% of 240, follow these steps:
1. **Convert the percentage to a decimal**:
\( 15\% = \frac{15}{100} = 0.15 \)
2. **Multiply by the number**:
\( 0.15 \times 240 = 36 \)
Alternatively, break it down:
- \( 10\% \text{ of } 240 = 240 \times 0.10 = 24 \)
- \( 5\% \text{ of } 240 = \frac{24}{2} = 12 \) (since 5% is half of 10%)
- \( 15\% = 10\% + 5\% = 24 + 12 = 36 \)
**Answer:** 36
```
#### 4.2.2 Tool Calling
Step-3.5 supports tool calling capabilities. Enable the tool call parser:
**Python Example:**
Start sglang server:
```shell Command
sglang serve \
--model-path stepfun-ai/Step-3.5-Flash \
--tp 4 \
--ep 4 \
--reasoning-parser step3p5 \
--tool-call-parser step3p5
```
```python Example
from openai import OpenAI
import json
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# 1. define tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string", "description": "The city name"},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit"}
},
"required": ["location"]
}
}
}
]
# 2. tool run
def get_weather(location, unit="celsius"):
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# 3. send first request
print("--- Sending first request ---")
response = client.chat.completions.create(
model="stepfun-ai/Step-3.5-Flash",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=1.0,
stream=False
)
message = response.choices[0].message
# 4. Handle Reasoning Content
reasoning = getattr(message, 'reasoning_content', None)
if reasoning:
print("=============== Thinking =================")
print(reasoning)
print("==========================================")
# 5. Handle Tool Calls
if message.tool_calls:
print("\n🔧 Tool Calls detected:")
history_messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
message
]
for tool_call in message.tool_calls:
print(f" Tool: {tool_call.function.name}")
print(f" Args: {tool_call.function.arguments}")
args = json.loads(tool_call.function.arguments)
tool_result = get_weather(args.get("location"), args.get("unit", "celsius"))
history_messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": tool_result
})
print("\n--- Sending tool results ---")
final_response = client.chat.completions.create(
model="stepfun-ai/Step-3.5-Flash",
messages=history_messages,
temperature=1.0,
stream=False
)
print("=============== Final Content =================")
print(final_response.choices[0].message.content)
else:
if message.content:
print("=============== Content =================")
print(message.content)
```
**Output Example:**
```text Output
--- Sending first request ---
=============== Thinking =================
The user is asking for the weather in Beijing. I should use the get_weather function with location="Beijing". The unit parameter is optional and the user didn't specify a preference, so I'll leave it out (the default should be fine).
==========================================
🔧 Tool Calls detected:
Tool: get_weather
Args: {"location": "Beijing"}
--- Sending tool results ---
=============== Final Content =================
The weather in Beijing is 22°C and sunny.
```
**Note:**
- The reasoning parser shows how the model decides to use a tool
- Tool calls are clearly marked with the function name and arguments
- You can then execute the function and send the result back to continue the conversation
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: NVIDIA H200 GPU (4x)
- Model: Step-3.5-Flash
- Tensor Parallelism: 4
- Expert Parallelism: 4
- sglang version: 0.5.8
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.
#### 5.1.1 Standard Scenario Benchmark
- Model Deployment Command:
```shell Command
sglang serve \
--model-path stepfun-ai/Step-3.5-Flash \
--tp 4 \
--ep 4
```
##### 5.1.1.1 Low Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model stepfun-ai/Step-3.5-Flash \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 35.30
Total input tokens: 6091
Total input text tokens: 6091
Total generated tokens: 4220
Total generated tokens (retokenized): 4212
Request throughput (req/s): 0.28
Input token throughput (tok/s): 172.57
Output token throughput (tok/s): 119.56
Peak output token throughput (tok/s): 124.00
Peak concurrent requests: 2
Total token throughput (tok/s): 292.14
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 3527.94
Median E2E Latency (ms): 2884.72
P90 E2E Latency (ms): 6350.38
P99 E2E Latency (ms): 7858.53
---------------Time to First Token----------------
Mean TTFT (ms): 107.53
Median TTFT (ms): 80.93
P99 TTFT (ms): 269.52
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 8.12
Median TPOT (ms): 8.13
P99 TPOT (ms): 8.14
---------------Inter-Token Latency----------------
Mean ITL (ms): 8.12
Median ITL (ms): 8.11
P95 ITL (ms): 8.61
P99 ITL (ms): 8.91
Max ITL (ms): 20.77
==================================================
```
##### 5.1.1.2 Medium Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model stepfun-ai/Step-3.5-Flash \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 54.06
Total input tokens: 39588
Total input text tokens: 39588
Total generated tokens: 40805
Total generated tokens (retokenized): 40479
Request throughput (req/s): 1.48
Input token throughput (tok/s): 732.33
Output token throughput (tok/s): 754.84
Peak output token throughput (tok/s): 928.00
Peak concurrent requests: 21
Total token throughput (tok/s): 1487.17
Concurrency: 14.06
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 9501.23
Median E2E Latency (ms): 10010.71
P90 E2E Latency (ms): 15655.09
P99 E2E Latency (ms): 18803.63
---------------Time to First Token----------------
Mean TTFT (ms): 198.34
Median TTFT (ms): 89.50
P99 TTFT (ms): 984.66
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 18.97
Median TPOT (ms): 18.80
P99 TPOT (ms): 35.67
---------------Inter-Token Latency----------------
Mean ITL (ms): 18.27
Median ITL (ms): 17.48
P95 ITL (ms): 18.44
P99 ITL (ms): 62.47
Max ITL (ms): 460.85
==================================================
```
##### 5.1.1.3 High Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model stepfun-ai/Step-3.5-Flash \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 125.88
Total input tokens: 249331
Total input text tokens: 249331
Total generated tokens: 252662
Total generated tokens (retokenized): 251323
Request throughput (req/s): 3.97
Input token throughput (tok/s): 1980.77
Output token throughput (tok/s): 2007.23
Peak output token throughput (tok/s): 2500.00
Peak concurrent requests: 109
Total token throughput (tok/s): 3987.99
Concurrency: 92.25
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 23223.31
Median E2E Latency (ms): 22631.90
P90 E2E Latency (ms): 42269.38
P99 E2E Latency (ms): 47637.53
---------------Time to First Token----------------
Mean TTFT (ms): 372.13
Median TTFT (ms): 127.26
P99 TTFT (ms): 1880.42
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 46.06
Median TPOT (ms): 47.61
P99 TPOT (ms): 51.34
---------------Inter-Token Latency----------------
Mean ITL (ms): 45.31
Median ITL (ms): 39.86
P95 ITL (ms): 72.49
P99 ITL (ms): 117.05
Max ITL (ms): 1359.81
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 GSM8K Benchmark
- **Benchmark Command:**
```shell Command
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
```
- **Results**:
- Step-3.5-Flash
```
Accuracy: 0.885
Invalid: 0.005
Latency: 9.986 s
Output throughput: 1972.911 token/s
```
docs_new/cookbook/autoregressive/Xiaomi/MiMo-V2-Flash.mdx
+106
View File
@@ -0,0 +1,106 @@
---
title: MiMo-V2-Flash
metatags:
description: "Deploy MiMo-V2-Flash 309B MoE model with SGLang - hybrid attention, multi-token prediction, and 256K context for efficient inference."
---
## Introduction
XiaomiMiMo/MiMo-V2-Flash, with 309B total parameters and 15B activated parameters, is a new inference-centric model designed to maximize decoding efficiency created by XiaomiMiMo Team explicitly co-designed for real-world serving workloads, enabling flexible tradeoffs between throughput and latency on different hardware.
This model creates a new balance between long-context modeling capability and inference efficiency. Key features include:
- **Hybrid Attention Architecture**: Interleaves Sliding Window Attention (SWA) and Global Attention (GA) with a 5:1 ratio and an aggressive 128-token window. This reduces KV-cache storage by nearly 6x while maintaining long-context performance via learnable attention sink bias.
- **Multi-Token Prediction (MTP)**: Equipped with a lightweight MTP module (0.33B params/block) using dense FFNs. This triples output speed during inference and will be good to accelerates rollout in RL training.
- **Efficient Pre-Training**: Trained on 27T tokens using FP8 mixed precision and native 32k seq length. The context window supports up to 256k length.
- **Agentic Capabilities**: Post-training utilizes Multi-Teacher On-Policy Distillation (MOPD) and large-scale agentic RL, achieving superior performance on SWE-Bench and complex reasoning tasks.
## Installation
MiMo-V2-Flash is currently available in SGLang via Docker image and pip install.
### Docker
```bash Command
# Pull the docker image
docker pull lmsysorg/sglang:dev-pr-15207
# Launch the container
docker run -it --gpus all \
--shm-size=32g \
--ipc=host \
--network=host \
lmsysorg/sglang:dev-pr-15207 bash
```
### Pip Installation
```bash Command
# On a machine with SGLang dependencies installed or inside a SGLang nightly container
# Start an SGLang nightly container
docker run -it --gpus all \
--shm-size=32g \
--ipc=host \
--network=host \
lmsysorg/sglang:nightly-dev-20251215-4449c170 bash
# If you already have SGLang installed, uninstall the current SGLang version
pip uninstall sglang -y
# Install the PyPI Package
pip install sglang==0.5.6.post2.dev8005+pr.15207.g39d5bd57a \
--extra-index-url https://sgl-project.github.io/whl/pr/
```
## Model Deployment
Use the configuration selector below to automatically generate the appropriate deployment command.
import { MiMoV2FlashDeployment } from "/src/snippets/autoregressive/mimo-v2-flash-deployment.jsx";
<MiMoV2FlashDeployment />
MI355X (ROCm) is validated in the selector above with `--tp-size 4`, Triton attention, and `--disable-custom-all-reduce`. `--tp-size 8` hit a QKV sharding error during validation. EAGLE speculative decoding is still WIP on MI355X.
## Testing the deployment
Once the server is running, test it with a chat completion request in another terminal:
```bash Command
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "XiaomiMiMo/MiMo-V2-Flash",
"messages": [
{"role": "user", "content": "Hello! What can you help me with?"}
],
"temperature": 0.7,
"max_tokens": 100
}'
```
**Expected response:**
```json Config
{
"id": "...",
"object": "chat.completion",
"model": "XiaomiMiMo/MiMo-V2-Flash",
"choices": [{
"message": {
"role": "assistant",
"content": "Hello! I can help you with..."
}
}]
}
```
## Troubleshooting
**DeepGEMM Timeout Error**
Occasionally DeepGEMM timeout errors occur during first launch. Simply rerun the server command in the same container - the compiled kernels are cached and subsequent launches will be fast.
**ROCm MI355X Attention Backend**
If you see an error such as `AiterAttnBackend.forward_decode() got an unexpected keyword argument 'sinks'` on MI355X, use the `MI355X` + `Performance Optimizations` command from the selector above, which switches to Triton attention and keeps `--disable-custom-all-reduce`.
docs_new/cookbook/autoregressive/intro.mdx
+118
View File
@@ -0,0 +1,118 @@
---
title: Overview
mode: wide
description: Practical guides for deploying and using large language models and vision language models with SGLang.
metatags:
description: "Explore SGLang autoregressive model cookbooks for LLM and VLM deployment, invocation, optimization, and benchmarking examples."
---
<CardGroup cols={3}>
<Card
title="Qwen"
mode="card"
href="/cookbook/autoregressive/Qwen/Qwen3.6"
img="/cards/logos/qwen.png"
/>
<Card
title="DeepSeek"
mode="card"
href="/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2"
img="/cards/logos/deepseek.png"
/>
<Card
title="Llama"
mode="card"
href="/cookbook/autoregressive/Llama/Llama3.3-70B"
img="/cards/logos/llama.png"
/>
<Card
title="GLM"
mode="card"
href="/cookbook/autoregressive/GLM/GLM-4.5"
img="/cards/logos/glm.png"
/>
<Card
title="Google"
mode="card"
href="/cookbook/autoregressive/Google/Gemma4"
img="/cards/logos/google.png"
/>
<Card
title="OpenAI"
mode="card"
href="/cookbook/autoregressive/OpenAI/GPT-OSS"
img="/cards/logos/openai.png"
/>
<Card
title="Moonshotai"
mode="card"
href="/cookbook/autoregressive/Moonshotai/Kimi-K2.6"
img="/cards/logos/moonshotai.png"
/>
<Card
title="MiniMax"
mode="card"
href="/cookbook/autoregressive/MiniMax/MiniMax-M2.5"
img="/cards/logos/minimax.png"
/>
<Card
title="NVIDIA"
mode="card"
href="/cookbook/autoregressive/NVIDIA/Nemotron3-Nano"
img="/cards/logos/nvidia.png"
/>
<Card
title="Ernie"
mode="card"
href="/cookbook/autoregressive/Ernie/Ernie4.5"
img="/cards/logos/ernie.png"
/>
<Card
title="StepFun"
mode="card"
href="/cookbook/autoregressive/StepFun/Step3.5"
img="/cards/logos/stepfun.png"
/>
<Card
title="InclusionAI"
mode="card"
href="/cookbook/autoregressive/InclusionAI/Ling-2.5-1T"
img="/cards/logos/inclusionai.png"
/>
<Card
title="InternLM"
mode="card"
href="/cookbook/autoregressive/InternLM/Intern-S1"
img="/cards/logos/internlm.png"
/>
<Card
title="InternVL"
mode="card"
href="/cookbook/autoregressive/InternVL/InternVL3.5"
img="/cards/logos/internvl.png"
/>
<Card
title="Jina AI"
mode="card"
href="/cookbook/autoregressive/Jina/Jina-reranker-m0"
img="/cards/logos/jina.png"
/>
<Card
title="Mistral"
mode="card"
href="/cookbook/autoregressive/Mistral/Ministral-3"
img="/cards/logos/mistral.png"
/>
<Card
title="Xiaomi"
mode="card"
href="/cookbook/autoregressive/Xiaomi/MiMo-V2-Flash"
img="/cards/logos/xiaomi.png"
/>
<Card
title="FlashLabs"
mode="card"
href="/cookbook/autoregressive/FlashLabs/Chroma1.0"
img="/cards/logos/flashlabs.png"
/>
</CardGroup>