minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[Docs] Rename docs_new/ to docs/ (#32123)

Browse Source 
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
zijiexia
2026-08-03 16:51:00 -07:00
committed by GitHub
co-authored by Claude Opus 4.8
parent c949e91f18
commit b819d2fb5b
491 changed files with 122 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/DeepSeek/DeepSeek-Math-V2.mdx
+526
View File
@@ -0,0 +1,526 @@
---
title: DeepSeek-Math-V2
metatags:
description: "Deploy DeepSeek-Math-V2 with SGLang - advanced mathematical reasoning model with gold-level IMO/CMO performance and theorem-proving capabilities."
---
import { DeepSeekMathV2Deployment } from '/src/snippets/autoregressive/deepseek-math-v2-deployment.jsx';
## 1. Model Introduction
[DeepSeek-Math-V2](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2) is DeepSeek's advanced mathematical reasoning model with strong theorem-proving capabilities. The model demonstrates exceptional performance on mathematical competitions, achieving gold-level scores on IMO 2025 and CMO 2024, and a near-perfect 118/120 on Putnam 2024 with scaled test-time compute.
**Key Features:**
- **Strong Theorem-Proving**: Gold-level performance on IMO 2025 and CMO 2024
- **Self-Verifiable Reasoning**: Implements self-verifiable mathematical reasoning for improved accuracy
- **Competition-Level Math**: Near-perfect score (118/120) on Putnam 2024
- **Large MoE Model**: ~671B total parameters, requires high-memory GPUs (B200 183GB or B300 275GB)
**Available Models:**
- **BF16 (Full Weights)**: [deepseek-ai/DeepSeek-Math-V2](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2) - Full precision weights
**License:**
To use DeepSeek-Math-V2, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2/blob/main/LICENSE) for details.
## 2. SGLang Installation
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy.
<DeepSeekMathV2Deployment />
<Warning>
DeepSeek-Math-V2 is built on DeepSeek-V3.2 and uses DSA sparse attention. All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on this model.
</Warning>
### 3.2 Configuration Tips
**Hardware Requirements:**
- **B200 (183GB)**: BF16 tp=8
- **B300 (275GB)**: BF16 tp=8
**DP Attention:**
- Enable DP attention for high-throughput scenarios
- The `--dp` value commonly matches the `--tp` value
- Trade-off: Higher throughput at the cost of slightly increased latency
## 4. Model Invocation
### 4.1 Deployment Command
Deploy the model using the command generated above. Example for B200:
```shell Command
sglang serve --model-path deepseek-ai/DeepSeek-Math-V2 \
--tp 8 \
--ep 8 \
--reasoning-parser deepseek-r1 \
--host 0.0.0.0 \
--port 30000
```
### 4.2 Mathematical Reasoning
DeepSeek-Math-V2 excels at mathematical problem-solving with step-by-step reasoning.
**Streaming with Thinking Process:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Mathematical reasoning problem
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-Math-V2",
messages=[
{"role": "user", "content": "Prove that for any positive integer n, the sum 1 + 2 + 3 + ... + n = n(n+1)/2"}
],
max_tokens=4096,
stream=True
)
# Process the stream
thinking_started = False
has_thinking = False
has_answer = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
We need to prove that for any positive integer n, the sum 1 + 2 + 3 + ... + n = n(n+1)/2.
This is a classic formula for the sum of the first n natural numbers. We can prove by induction.
Base case: n=1, LHS = 1, RHS = 1*(1+1)/2 = 1*2/2 = 1. Holds.
Inductive step: Assume true for n = k, i.e., 1 + 2 + ... + k = k(k+1)/2. Then for n = k+1, sum = 1 + 2 + ... + k + (k+1) = [k(k+1)/2] + (k+1) = (k(k+1) + 2(k+1))/2 = (k+1)(k+2)/2 = (k+1
)((k+1)+1)/2. So holds for k+1. By induction, holds for all positive integers n.
...
=============== Content =================
We can prove the well-known formula for the sum of the first \(n\) positive integers in several ways. Two of the most elementary are presented below.
---
### 1. Proof by mathematical induction
**Base case (\(n=1\))**:
\[
1 = \frac{1\cdot(1+1)}{2}= \frac{1\cdot2}{2}=1,
\]
so the formula holds for \(n=1\).
**Inductive hypothesis:**
Assume that for some positive integer \(k\) the formula is true, i.e.
\[
1+2+\dots+k = \frac{k(k+1)}{2}.
\]
**Inductive step (\(k \to k+1\))**:
Consider the sum up to \(k+1\):
\[
\begin{aligned}
1+2+\dots+k+(k+1) &= \bigl(1+2+\dots+k\bigr) + (k+1) \\[4pt]
&= \frac{k(k+1)}{2} + (k+1) \qquad\text{(by the induction hypothesis)}\\[4pt]
&= (k+1)\left(\frac{k}{2}+1\right)\\[4pt]
&= (k+1)\frac{k+2}{2}\\[4pt]
&= \frac{(k+1)(k+2)}{2}\\[4pt]
&= \frac{(k+1)\bigl((k+1)+1\bigr)}{2}.
\end{aligned}
\]
Thus the formula also holds for \(n=k+1\).
By the principle of mathematical induction,
\[
1+2+3+\dots+n = \frac{n(n+1)}{2}
\]
for every positive integer \(n\).
---
### 2. Proof by pairing (Gauss’s trick)
Let
\[
S = 1 + 2 + 3 + \dots + n.
\]
Write the same sum in reverse order:
\[
S = n + (n-1) + (n-2) + \dots + 1.
\]
Add the two equalities term‑by‑term:
\[
\begin{aligned}
2S &= (1+n) + \bigl(2+(n-1)\bigr) + \bigl(3+(n-2)\bigr) + \dots + (n+1)\\
&= \underbrace{(n+1)+(n+1)+\dots+(n+1)}_{n\ \text{times}}\\
&= n\,(n+1).
\end{aligned}
\]
Therefore
\[
S = \frac{n(n+1)}{2}.
\]
Both proofs are rigorous and show that the formula holds for all positive integers \(n\).
```
### 4.3 Competition-Level Problems
**Example: IMO-style Problem:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# IMO-style problem
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-Math-V2",
messages=[
{"role": "user", "content": "Let a, b, c be positive real numbers such that abc = 1. Prove that (a-1+1/b)(b-1+1/c)(c-1+1/a) <= 1."}
],
max_tokens=8192,
stream=True
)
# Process the stream
thinking_started = False
has_thinking = False
has_answer = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
if delta.content:
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
We need to prove that for positive real numbers a,b,c with abc = 1, we have:
\[
(a - 1 + \frac{1}{b})(b - 1 + \frac{1}{c})(c - 1 + \frac{1}{a}) \le 1.
\]
We can rewrite the expressions: Since abc=1, we have 1/b = ac, 1/c = ab, 1/a = bc. Wait careful: abc=1 => 1/b = ac? Actually 1/b = ac? Let's check: abc=1 => ac = 1/b? Multiply both sides by something: abc=1 => (ac) b = 1 => ac = 1/b. Yes, because (ac) * b = 1 => ac = 1/b. Similarly, ab = 1/c, bc = 1/a. So we can rewrite:
...
=============== Content =================
We are given positive real numbers \(a,b,c\) with \(abc=1\). We must prove
\[
\Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr)\le 1 .
\]
---
### 1. A convenient substitution
Because \(abc=1\), we can write
\[
a=\frac{x}{y},\qquad b=\frac{y}{z},\qquad c=\frac{z}{x}
\]
with positive numbers \(x,y,z\).
(For instance, take \(x=1,\;y=\frac1a,\;z=\frac1{ab}\); then indeed \(a=\frac{x}{y},\;b=\frac{y}{z}\) and, using \(abc=1\), we obtain \(c=\frac{z}{x}=\frac1{ab}=c\).)
---
### 2. Rewriting the factors
\[
\begin{aligned}
a-1+\frac1b &=\frac{x}{y}-1+\frac{z}{y}= \frac{x+z-y}{y},\\[2mm]
b-1+\frac1c &=\frac{y}{z}-1+\frac{x}{z}= \frac{x+y-z}{z},\\[2mm]
c-1+\frac1a &=\frac{z}{x}-1+\frac{y}{x}= \frac{y+z-x}{x}.
\end{aligned}
\]
Hence the product becomes
\[
P=\Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr)
=\frac{(x+z-y)(x+y-z)(y+z-x)}{xyz}.
\]
---
### 3. Reducing to a known inequality
We have to show \(P\le1\), i.e.
\[
(x+z-y)(x+y-z)(y+z-x)\le xyz .
\tag{1}
\]
Set
\[
p=x+y+z,\qquad q=xy+yz+zx,\qquad r=xyz .
\]
Notice that
\[
x+z-y=p-2y,\quad x+y-z=p-2z,\quad y+z-x=p-2x .
\]
Therefore
\[
\begin{aligned}
(x+z-y)(x+y-z)(y+z-x)
&=(p-2x)(p-2y)(p-2z)\\
&=p^{3}-2p^{2}(x+y+z)+4p(xy+yz+zx)-8xyz\\
&=-p^{3}+4pq-8r .
\end{aligned}
\]
Inequality (1) is thus equivalent to
\[
-p^{3}+4pq-8r\le r\quad\Longleftrightarrow\quad 4pq-p^{3}\le 9r .
\tag{2}
\]
---
### 4. Applying Schur’s inequality
Schur’s inequality of third degree states that for any non‑negative \(x,y,z\)
\[
p^{3}+9r\ge 4pq .
\]
Rearranged, this is exactly \(4pq-p^{3}\le 9r\), which is (2).
Since our \(x,y,z\) are positive, Schur’s inequality applies and (2) holds.
Consequently (1) is true, and we obtain \(P\le1\).
---
### 5. Equality case
Equality in Schur’s inequality for positive numbers occurs only when \(x=y=z\).
Then \(a=b=c=1\), and indeed the product equals \(1\).
---
Thus for all positive \(a,b,c\) with \(abc=1\),
\[
\Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr)\le 1 .
\]
∎
```
## 5. Benchmark
### 5.1 Accuracy Benchmark
#### 5.1.1 GSM8K Benchmark
**Benchmark Command:**
```shell Command
python3 benchmark/gsm8k/bench_sglang.py --num-questions 200 --port 30000
```
**Test Results:**
```text Output
Accuracy: 0.975
Invalid: 0.000
Latency: 34.358 s
Output throughput: 540.162 token/s
```
### 5.2 Speed Benchmark
**Test Environment:**
- Hardware: NVIDIA B200 GPU (8x, 183GB each)
- Model: DeepSeek-Math-V2
- Tensor Parallelism: 8
- SGLang Version: 0.5.8
#### 5.2.1 Latency Benchmark
**Benchmark Command:**
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model deepseek-ai/DeepSeek-Math-V2 \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 10 \
--max-concurrency 1
```
**Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 53.34
Total input tokens: 1972
Total input text tokens: 1972
Total generated tokens: 2784
Total generated tokens (retokenized): 2778
Request throughput (req/s): 0.19
Input token throughput (tok/s): 36.97
Output token throughput (tok/s): 52.19
Peak output token throughput (tok/s): 56.00
Peak concurrent requests: 3
Total token throughput (tok/s): 89.16
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 5330.72
Median E2E Latency (ms): 5879.28
P90 E2E Latency (ms): 8320.33
P99 E2E Latency (ms): 9921.29
---------------Time to First Token----------------
Mean TTFT (ms): 183.38
Median TTFT (ms): 177.92
P99 TTFT (ms): 217.64
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 17.96
Median TPOT (ms): 18.39
P99 TPOT (ms): 19.03
---------------Inter-Token Latency----------------
Mean ITL (ms): 18.57
Median ITL (ms): 18.63
P95 ITL (ms): 19.26
P99 ITL (ms): 19.48
Max ITL (ms): 24.93
==================================================
```
#### 5.2.2 Throughput Benchmark
**Benchmark Command:**
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model deepseek-ai/DeepSeek-Math-V2 \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 1000 \
--max-concurrency 100
```
**Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 217.36
Total input tokens: 301701
Total input text tokens: 301701
Total generated tokens: 188375
Total generated tokens (retokenized): 187456
Request throughput (req/s): 4.60
Input token throughput (tok/s): 1388.05
Output token throughput (tok/s): 866.67
Peak output token throughput (tok/s): 2589.00
Peak concurrent requests: 109
Total token throughput (tok/s): 2254.72
Concurrency: 89.81
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 19521.73
Median E2E Latency (ms): 12076.76
P90 E2E Latency (ms): 47248.87
P99 E2E Latency (ms): 86862.79
---------------Time to First Token----------------
Mean TTFT (ms): 790.40
Median TTFT (ms): 456.81
P99 TTFT (ms): 4223.33
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 106.52
Median TPOT (ms): 107.24
P99 TPOT (ms): 238.33
---------------Inter-Token Latency----------------
Mean ITL (ms): 100.29
Median ITL (ms): 38.34
P95 ITL (ms): 237.00
P99 ITL (ms): 347.49
Max ITL (ms): 3642.56
==================================================
```
docs/cookbook/autoregressive/DeepSeek/DeepSeek-OCR-2.mdx
+255
View File
@@ -0,0 +1,255 @@
---
title: DeepSeek-OCR-2
metatags:
description: "Deploy DeepSeek-OCR-2 with SGLang - high-accuracy text extraction from images and documents for OCR tasks."
---
import { DeepSeekOCR2Deployment } from '/src/snippets/autoregressive/deepseek-ocr-v2-deployment.jsx';
## 1. Model Introduction
[DeepSeek-OCR-2](https://github.com/deepseek-ai/DeepSeek-OCR-2) is DeepSeek's next-generation OCR (Optical Character Recognition) model, building on DeepSeek-OCR with improved accuracy and broader document understanding capabilities. The model is optimized for high-accuracy text extraction from images across a wide variety of document types and formats.
**Key Features:**
- **Semantic-Aware Visual Encoding (DeepEncoder V2)**: DeepSeek-OCR-2 introduces DeepEncoder V2, which models document reading order in a more human-like, semantic-driven manner rather than relying on fixed raster scanning. This significantly improves logical reading flow in complex layouts (e.g., multi-column documents).
- **Stronger Layout and Structural Understanding**: DeepSeek-OCR-2 demonstrates improved performance on structured documents such as tables, forms, and dense multi-column pages. It reduces reading-order errors and improves overall document parsing robustness compared to the original version.
- **Improved Accuracy While Maintaining Token Efficiency**: The original DeepSeek-OCR emphasized aggressive visual token compression. OCR-2 maintains high token efficiency while delivering higher benchmark performance, particularly on document-level understanding tasks.
- **Better Generalization Across Complex Document Tasks**: DeepSeek-OCR-2 performs more consistently across multilingual documents, structured data extraction, and visually complex content, making it more suitable for real-world document intelligence scenarios beyond plain text OCR.
**Available Models:**
- **Base Model**: [deepseek-ai/DeepSeek-OCR-2](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2) - Recommended for OCR tasks
**License:**
To use DeepSeek-OCR-2, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2/blob/main/LICENSE.txt) for details.
For more details, please refer to the [official DeepSeek-OCR-2 repository](https://github.com/deepseek-ai/DeepSeek-OCR-2).
## 2. SGLang Installation
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy. SGLang supports serving DeepSeek-OCR-2 on NVIDIA H200 and B200, AMD MI300X, MI355X, and MI325X GPUs, as well as Intel Xeon CPUs.
<DeepSeekOCR2Deployment />
**Note**: DeepSeek-OCR-2 has ~3B parameters and easily fits on a single modern GPU. For low-latency serving, no model parallelism is needed. For high-throughput requirements, consider using data parallelism with the SGLang Model Gateway — see [DP, DPA and SGLang DP Router](../../../docs/advanced_features/sgl_model_gateway) for more details.
### 3.2 Configuration Tips
- **Single GPU Deployment:** DeepSeek-OCR-2 (~3B parameters) fits on a single modern GPU — no tensor parallelism required for low-latency serving.
- **High Throughput:** For high-throughput scenarios, use data parallelism with the SGLang Model Gateway. See [DP, DPA and SGLang DP Router](../../../docs/advanced_features/sgl_model_gateway).
- **NCCL timeout:** If model loading is slow, increase `--dist-timeout 3600`.
- **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for NUMA binding settings.
## 4. Model Invocation
### 4.1 Basic Usage
**OpenAI-compatible request example**
```python Example
import requests
url = "http://localhost:30000/v1/chat/completions"
data = {
"model": "deepseek-ai/DeepSeek-OCR-2",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "<image>\n<|grounding|>Convert the document to markdown."},
{"type": "image_url", "image_url": {"url": "https://example.com/your_image.jpg"}},
],
}
],
"max_tokens": 512,
}
response = requests.post(url, json=data)
print(response.text)
```
**Reference**
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Recommended Prompts
The following prompts are recommended by the [official model card](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2#main-prompts).
**Structured document conversion** — extracts text while preserving layout:
```text Example
<image>
<|grounding|>Convert the document to markdown.
```
**Free-form OCR** — extracts without layouts:
```text Example
<image>
Free OCR.
```
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: NVIDIA H200 GPU (1x)
- Model: DeepSeek-OCR-2
- Tensor Parallelism: 1
- sglang version: 0.0.0.dev1+g93fca0bbc
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses. For more details on how to perform evaluation, see [Evaluating New Models with SGLang](../../../docs/developer_guide/evaluating_new_models).
#### 5.1.1 Latency-Sensitive Benchmark
- Model Deployment Command:
```shell Command
sglang serve \
--model-path deepseek-ai/DeepSeek-OCR-2 \
--enable-multimodal \
--host 0.0.0.0 \
--port 30000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 0.0.0.0 \
--port 30000 \
--model deepseek-ai/DeepSeek-OCR-2 \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 10 \
--max-concurrency 1
```
- **Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 3.54
Total input tokens: 1972
Total input text tokens: 1972
Total generated tokens: 2784
Total generated tokens (retokenized): 2710
Request throughput (req/s): 2.83
Input token throughput (tok/s): 557.53
Output token throughput (tok/s): 787.10
Peak output token throughput (tok/s): 818.00
Peak concurrent requests: 5
Total token throughput (tok/s): 1344.63
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 352.69
Median E2E Latency (ms): 392.34
P90 E2E Latency (ms): 540.64
P99 E2E Latency (ms): 639.01
---------------Time to First Token----------------
Mean TTFT (ms): 18.08
Median TTFT (ms): 16.57
P99 TTFT (ms): 25.67
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 1.18
Median TPOT (ms): 1.21
P99 TPOT (ms): 1.22
---------------Inter-Token Latency----------------
Mean ITL (ms): 1.21
Median ITL (ms): 1.21
P95 ITL (ms): 1.28
P99 ITL (ms): 1.44
Max ITL (ms): 4.32
==================================================
```
#### 5.1.2 Throughput-Sensitive Benchmark
- Model Deployment Command:
```shell Command
sglang serve \
--model-path deepseek-ai/DeepSeek-OCR-2 \
--enable-multimodal \
--tp 1 \
--ep 1 \
--dp 1 \
--enable-dp-attention \
--host 0.0.0.0 \
--port 30000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 0.0.0.0 \
--port 30000 \
--model deepseek-ai/DeepSeek-OCR-2 \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 1000 \
--max-concurrency 100
```
- **Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 14.79
Total input tokens: 301698
Total input text tokens: 301698
Total generated tokens: 188375
Total generated tokens (retokenized): 185236
Request throughput (req/s): 67.63
Input token throughput (tok/s): 20402.54
Output token throughput (tok/s): 12738.99
Peak output token throughput (tok/s): 17508.00
Peak concurrent requests: 187
Total token throughput (tok/s): 33141.53
Concurrency: 86.87
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 1284.50
Median E2E Latency (ms): 866.07
P90 E2E Latency (ms): 3027.32
P99 E2E Latency (ms): 5490.63
---------------Time to First Token----------------
Mean TTFT (ms): 86.08
Median TTFT (ms): 50.09
P99 TTFT (ms): 613.92
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 7.79
Median TPOT (ms): 6.54
P99 TPOT (ms): 50.10
---------------Inter-Token Latency----------------
Mean ITL (ms): 6.42
Median ITL (ms): 4.64
P95 ITL (ms): 23.65
P99 ITL (ms): 39.62
Max ITL (ms): 452.65
==================================================
```
docs/cookbook/autoregressive/DeepSeek/DeepSeek-OCR.mdx
+247
View File
@@ -0,0 +1,247 @@
---
title: DeepSeek-OCR
metatags:
description: "Deploy DeepSeek-OCR with SGLang - high-accuracy text extraction from images and documents for OCR tasks."
---
## 1. Model Introduction
[DeepSeek-OCR](https://github.com/deepseek-ai/DeepSeek-OCR) is DeepSeek's advanced OCR (Optical Character Recognition) model designed for high-accuracy text extraction from images. The model is optimized for various document processing and image-to-text conversion tasks.
**Key Features:**
- **Advanced OCR**: High-accuracy text recognition from images and documents
- **Multi-Modality**: Supports various image formats and document types
**Available Models:**
- **Base Model**: [deepseek-ai/DeepSeek-OCR](https://huggingface.co/deepseek-ai/DeepSeek-OCR) - Recommended for OCR tasks
**License:**
To use DeepSeek-OCR, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-OCR/blob/main/LICENSE) for details.
For more details, please refer to the [official DeepSeek-OCR repository](https://github.com/deepseek-ai/DeepSeek-OCR).
## 2. SGLang Installation
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy.
import { DeepSeekOCRDeployment } from "/src/snippets/autoregressive/deepseek-ocr-deployment.jsx";
<DeepSeekOCRDeployment />
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 OCR-Specific Prompts
DeepSeek-OCR accepts recommended prompts from the model card:
```text
<image>
<|grounding|>Convert the document to markdown.
```
```text
<image>
Free OCR.
```
**OpenAI-compatible image request example:**
```python Example
import requests
url = "http://localhost:30000/v1/chat/completions"
data = {
"model": "deepseek-ai/DeepSeek-OCR",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "<image>\n<|grounding|>Convert the document to markdown."
},
{
"type": "image_url",
"image_url": {"url": "https://example.com/your_image.jpg"}
},
],
}
],
"max_tokens": 512,
}
response = requests.post(url, json=data)
print(response.text)
```
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: AMD MI300X GPU (1x)
- Model: DeepSeek-OCR
- Tensor Parallelism: 1
- sglang version: 0.5.7
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses.
#### 5.1.1 Latency-Sensitive Benchmark
- Model Deployment Command:
```shell Command
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-OCR \
--tp 1 \
--dtype float16 \
--trust-remote-code \
--host 0.0.0.0 \
--port 8000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 8000 \
--model deepseek-ai/DeepSeek-OCR \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 10 \
--max-concurrency 1
```
- **Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 4.45
Total input tokens: 1972
Total input text tokens: 1972
Total input vision tokens: 0
Total generated tokens: 2784
Total generated tokens (retokenized): 2770
Request throughput (req/s): 2.25
Input token throughput (tok/s): 442.89
Output token throughput (tok/s): 625.26
Peak output token throughput (tok/s): 635.00
Peak concurrent requests: 4
Total token throughput (tok/s): 1068.16
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 443.32
Median E2E Latency (ms): 493.29
---------------Time to First Token----------------
Mean TTFT (ms): 21.59
Median TTFT (ms): 20.89
P99 TTFT (ms): 24.81
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 1.47
Median TPOT (ms): 1.52
P99 TPOT (ms): 1.53
---------------Inter-Token Latency----------------
Mean ITL (ms): 1.52
Median ITL (ms): 1.51
P95 ITL (ms): 1.76
P99 ITL (ms): 1.93
Max ITL (ms): 8.28
==================================================
```
#### 5.1.2 Throughput-Sensitive Benchmark
- Model Deployment Command:
```shell Command
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-OCR \
--tp 1 \
--ep 1 \
--dp 1 \
--enable-dp-attention \
--dtype float16 \
--host 0.0.0.0 \
--port 8000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 8000 \
--model deepseek-ai/DeepSeek-OCR \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 1000 \
--max-concurrency 100
```
- **Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 16.24
Total input tokens: 301698
Total input text tokens: 301698
Total input vision tokens: 0
Total generated tokens: 188375
Total generated tokens (retokenized): 186927
Request throughput (req/s): 61.59
Input token throughput (tok/s): 18582.90
Output token throughput (tok/s): 11602.84
Peak output token throughput (tok/s): 15479.00
Peak concurrent requests: 179
Total token throughput (tok/s): 30185.75
Concurrency: 85.53
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 1388.60
Median E2E Latency (ms): 901.43
---------------Time to First Token----------------
Mean TTFT (ms): 73.36
Median TTFT (ms): 50.21
P99 TTFT (ms): 349.53
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 7.42
Median TPOT (ms): 7.31
P99 TPOT (ms): 27.99
---------------Inter-Token Latency----------------
Mean ITL (ms): 7.04
Median ITL (ms): 4.62
P95 ITL (ms): 21.11
P99 ITL (ms): 36.92
Max ITL (ms): 172.15
==================================================
```
docs/cookbook/autoregressive/DeepSeek/DeepSeek-R1.mdx
+993
View File
@@ -0,0 +1,993 @@
---
title: DeepSeek-R1
metatags:
description: "Deploy DeepSeek-R1 reasoning model with SGLang - advanced step-by-step reasoning with FP8/FP4 quantization for NVIDIA and AMD GPUs."
---
import { DeepSeekR1BasicDeployment } from '/src/snippets/autoregressive/deepseek-r1-basic-deployment.jsx';
import { DeepSeekR1AdvancedDeployment } from '/src/snippets/autoregressive/deepseek-r1-advanced-deployment.jsx';
## 1. Model Introduction
[DeepSeek-R1](https://github.com/deepseek-ai/DeepSeek-R1) is DeepSeek's advanced reasoning model that combines powerful language understanding with step-by-step reasoning capabilities. The model is available in multiple quantization formats optimized for different hardware platforms.
**Key Features:**
- **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving
- **Multiple Quantizations**: FP8 and FP4 variants for different performance/memory trade-offs
- **Hardware Optimization**: Specifically tuned for NVIDIA B200 (Blackwell) and H200 (Hopper) GPUs, AMD MI300X, MI325X and MI355X GPUs, as well as Intel Xeon CPUs
- **High Performance**: Optimized for both throughput and latency scenarios
**Available Models:**
- **FP8 (8-bit quantized)**: [deepseek-ai/DeepSeek-R1-0528](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528) - Recommended for H200 and MI300X
- **FP4 (4-bit quantized)**: [nvidia/DeepSeek-R1-0528-FP4-v2](https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2) - Recommended for B200 and MI355X
- **BF16 (upcast from FP8)**: [unsloth/DeepSeek-R1-0528-BF16](https://huggingface.co/unsloth/DeepSeek-R1-0528-BF16)
- **INT8 (channel-wise)**: [meituan/DeepSeek-R1-Channel-INT8](https://huggingface.co/meituan/DeepSeek-R1-Channel-INT8)
- **W4A8**: [novita/Deepseek-R1-0528-W4AFP8](https://huggingface.co/novita/Deepseek-R1-0528-W4AFP8)
- **AWQ (4-bit)**: [QuixiAI/DeepSeek-R1-0528-AWQ](https://huggingface.co/QuixiAI/DeepSeek-R1-0528-AWQ)
- **MXFP4**: [amd/DeepSeek-R1-MXFP4](https://huggingface.co/amd/DeepSeek-R1-MXFP4)
**License:**
To use DeepSeek-R1, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528/blob/main/LICENSE) for details.
For more details, please refer to the [official DeepSeek-R1 repository](https://github.com/deepseek-ai/DeepSeek-R1).
## 2. SGLang Installation
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate a basic deployment command for your hardware platform, quantization method, and deployment strategy.
<DeepSeekR1BasicDeployment />
### 3.2 Optimal Configurations
Pareto-optimal configurations for B200, H200, MI300X, MI325X, and MI355X hardware.
<DeepSeekR1AdvancedDeployment />
### 3.3 Configuration Tips
DeepSeek-R1 shares the same MoE architecture as DeepSeek-V3, so the same hardware and optimization recommendations apply.
**Recommended GPU configurations by weight type:**
<table style={{width: "100%", borderCollapse: "collapse"}}>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>Weight Type</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Supported Hardware</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>FP8</strong> (recommended)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8× H200, 8× B200, 8× MI300X, 2×8× H100/H800/H20</td>
</tr>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>BF16</strong> (upcast from FP8)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>2×8× H200, 2×8× MI300X, 4×8× H100/H800, 4×8× A100/A800</td>
</tr>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>INT8</strong></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Atlas 800I A3</td>
</tr>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>W4A8 / AWQ / MXFP4 / NVFP4</strong></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8× H20/H100, 4× H200; 8× H100/A100; 8/4× MI355X/MI350X; 8/4× B200</td>
</tr>
</tbody>
</table>
> The official DeepSeek-R1 checkpoint is already in FP8 format — do **not** add `--quantization fp8` when serving it.
**DeepGEMM precompilation (NVIDIA Hopper / Blackwell):** Precompile GEMM kernels to avoid JIT overhead (~10 min):
```bash
python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-R1 --tp 8 --trust-remote-code
```
**Data Parallelism Attention (`--enable-dp-attention`):** Recommended for high-throughput scenarios. Use `--enable-dp-attention --tp 8 --dp 8` on a single 8-GPU node.
**NCCL timeout:** If model loading is slow, increase: `--dist-timeout 3600`.
**Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
DeepSeek-R1 supports advanced reasoning capabilities with built-in thinking process. Enable the reasoning parser during deployment to separate the thinking and content sections:
```shell Command
python -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-R1-0528 \
--reasoning-parser deepseek-r1 \
--tp 8
```
**Streaming with Thinking Process:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-R1-0528",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
To solve this problem, I need to calculate 15% of 240.
Step 1: Convert 15% to decimal: 15% = 0.15
Step 2: Multiply 240 by 0.15
Step 3: 240 × 0.15 = 36
=============== Content =================
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
```
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
#### 4.2.2 Tool Calling
DeepSeek-R1 supports tool calling capabilities. Enable the tool call parser:
```shell Command
python -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-R1-0528 \
--reasoning-parser deepseek-r1 \
--tool-call-parser deepseekv3 \
--chat-template examples/chat_template/tool_chat_template_deepseekr1.jinja \
--tp 8
```
**Python Example (with Thinking Process):**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-R1-0528",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
if tool_call.function:
print(f"🔧 Tool Call: {tool_call.function.name}")
print(f" Arguments: {tool_call.function.arguments}")
# Print content
if delta.content:
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
I should call the function with location="Beijing".
=============== Content =================
🔧 Tool Call: get_weather
Arguments:
🔧 Tool Call: None
Arguments: {"location": "Beijing"}
```
**Note:**
- The reasoning parser shows how the model decides to use a tool
- Tool calls are clearly marked with the function name and arguments
- You can then execute the function and send the result back to continue the conversation
**Handling Tool Call Results:**
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-R1-0528",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```
#### 4.2.3 Multi-Token Prediction (EAGLE Speculative Decoding)
DeepSeek-R1 supports EAGLE-based Multi-Token Prediction (MTP), the same mechanism as DeepSeek-V3. Refer to [DeepSeek-V3 §4.2.3](/cookbook/autoregressive/DeepSeek/DeepSeek-V3#4-2-3-multi-token-prediction-eagle-speculative-decoding) for the complete launch command, flag reference, tuning guidance (`--speculative-num-steps`, `--speculative-eagle-topk`, `--max-running-requests`), and `bench_speculative.py` link. R1's speed benchmark commands that include `--speculative-*` flags use this mechanism.
#### 4.2.4 Thinking Budget
Limit the model's thinking token budget using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`:
```shell Command
python3 -m sglang.launch_server \
--model deepseek-ai/DeepSeek-R1 \
--tp 8 \
--port 30000 \
--reasoning-parser deepseek-r1 \
--enable-custom-logit-processor
```
```python Example
import openai
from sglang.srt.sampling.custom_logit_processor import DeepSeekR1ThinkingBudgetLogitProcessor
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-R1",
messages=[{"role": "user", "content": "Is Paris the Capital of France?"}],
max_tokens=1024,
extra_body={
"custom_logit_processor": DeepSeekR1ThinkingBudgetLogitProcessor().to_str(),
"custom_params": {"thinking_budget": 512},
},
)
print(response)
```
## 5. Benchmark
This section uses **industry-standard configurations** for comparable benchmark results.
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: B200 GPU (8x)
- Model: DeepSeek-R1-0528
- Tensor Parallelism: 8
- SGLang Version: 0.5.6.post1
**Benchmark Methodology:**
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
#### 5.1.1 Standard Test Scenarios
Three core scenarios reflect real-world usage patterns:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
</tr>
</tbody>
</table>
#### 5.1.2 Concurrency Levels
Test each scenario at different concurrency levels to capture the throughput vs. latency trade-off:
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
#### 5.1.3 Number of Prompts
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
---
#### 5.1.4 Benchmark Commands
**Scenario 1: Chat (1K/1K) - Most Important**
- **Model Deployment**
```bash Command
python -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-R1-0528 \
--tp 8
```
- Low Concurrency (Latency-Optimized)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-R1-0528 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 40.00
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 4210
Total generated tokens (retokenized): 4205
Request throughput (req/s): 0.25
Input token throughput (tok/s): 152.52
Output token throughput (tok/s): 105.24
Peak output token throughput (tok/s): 110.00
Peak concurrent requests: 2
Total token throughput (tok/s): 257.76
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 3998.40
Median E2E Latency (ms): 3207.53
---------------Time to First Token----------------
Mean TTFT (ms): 153.00
Median TTFT (ms): 140.76
P99 TTFT (ms): 214.66
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 9.16
Median TPOT (ms): 9.15
P99 TPOT (ms): 9.21
---------------Inter-Token Latency----------------
Mean ITL (ms): 9.16
Median ITL (ms): 9.15
P95 ITL (ms): 9.47
P99 ITL (ms): 9.63
Max ITL (ms): 15.45
==================================================
```
- Medium Concurrency (Balanced)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-R1-0528 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 51.21
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 40725
Total generated tokens (retokenized): 40458
Request throughput (req/s): 1.56
Input token throughput (tok/s): 774.66
Output token throughput (tok/s): 795.30
Peak output token throughput (tok/s): 1088.00
Peak concurrent requests: 21
Total token throughput (tok/s): 1569.96
Concurrency: 13.93
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 8918.33
Median E2E Latency (ms): 9466.16
---------------Time to First Token----------------
Mean TTFT (ms): 273.51
Median TTFT (ms): 131.71
P99 TTFT (ms): 839.57
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 17.56
Median TPOT (ms): 17.46
P99 TPOT (ms): 28.68
---------------Inter-Token Latency----------------
Mean ITL (ms): 17.02
Median ITL (ms): 14.70
P95 ITL (ms): 16.41
P99 ITL (ms): 112.38
Max ITL (ms): 461.90
==================================================
```
- High Concurrency (Throughput-Optimized)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-R1-0528 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 110.46
Total input tokens: 249831
Total input text tokens: 249831
Total input vision tokens: 0
Total generated tokens: 252162
Total generated tokens (retokenized): 251441
Request throughput (req/s): 4.53
Input token throughput (tok/s): 2261.80
Output token throughput (tok/s): 2282.90
Peak output token throughput (tok/s): 3900.00
Peak concurrent requests: 109
Total token throughput (tok/s): 4544.71
Concurrency: 92.26
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 20380.71
Median E2E Latency (ms): 19391.65
---------------Time to First Token----------------
Mean TTFT (ms): 563.14
Median TTFT (ms): 147.62
P99 TTFT (ms): 2632.11
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 40.11
Median TPOT (ms): 41.98
P99 TPOT (ms): 50.10
---------------Inter-Token Latency----------------
Mean ITL (ms): 39.37
Median ITL (ms): 26.36
P95 ITL (ms): 98.16
P99 ITL (ms): 150.08
Max ITL (ms): 2052.85
==================================================
```
**Scenario 2: Reasoning (1K/8K)**
- Low Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-R1-0528 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 411.34
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 44452
Total generated tokens (retokenized): 44390
Request throughput (req/s): 0.02
Input token throughput (tok/s): 14.83
Output token throughput (tok/s): 108.07
Peak output token throughput (tok/s): 110.00
Peak concurrent requests: 2
Total token throughput (tok/s): 122.90
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 41132.04
Median E2E Latency (ms): 44288.71
---------------Time to First Token----------------
Mean TTFT (ms): 125.76
Median TTFT (ms): 126.19
P99 TTFT (ms): 137.69
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 9.21
Median TPOT (ms): 9.20
P99 TPOT (ms): 9.27
---------------Inter-Token Latency----------------
Mean ITL (ms): 9.23
Median ITL (ms): 9.22
P95 ITL (ms): 9.64
P99 ITL (ms): 9.86
Max ITL (ms): 15.18
==================================================
```
- Medium Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-R1-0528 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 348.93
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 318226
Total generated tokens (retokenized): 317630
Request throughput (req/s): 0.23
Input token throughput (tok/s): 113.69
Output token throughput (tok/s): 912.02
Peak output token throughput (tok/s): 1088.00
Peak concurrent requests: 19
Total token throughput (tok/s): 1025.70
Concurrency: 14.07
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 61360.70
Median E2E Latency (ms): 62071.20
---------------Time to First Token----------------
Mean TTFT (ms): 176.02
Median TTFT (ms): 153.75
P99 TTFT (ms): 268.44
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 15.42
Median TPOT (ms): 15.59
P99 TPOT (ms): 16.07
---------------Inter-Token Latency----------------
Mean ITL (ms): 15.39
Median ITL (ms): 15.17
P95 ITL (ms): 16.62
P99 ITL (ms): 18.13
Max ITL (ms): 226.59
==================================================
```
- High Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-R1-0528 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 320 \
--max-concurrency 64 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 589.31
Total input tokens: 158939
Total input text tokens: 158939
Total input vision tokens: 0
Total generated tokens: 1300705
Total generated tokens (retokenized): 1297658
Request throughput (req/s): 0.54
Input token throughput (tok/s): 269.70
Output token throughput (tok/s): 2207.16
Peak output token throughput (tok/s): 2944.00
Peak concurrent requests: 68
Total token throughput (tok/s): 2476.86
Concurrency: 57.03
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 105032.36
Median E2E Latency (ms): 108229.09
---------------Time to First Token----------------
Mean TTFT (ms): 223.91
Median TTFT (ms): 158.15
P99 TTFT (ms): 474.86
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 25.94
Median TPOT (ms): 26.72
P99 TPOT (ms): 27.99
---------------Inter-Token Latency----------------
Mean ITL (ms): 25.79
Median ITL (ms): 25.37
P95 ITL (ms): 26.70
P99 ITL (ms): 105.49
Max ITL (ms): 237.91
==================================================
```
**Scenario 3: Summarization (8K/1K)**
- Low Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-R1-0528 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 40.65
Total input tokens: 41941
Total input text tokens: 41941
Total input vision tokens: 0
Total generated tokens: 4210
Total generated tokens (retokenized): 4195
Request throughput (req/s): 0.25
Input token throughput (tok/s): 1031.65
Output token throughput (tok/s): 103.56
Peak output token throughput (tok/s): 110.00
Peak concurrent requests: 2
Total token throughput (tok/s): 1135.20
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 4063.62
Median E2E Latency (ms): 3296.13
---------------Time to First Token----------------
Mean TTFT (ms): 165.91
Median TTFT (ms): 154.96
P99 TTFT (ms): 240.92
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 9.26
Median TPOT (ms): 9.27
P99 TPOT (ms): 9.42
---------------Inter-Token Latency----------------
Mean ITL (ms): 9.28
Median ITL (ms): 9.28
P95 ITL (ms): 9.66
P99 ITL (ms): 9.83
Max ITL (ms): 14.06
==================================================
```
- Medium Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-R1-0528 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 56.71
Total input tokens: 300020
Total input text tokens: 300020
Total input vision tokens: 0
Total generated tokens: 41589
Total generated tokens (retokenized): 41490
Request throughput (req/s): 1.41
Input token throughput (tok/s): 5290.75
Output token throughput (tok/s): 733.41
Peak output token throughput (tok/s): 1024.00
Peak concurrent requests: 20
Total token throughput (tok/s): 6024.16
Concurrency: 14.25
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 10098.99
Median E2E Latency (ms): 10623.46
---------------Time to First Token----------------
Mean TTFT (ms): 486.80
Median TTFT (ms): 189.59
P99 TTFT (ms): 2138.73
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 19.06
Median TPOT (ms): 19.23
P99 TPOT (ms): 30.69
---------------Inter-Token Latency----------------
Mean ITL (ms): 18.53
Median ITL (ms): 15.63
P95 ITL (ms): 16.64
P99 ITL (ms): 109.71
Max ITL (ms): 1471.36
==================================================
```
- High Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-R1-0528 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 320 \
--max-concurrency 64 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 115.55
Total input tokens: 1273893
Total input text tokens: 1273893
Total input vision tokens: 0
Total generated tokens: 169680
Total generated tokens (retokenized): 169275
Request throughput (req/s): 2.77
Input token throughput (tok/s): 11024.93
Output token throughput (tok/s): 1468.50
Peak output token throughput (tok/s): 2254.00
Peak concurrent requests: 70
Total token throughput (tok/s): 12493.43
Concurrency: 59.45
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 21465.98
Median E2E Latency (ms): 20686.26
---------------Time to First Token----------------
Mean TTFT (ms): 913.93
Median TTFT (ms): 224.92
P99 TTFT (ms): 6257.83
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 39.93
Median TPOT (ms): 40.99
P99 TPOT (ms): 60.91
---------------Inter-Token Latency----------------
Mean ITL (ms): 38.83
Median ITL (ms): 26.29
P95 ITL (ms): 113.81
P99 ITL (ms): 176.94
Max ITL (ms): 5521.53
==================================================
```
#### 5.1.5 Understanding the Results
**Key Metrics:**
- **Request Throughput (req/s)**: Number of requests processed per second
- **Output Token Throughput (tok/s)**: Total tokens generated per second
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
**Why These Configurations Matter:**
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
- **Variable Concurrency**: Captures the Pareto frontier - the optimal trade-off between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
**Interpreting Results:**
- Compare your results against baseline numbers for your hardware
- Higher throughput at same latency = better performance
- Lower TTFT = more responsive user experience
- Lower TPOT = faster generation speed
### 5.2 Accuracy Benchmark
Document model accuracy on standard benchmarks:
#### 5.2.1 GSM8K Benchmark
- Benchmark Command
```bash Command
python3 benchmark/gsm8k/bench_sglang.py \
--num-shots 8 \
--num-questions 1316 \
--parallel 1316
```
**Test Results:**
```text Output
Accuracy: 0.959
Invalid: 0.000
Latency: 29.185 s
Output throughput: 4854.672 token/s
```
docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3.mdx
+637
View File
@@ -0,0 +1,637 @@
---
title: "DeepSeek-V3"
metatags:
description: "Deploy DeepSeek-V3 MoE model with SGLang - efficient architecture with strong reasoning, coding, and tool-augmented capabilities."
---
## 1. Model Introduction
[DeepSeek V3](https://huggingface.co/deepseek-ai/DeepSeek-V3) is a large-scale Mixture-of-Experts (MoE) language model developed by DeepSeek, designed to deliver strong general-purpose reasoning, coding, and tool-augmented capabilities with high training and inference efficiency. As the latest generation in the DeepSeek model family, DeepSeek V3 introduces systematic architectural and training innovations that significantly improve performance across reasoning, mathematics, coding, and long-context understanding, while maintaining a competitive compute cost.
Key highlights include:
- **Efficient MoE architecture**: DeepSeek V3 adopts a fine-grained Mixture-of-Experts design with a large number of experts and sparse activation, enabling high model capacity while keeping inference and training costs manageable.
- **Advanced reasoning and coding**: The model demonstrates strong performance on mathematical reasoning, logical inference, and real-world coding benchmarks, benefiting from improved data curation and training strategies.
- **Long-context capability**: DeepSeek V3 supports extended context lengths, allowing it to handle long documents, complex multi-step reasoning, and agent-style workflows more effectively.
- **Tool use and function calling**: The model is trained to support structured outputs and tool invocation, enabling seamless integration with external tools and agent frameworks during inference.
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
import { DeepSeekV3Deployment } from "/src/snippets/autoregressive/deepseek-v3-deployment.jsx";
<DeepSeekV3Deployment />
### 3.2 Configuration Tips
**Recommended GPU configurations by weight type:**
<table style={{width: "100%", borderCollapse: "collapse"}}>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>Weight Type</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Supported Hardware</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>FP8</strong> (recommended)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8× H200, 8× B200, 8× MI300X, 2×8× H100/H800/H20</td>
</tr>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>BF16</strong> (upcast from FP8)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>2×8× H200, 2×8× MI300X, 4×8× H100/H800, 4×8× A100/A800</td>
</tr>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>INT8</strong></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Atlas 800I A3</td>
</tr>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>W4A8 / AWQ / MXFP4 / NVFP4</strong></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8× H20/H100, 4× H200; 8× H100/A100; 8/4× MI355X/MI350X; 8/4× B200</td>
</tr>
</tbody>
</table>
> The official DeepSeek-V3 checkpoint is already in FP8 format — do **not** add `--quantization fp8` when serving it.
**DeepGEMM precompilation (NVIDIA Hopper / Blackwell):** Precompile GEMM kernels before the first server run to avoid JIT overhead (~10 min):
```bash
python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-V3 --tp 8 --trust-remote-code
```
DeepGEMM is enabled by default on Hopper/Blackwell and can be disabled with `SGLANG_ENABLE_JIT_DEEPGEMM=0`.
**Data Parallelism Attention (`--enable-dp-attention`):** Recommended for high-throughput scenarios with large batch sizes. Reduces KV-cache duplication across TP ranks. Use `--enable-dp-attention --tp 8 --dp 8` on a single 8-GPU node. Not recommended for low-latency, small-batch workloads.
**NCCL timeout:** If model loading is slow and you hit an NCCL timeout, increase it: `--dist-timeout 3600`.
**Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [Basic API Usage](../../../docs/get-started/quickstart)
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
DeepSeek-V3 supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
```shell Command
python -m sglang.launch_server \
--model deepseek-ai/DeepSeek-V3 \
--reasoning-parser deepseek-v3 \
--tp 8
```
**Streaming with Thinking Process:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
extra_body = {"chat_template_kwargs": {"thinking": True}},
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
To determine 15% of a number, follow these steps:
**Step 1: Understand the Problem**
You need to find 15% of a given number. Let's assume the number is 240 for this example.
**Step 2: Convert the Percentage to a Decimal**
To work with percentages in calculations, convert the percentage to its decimal form. To do this, divide the percentage by 100.
\[ 15\% = \frac{15}{100} = 0.15 \]
**Step 3: Multiply the Decimal by the Number**
Now, multiply the decimal form of the percentage by the number you want to find the percentage of.
\[ 0.15 \times 240 \]
**Step 4: Perform the Multiplication**
Calculate the product:
\[ 0.15 \times 240 = 36 \]
**Step 5: Conclusion**
Therefore, 15% of 240 is:
\boxed{36}
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
```
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
#### 4.2.2 Tool Calling
DeepSeek-V3 supports tool calling capabilities. Enable the tool call parser:
**Deployment Command:**
```shell Command
python -m sglang.launch_server \
--model deepseek-ai/DeepSeek-V3 \
--tool-call-parser deepseekv3 \
--reasoning-parser deepseek-v3 \
--chat-template ./examples/chat_template/tool_chat_template_deepseekv3.jinja \
--tp 8 \
--host 0.0.0.0 \
--port 30000
```
**Quick Test (curl):**
```shell Command
curl "http://127.0.0.1:30000/v1/chat/completions" \
-H "Content-Type: application/json" \
-d '{
"temperature": 0,
"max_tokens": 100,
"model": "deepseek-ai/DeepSeek-V3",
"tools": [{"type": "function", "function": {"name": "query_weather", "description": "Get weather of a city", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}],
"messages": [{"role": "user", "content": "How'\''s the weather in Beijing today?"}]
}'
```
<Note>
Use a low `temperature` (e.g. `0`) for more consistent tool call results. The `--chat-template` flag above provides an improved unified prompt for tool use.
</Note>
**Python Example (with Thinking Process):**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
extra_body = {"chat_template_kwargs": {"thinking": True}},
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Accumulate tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================\n", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {
'name': None,
'arguments': ''
}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]['name'] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
# Print content
if delta.content:
print(delta.content, end="", flush=True)
# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"🔧 Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
print()
```
**Output Example:**
```text Output
🔧 Tool Call: get_weather
Arguments: {"location": "Beijing", "unit": "celsius"}
```
**Note:**
- The reasoning parser shows how the model decides to use a tool
- Tool calls are clearly marked with the function name and arguments
- You can then execute the function and send the result back to continue the conversation
**Handling Tool Call Results:**
Please attach the code blocks below to the previous Python script.
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```
#### 4.2.3 Multi-Token Prediction (EAGLE Speculative Decoding)
SGLang implements DeepSeek V3 Multi-Token Prediction (MTP) based on [EAGLE speculative decoding](../../../docs/advanced_features/speculative_decoding#eagle-decoding). With this optimization, decoding speed improves by up to **1.8×** at batch size 1 and **1.5×** at batch size 32 on H200 TP8.
**Enable with:**
```shell Command
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3-0324 \
--speculative-algorithm EAGLE \
--trust-remote-code \
--tp 8
```
The default configuration is `--speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. Find the best values for your workload with [bench_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py). The minimum viable config is `--speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2`.
<Note>
For large batch sizes (>48), increase `--max-running-requests` beyond the default of 48 for MTP. Also set `--cuda-graph-bs` to include your target batch sizes (default captured sizes for speculative decoding: 48).
</Note>
<Tip>
The spec-v2 overlap scheduler is enabled by default. It improves performance by overlapping draft and verification stages. Pass `--disable-overlap-schedule` to disable.
</Tip>
#### 4.2.4 MLA Optimizations
DeepSeek V3 uses [Multi-head Latent Attention (MLA)](https://arxiv.org/pdf/2405.04434), an attention mechanism that improves inference efficiency. SGLang implements several optimizations:
- **Weight Absorption:** Reorders matrix multiplications to improve decoding phase efficiency.
- **MLA Attention Backends:** FA3, Flashinfer, FlashMLA, CutlassMLA, TRTLLM MLA (Blackwell), and Triton. FA3 is the default.
- **FP8 Quantization:** W8A8 FP8 and KV Cache FP8, with BMM operators for weight-absorbed MLA in FP8.
- **CUDA Graph & Torch.compile:** Both MLA and MoE support CUDA Graph and Torch.compile for reduced decoding latency.
- **Chunked Prefix Cache:** Increases throughput for long-sequence chunked prefill (FlashAttention3 backend only).
Overall, these optimizations achieve up to **7×** output throughput improvement vs. the baseline.
**Reference:** See [SGLang v0.3 blog](https://lmsys.org/blog/2024-09-04-sglang-v0-3/#deepseek-multi-head-latent-attention-mla-throughput-optimizations) and [Slides](https://github.com/sgl-project/sgl-learning-materials/blob/main/slides/lmsys_1st_meetup_deepseek_mla.pdf) for details.
#### 4.2.5 Multi-Node Deployment
For multi-node serving and hardware-specific examples:
- [8× H200 / 4–8× B200](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#using-docker-recommended)
- [8× MI300X](../../../docs/hardware-platforms/amd_gpu#running-deepseek-v3)
- [2×8× H200 with Docker](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-two-h2008-nodes-and-docker)
- [4×8× A100](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-four-a1008-nodes)
- [8× A100 AWQ](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-8-a100a800-with-awq-quantization)
- [16× A100 INT8](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-16-a100a800-with-int8-quantization)
- [32× L40S INT8](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-32-l40s-with-int8-quantization)
- [Xeon 6980P CPU](../../../docs/hardware-platforms/cpu_server#example-running-deepseek-v3-1-terminus)
- [4× Atlas 800I A3 (int8)](../../../docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1#multi-node-pd-disaggregation-deployment)
**Blog references for large-scale deployment:**
- [Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP](https://lmsys.org/blog/2025-06-16-gb200-part-1/) ([Part I](https://lmsys.org/blog/2025-06-16-gb200-part-1/), [Part II](https://lmsys.org/blog/2025-09-25-gb200-part-2/))
- [PD Disaggregation and Large-Scale Expert Parallelism on 96× H100](https://lmsys.org/blog/2025-05-05-large-scale-ep/)
- [Best Practices for Serving DeepSeek-R1 on H20](https://lmsys.org/blog/2025-09-26-sglang-ant-group/)
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: AMD MI300X GPU (8x)
- Model: DeepSeek-V3
- Tensor Parallelism: 8
- sglang version: 0.5.7
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses.
#### 5.1.1 Latency-Sensitive Benchmark
- Model Deployment Command:
```shell Command
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3 \
--tp 8 \
--dp 8 \
--enable-dp-attention \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--host 0.0.0.0 \
--port 8000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 8000 \
--model deepseek-ai/DeepSeek-V3 \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 10 \
--max-concurrency 1
```
- **Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 81.27
Total input tokens: 1972
Total input text tokens: 1972
Total input vision tokens: 0
Total generated tokens: 2784
Total generated tokens (retokenized): 2774
Request throughput (req/s): 0.12
Input token throughput (tok/s): 24.27
Output token throughput (tok/s): 34.26
Peak output token throughput (tok/s): 65.00
Peak concurrent requests: 2
Total token throughput (tok/s): 58.52
Concurrency: 1.00
Accept length: 2.61
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 8123.17
Median E2E Latency (ms): 7982.65
---------------Time to First Token----------------
Mean TTFT (ms): 1080.76
Median TTFT (ms): 1248.82
P99 TTFT (ms): 1896.37
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 25.04
Median TPOT (ms): 24.76
P99 TPOT (ms): 32.09
---------------Inter-Token Latency----------------
Mean ITL (ms): 25.41
Median ITL (ms): 20.14
P95 ITL (ms): 60.28
P99 ITL (ms): 60.99
Max ITL (ms): 61.49
==================================================
```
#### 5.1.2 Throughput-Sensitive Benchmark
- Model Deployment Command:
```shell Command
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3 \
--tp 8 \
--ep 8 \
--dp 8 \
--enable-dp-attention \
--host 0.0.0.0 \
--port 8000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 8000 \
--model deepseek-ai/DeepSeek-V3 \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 1000 \
--max-concurrency 100
```
- **Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 406.16
Total input tokens: 301701
Total input text tokens: 301701
Total input vision tokens: 0
Total generated tokens: 188375
Total generated tokens (retokenized): 187542
Request throughput (req/s): 2.46
Input token throughput (tok/s): 742.81
Output token throughput (tok/s): 463.80
Peak output token throughput (tok/s): 1299.00
Peak concurrent requests: 109
Total token throughput (tok/s): 1206.61
Concurrency: 87.53
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 35552.98
Median E2E Latency (ms): 21466.07
---------------Time to First Token----------------
Mean TTFT (ms): 1521.51
Median TTFT (ms): 476.80
P99 TTFT (ms): 8329.50
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 214.73
Median TPOT (ms): 152.00
P99 TPOT (ms): 1155.85
---------------Inter-Token Latency----------------
Mean ITL (ms): 182.10
Median ITL (ms): 79.18
P95 ITL (ms): 398.60
P99 ITL (ms): 1488.96
Max ITL (ms): 43465.60
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 GSM8K Benchmark
- **Benchmark Command:**
```shell Command
python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000
```
- **Test Results**:
- DeepSeek-V3
```text Output
Accuracy: 0.960
Invalid: 0.000
Latency: 32.450 s
Output throughput: 614.211 token/s
```
#### 5.2.2 MMLU Benchmark
- **Benchmark Command:**
```shell Command
cd sglang
bash benchmark/mmlu/download_data.sh
python3 benchmark/mmlu/bench_sglang.py --nsub 10 --port 8000
```
- **Test Results**:
- DeepSeek-V3
```text Output
subject: abstract_algebra, #q:100, acc: 0.800
subject: anatomy, #q:135, acc: 0.874
subject: astronomy, #q:152, acc: 0.928
subject: business_ethics, #q:100, acc: 0.880
subject: clinical_knowledge, #q:265, acc: 0.928
subject: college_biology, #q:144, acc: 0.965
subject: college_chemistry, #q:100, acc: 0.670
subject: college_computer_science, #q:100, acc: 0.840
subject: college_mathematics, #q:100, acc: 0.800
subject: college_medicine, #q:173, acc: 0.861
Total latency: 58.339
Average accuracy: 0.871
```
docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3_1.mdx
+989
View File
@@ -0,0 +1,989 @@
---
title: DeepSeek-V3.1
metatags:
description: "Deploy DeepSeek-V3.1 MoE model with SGLang - hybrid reasoning, improved tool calling, and agentic behavior for complex multi-step tasks."
---
## 1. Model Introduction
[DeepSeek V3.1](https://huggingface.co/deepseek-ai/DeepSeek-V3.1) is an advanced Mixture-of-Experts (MoE) large language model developed by DeepSeek, representing a major capability and usability upgrade over DeepSeek V3. As a refined iteration in the DeepSeek V3 family, DeepSeek V3.1 introduces a hybrid reasoning paradigm that supports both fast non-thinking responses and explicit multi-step reasoning, alongside significantly improved tool calling and agentic behavior. The model demonstrates strong performance across reasoning, mathematics, coding, long-context understanding, and real-world agent workflows, benefiting from continued training, alignment optimization, and inference-time refinements. DeepSeek V3.1 is designed to serve as a robust general-purpose foundation model, well suited for conversational AI, structured tool invocation, search-augmented generation, and complex multi-step tasks, while maintaining high efficiency through its sparse MoE architecture.
**[DeepSeek-V3.1-Terminus](https://huggingface.co/deepseek-ai/DeepSeek-V3.1-Terminus)** is an experimental version designed for general conversations and long-context processing. It features hybrid thinking capabilities, allowing you to toggle between "Think" mode for deliberate reasoning and "Non-Think" mode for faster responses. Recommended for general conversations, long-context processing, and experimental use cases.
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
import { DeepSeekV31Deployment } from "/src/snippets/autoregressive/deepseek-v31-deployment.jsx";
<DeepSeekV31Deployment />
### 3.2 Configuration Tips
DeepSeek-V3.1 shares the same model architecture as DeepSeek-V3, so the same hardware and optimization recommendations apply.
**Recommended GPU configurations by weight type:**
<table style={{width: "100%", borderCollapse: "collapse"}}>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>Weight Type</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Supported Hardware</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>FP8</strong> (recommended)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8× H200, 8× B200, 8× MI300X, 2×8× H100/H800/H20</td>
</tr>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>BF16</strong> (upcast from FP8)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>2×8× H200, 2×8× MI300X, 4×8× H100/H800, 4×8× A100/A800</td>
</tr>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>INT8</strong></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Atlas 800I A3</td>
</tr>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>W4A8 / AWQ / MXFP4 / NVFP4</strong></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8× H20/H100, 4× H200; 8× H100/A100; 8/4× MI355X/MI350X; 8/4× B200</td>
</tr>
</tbody>
</table>
> The official DeepSeek-V3.1 checkpoint is already in FP8 format — do **not** add `--quantization fp8` when serving it.
**DeepGEMM precompilation (NVIDIA Hopper / Blackwell):** Precompile GEMM kernels before the first server run to avoid JIT overhead (~10 min):
```bash
python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-V3.1 --tp 8 --trust-remote-code
```
DeepGEMM is enabled by default on Hopper/Blackwell and can be disabled with `SGLANG_ENABLE_JIT_DEEPGEMM=0`.
**Data Parallelism Attention (`--enable-dp-attention`):** Recommended for high-throughput scenarios with large batch sizes. Reduces KV-cache duplication across TP ranks. Use `--enable-dp-attention --tp 8 --dp 8` on a single 8-GPU node. Not recommended for low-latency, small-batch workloads.
**NCCL timeout:** If model loading is slow and you hit an NCCL timeout, increase it: `--dist-timeout 3600`.
**Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [Basic API Usage](../../../docs/get-started/quickstart)
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
DeepSeek-V3.1 supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
```shell Command
python -m sglang.launch_server \
--model deepseek-ai/DeepSeek-V3.1-Terminus \
--reasoning-parser deepseek-v3 \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
**Streaming with Thinking Process:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3.1-Terminus",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
extra_body = {"chat_template_kwargs": {"thinking": True}},
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
First, the problem is asking for 15% of 240. Percent means per hundred, so 15% is the same as 15 out of 100, or 15/100.
To find a percentage of a number, I can multiply the number by the percentage expressed as a decimal. So, I need to convert 15% to a decimal. To do that, I divide 15 by 100, which gives me 0.15.
Now, I multiply 0.15 by 240. So, the calculation is 0.15 × 240.
I can compute this step by step. First, I know that 15% of 100 is 15, but since 240 is larger, I need to adjust. Alternatively, I can think of 10% of 240, which is easy because 10% is just 240 divided by 10, which is 24. Then, 5% is half of 10%, so half of 24 is 12. Therefore, 15% is 10% plus 5%, so 24 plus 12, which equals 36.
I should also do the multiplication to confirm. 0.15 × 240. I can break it down: 0.15 × 200 = 30, and 0.15 × 40 = 6, so 30 + 6 = 36. Same answer.
So, 15% of 240 is 36.
The problem says "step by step," so I should present it clearly.
=============== Content =================
To find 15% of 240, follow these steps:
1. Understand that "percent" means "per hundred," so 15% is equivalent to \( \frac{15}{100} \).
2. Convert 15% to a decimal by dividing by 100: \( 15\% = \frac{15}{100} = 0.15 \).
3. Multiply the decimal by 240: \( 0.15 \times 240 \).
4. Perform the multiplication:
- \( 0.15 \times 200 = 30 \)
- \( 0.15 \times 40 = 6 \)
- Add the results: \( 30 + 6 = 36 \).
Alternatively, you can find 15% by breaking it into parts:
- 10% of 240 is \( \frac{10}{100} \times 240 = 0.10 \times 240 = 24 \).
- 5% of 240 is half of 10%, so \( \frac{24}{2} = 12 \).
- Add 10% and 5%: \( 24 + 12 = 36 \).
Thus, 15% of 240 is 36.
```
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
#### 4.2.2 Tool Calling
DeepSeek-V3.1 and DeepSeek-V3.1-Terminus support tool calling capabilities. Enable the tool call parser:
**Deployment Command:**
```shell Command
python -m sglang.launch_server \
--model deepseek-ai/DeepSeek-V3.1-Terminus \
--tool-call-parser deepseekv31 \
--reasoning-parser deepseek-v3 \
--chat-template ./examples/chat_template/tool_chat_template_deepseekv31.jinja \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
For DeepSeek-V3.1, use `--tool-call-parser deepseekv31` as well.
**Python Example (with Thinking Process):**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3.1-Terminus",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
extra_body = {"chat_template_kwargs": {"thinking": True}},
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Accumulate tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================\n", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {
'name': None,
'arguments': ''
}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]['name'] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
# Print content
if delta.content:
print(delta.content, end="", flush=True)
# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"🔧 Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
print()
```
**Output Example:**
```text Output
=============== Thinking =================
Hmm, the user is asking for the weather in Beijing. This is a straightforward request that matches exactly what the weather tool can provide.
I need to call the get_weather function with Beijing as the location parameter. The user didn't specify a temperature unit, so I'll default to Celsius since that's commonly used in most parts of the world.
The tool call format needs to be precise - just the city name and unit selection. Once I get the weather data back, I'll present it clearly to the user.I'll check the weather in Beijing for you.
=============== Content =================
🔧 Tool Call: get_weather
Arguments: {"location": "Beijing", "unit": "celsius"}
```
**Note:**
- The reasoning parser shows how the model decides to use a tool
- Tool calls are clearly marked with the function name and arguments
- You can then execute the function and send the result back to continue the conversation
**Handling Tool Call Results:**
Please attach the code blocks below to the previous Python script.
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V3.1-Terminus",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "Currently, it is **22°C and sunny** in Beijing."
```
#### 4.2.3 Multi-Token Prediction (EAGLE Speculative Decoding)
DeepSeek-V3.1 shares the same architecture as DeepSeek-V3 and supports the same EAGLE-based MTP speculative decoding path. Refer to [DeepSeek-V3 §4.2.3](/cookbook/autoregressive/DeepSeek/DeepSeek-V3#4-2-3-multi-token-prediction-eagle-speculative-decoding) for the full configuration, tuning guidance, and `bench_speculative.py` reference. The `--speculative-num-steps`, `--speculative-eagle-topk`, and `--max-running-requests` recommendations apply equally to V3.1.
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: AMD MI300X GPU (8x)
- Model: DeepSeek-V3.1-Terminus
- Tensor Parallelism: 8
- sglang version: 0.5.7
**Benchmark Methodology:**
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
#### 5.1.1 Standard Test Scenarios
Three core scenarios reflect real-world usage patterns:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
</tr>
</tbody>
</table>
#### 5.1.2 Concurrency Levels
Test each scenario at different concurrency levels to capture the throughput vs. latency trade-off:
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
#### 5.1.3 Number of Prompts
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
---
#### 5.1.4 Benchmark Commands
**Scenario 1: Chat (1K/1K) - Most Important**
- **Model Deployment**
```bash Command
python -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3.1 \
--tp 8
```
- Low Concurrency (Latency-Optimized)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 106.24
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4201
Request throughput (req/s): 0.09
Input token throughput (tok/s): 57.43
Output token throughput (tok/s): 39.72
Peak output token throughput (tok/s): 43.00
Peak concurrent requests: 2
Total token throughput (tok/s): 97.15
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 10620.29
Median E2E Latency (ms): 8868.09
---------------Time to First Token----------------
Mean TTFT (ms): 557.85
Median TTFT (ms): 213.58
P99 TTFT (ms): 1625.28
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 23.84
Median TPOT (ms): 23.90
P99 TPOT (ms): 24.03
---------------Inter-Token Latency----------------
Mean ITL (ms): 23.90
Median ITL (ms): 23.92
P95 ITL (ms): 24.15
P99 ITL (ms): 24.25
Max ITL (ms): 25.44
==================================================
```
- Medium Concurrency (Balanced)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 107.71
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 40805
Total generated tokens (retokenized): 40625
Request throughput (req/s): 0.74
Input token throughput (tok/s): 368.28
Output token throughput (tok/s): 378.84
Peak output token throughput (tok/s): 508.00
Peak concurrent requests: 19
Total token throughput (tok/s): 747.12
Concurrency: 13.72
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 18473.65
Median E2E Latency (ms): 19558.42
---------------Time to First Token----------------
Mean TTFT (ms): 607.91
Median TTFT (ms): 191.32
P99 TTFT (ms): 2135.13
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 35.50
Median TPOT (ms): 35.99
P99 TPOT (ms): 43.62
---------------Inter-Token Latency----------------
Mean ITL (ms): 35.10
Median ITL (ms): 32.18
P95 ITL (ms): 33.03
P99 ITL (ms): 159.99
Max ITL (ms): 453.99
==================================================
```
- High Concurrency (Throughput-Optimized)
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 207.65
Total input tokens: 249831
Total input text tokens: 249831
Total input vision tokens: 0
Total generated tokens: 252662
Total generated tokens (retokenized): 251238
Request throughput (req/s): 2.41
Input token throughput (tok/s): 1203.15
Output token throughput (tok/s): 1216.79
Peak output token throughput (tok/s): 2100.00
Peak concurrent requests: 106
Total token throughput (tok/s): 2419.94
Concurrency: 91.02
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 37800.20
Median E2E Latency (ms): 35921.56
---------------Time to First Token----------------
Mean TTFT (ms): 835.15
Median TTFT (ms): 236.88
P99 TTFT (ms): 2868.52
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 73.33
Median TPOT (ms): 76.35
P99 TPOT (ms): 97.63
---------------Inter-Token Latency----------------
Mean ITL (ms): 73.30
Median ITL (ms): 50.82
P95 ITL (ms): 180.67
P99 ITL (ms): 186.83
Max ITL (ms): 1661.39
==================================================
```
**Scenario 2: Reasoning (1K/8K)**
- Low Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 1097.29
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 44462
Total generated tokens (retokenized): 44313
Request throughput (req/s): 0.01
Input token throughput (tok/s): 5.56
Output token throughput (tok/s): 40.52
Peak output token throughput (tok/s): 43.00
Peak concurrent requests: 2
Total token throughput (tok/s): 46.08
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 109725.52
Median E2E Latency (ms): 117748.67
---------------Time to First Token----------------
Mean TTFT (ms): 156.67
Median TTFT (ms): 156.19
P99 TTFT (ms): 159.87
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 24.41
Median TPOT (ms): 24.51
P99 TPOT (ms): 24.96
---------------Inter-Token Latency----------------
Mean ITL (ms): 24.65
Median ITL (ms): 24.58
P95 ITL (ms): 25.68
P99 ITL (ms): 25.93
Max ITL (ms): 29.80
==================================================
```
- Medium Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 775.02
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 318306
Total generated tokens (retokenized): 317426
Request throughput (req/s): 0.10
Input token throughput (tok/s): 51.18
Output token throughput (tok/s): 410.70
Peak output token throughput (tok/s): 512.00
Peak concurrent requests: 18
Total token throughput (tok/s): 461.89
Concurrency: 13.86
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 134236.65
Median E2E Latency (ms): 135181.28
---------------Time to First Token----------------
Mean TTFT (ms): 214.35
Median TTFT (ms): 194.12
P99 TTFT (ms): 300.27
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 33.72
Median TPOT (ms): 34.00
P99 TPOT (ms): 34.75
---------------Inter-Token Latency----------------
Mean ITL (ms): 33.69
Median ITL (ms): 33.71
P95 ITL (ms): 34.50
P99 ITL (ms): 34.92
Max ITL (ms): 164.76
==================================================
```
- High Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 320 \
--max-concurrency 64 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 1231.97
Total input tokens: 158939
Total input text tokens: 158939
Total input vision tokens: 0
Total generated tokens: 1301025
Total generated tokens (retokenized): 1296845
Request throughput (req/s): 0.26
Input token throughput (tok/s): 129.01
Output token throughput (tok/s): 1056.05
Peak output token throughput (tok/s): 1472.00
Peak concurrent requests: 67
Total token throughput (tok/s): 1185.07
Concurrency: 56.17
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 216256.25
Median E2E Latency (ms): 224192.84
---------------Time to First Token----------------
Mean TTFT (ms): 317.68
Median TTFT (ms): 235.28
P99 TTFT (ms): 649.39
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 53.30
Median TPOT (ms): 55.10
P99 TPOT (ms): 56.58
---------------Inter-Token Latency----------------
Mean ITL (ms): 53.13
Median ITL (ms): 52.95
P95 ITL (ms): 56.23
P99 ITL (ms): 181.04
Max ITL (ms): 208.61
==================================================
```
**Scenario 3: Summarization (8K/1K)**
- Low Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 114.47
Total input tokens: 41941
Total input text tokens: 41941
Total input vision tokens: 0
Total generated tokens: 4220
Total generated tokens (retokenized): 4194
Request throughput (req/s): 0.09
Input token throughput (tok/s): 366.39
Output token throughput (tok/s): 36.87
Peak output token throughput (tok/s): 42.00
Peak concurrent requests: 2
Total token throughput (tok/s): 403.26
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 11442.86
Median E2E Latency (ms): 9508.87
---------------Time to First Token----------------
Mean TTFT (ms): 883.78
Median TTFT (ms): 481.38
P99 TTFT (ms): 2217.45
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 24.93
Median TPOT (ms): 25.05
P99 TPOT (ms): 26.11
---------------Inter-Token Latency----------------
Mean ITL (ms): 25.08
Median ITL (ms): 25.08
P95 ITL (ms): 26.18
P99 ITL (ms): 26.28
Max ITL (ms): 27.41
==================================================
```
- Medium Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 162.33
Total input tokens: 300020
Total input text tokens: 300020
Total input vision tokens: 0
Total generated tokens: 41669
Total generated tokens (retokenized): 41443
Request throughput (req/s): 0.49
Input token throughput (tok/s): 1848.27
Output token throughput (tok/s): 256.70
Peak output token throughput (tok/s): 467.00
Peak concurrent requests: 19
Total token throughput (tok/s): 2104.97
Concurrency: 14.52
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 29456.89
Median E2E Latency (ms): 27628.16
---------------Time to First Token----------------
Mean TTFT (ms): 1784.30
Median TTFT (ms): 1347.21
P99 TTFT (ms): 5384.54
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 53.65
Median TPOT (ms): 52.09
P99 TPOT (ms): 74.39
---------------Inter-Token Latency----------------
Mean ITL (ms): 53.23
Median ITL (ms): 34.52
P95 ITL (ms): 35.81
P99 ITL (ms): 513.25
Max ITL (ms): 2865.73
==================================================
```
- High Concurrency
```bash Command
python -m sglang.bench_serving \
--backend sglang \
--model deepseek-ai/DeepSeek-V3.1 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 320 \
--max-concurrency 64 \
--request-rate inf
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 282.55
Total input tokens: 1273893
Total input text tokens: 1273893
Total input vision tokens: 0
Total generated tokens: 170000
Total generated tokens (retokenized): 169081
Request throughput (req/s): 1.13
Input token throughput (tok/s): 4508.6
Output token throughput (tok/s): 601.67
Peak output token throughput (tok/s): 1216
Peak concurrent requests: 68
Total token throughput (tok/s): 5110.27
Concurrency: 59.81
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 52810.32
Median E2E Latency (ms): 50981.81
---------------Time to First Token----------------
Mean TTFT (ms): 786.69
Median TTFT (ms): 499.38
P99 TTFT (ms): 2925.98
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 97.93
Median TPOT (ms): 103.45
P99 TPOT (ms): 157.84
---------------Inter-Token Latency----------------
Mean ITL (ms): 98.11
Median ITL (ms): 55.7
P95 ITL (ms): 240.71
P99 ITL (ms): 1114.36
==================================================
```
#### 5.1.5 Understanding the Results
**Key Metrics:**
- **Request Throughput (req/s)**: Number of requests processed per second
- **Output Token Throughput (tok/s)**: Total tokens generated per second
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
**Why These Configurations Matter:**
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
- **Variable Concurrency**: Captures the Pareto frontier - the optimal trade-off between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
**Interpreting Results:**
- Compare your results against baseline numbers for your hardware
- Higher throughput at same latency = better performance
- Lower TTFT = more responsive user experience
- Lower TPOT = faster generation speed
### 5.2 Accuracy Benchmark
Document model accuracy on standard benchmarks:
#### 5.2.1 GSM8K Benchmark
- Benchmark Command
```bash Command
python3 benchmark/gsm8k/bench_sglang.py \
--num-shots 8 \
--num-questions 1316 \
--parallel 1316
```
**Test Results:**
```text Output
Accuracy: 0.959
Invalid: 0.000
Latency: 29.185 s
Output throughput: 4854.672 token/s
```
docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2.mdx
+1072
View File
File diff suppressed because it is too large Load Diff
docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx
+593
View File
@@ -0,0 +1,593 @@
---
title: DeepSeek-V4
description: "Deploy DeepSeek-V4 with SGLang — verified launch commands, benchmarks, and tuning for Flash Official (0731), Flash, and Pro."
tag: NEW
---
## Deployment
<a id="install" />
<Accordion title="Install SGLang">
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
<Tabs>
<Tab title="Python (pip / uv)">
```bash Command
pip install --upgrade pip
pip install uv
uv pip install sglang
```
Then run the **Python** output of the command panel below in that environment.
</Tab>
<Tab title="Docker">
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). A minimal example (substitute the inner `sglang serve ...` with whatever the command generator below produces):
**NVIDIA GPUs**
A single image — `lmsysorg/sglang:latest` — covers the **datacenter GPUs** in this cookbook (B200 / B300 / GB200 / GB300 / H100 / H200 / RTX PRO 6000).
```bash Command
docker pull lmsysorg/sglang:latest
docker run --gpus all \
--shm-size 32g \
-p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=<your-hf-token>" \
--ipc=host \
lmsysorg/sglang:latest \
sglang serve <use args below>
```
**AMD GPUs (ROCm)**
AMD uses the daily-updated `lmsysorg/sglang-rocm` images. You can find the latest images on [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang-rocm/tags). We recommend the ROCm 7.2 version.
For example:
- **MI355X** → `lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710`
- **MI300X** → `lmsysorg/sglang-rocm:v0.5.13.post1-rocm720-mi30x-20260623`
```bash Command
docker pull lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710
docker run \
--device=/dev/kfd --device=/dev/dri \
--group-add video \
--cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
--shm-size 32g --ipc=host \
-p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=<your-hf-token>" \
lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710 \
sglang serve <use args below>
```
</Tab>
</Tabs>
</Accordion>
Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points:
- **Low-Latency** — fastest reply for a single user. Pick for chat.
- **Balanced** — good speed with several users at once. Use for typical multi-user serving.
- **High-Throughput** — most tokens per second across many users. Best for batch jobs.
import { Deployment } from "/src/snippets/_deployment.jsx";
import { config } from "/src/snippets/configs/deepseek-ai/deepseek-v4.jsx";
import { benchmarks } from "/src/snippets/configs/deepseek-ai/deepseek-v4-benchmarks.jsx";
<Deployment config={config} benchmarks={benchmarks} />
<Note>
For a runnable end-to-end example, see the [DeepSeek-V4-Flash demo notebook](https://github.com/sgl-project/sglang/blob/main/docs/demo/deepseek_v4_flash.ipynb).
</Note>
<div style={{fontSize: "0.85em", lineHeight: "1.55", color: "#6b7280", margin: "0.5rem 0 1rem 0"}}>
<p style={{margin: "0 0 0.3rem 0"}}><strong>Panel controls</strong> (top of the command box):</p>
<ul style={{margin: 0, paddingLeft: "1.25rem"}}>
<li style={{marginBottom: "0.2rem"}}><strong>Python / Docker</strong> — bare <code>sglang serve …</code> for an existing SGLang env, or a <code>docker run … sglang serve …</code> wrap against the per-hardware image from the <a href="#install">Install SGLang</a> panel above.</li>
<li style={{marginBottom: "0.2rem"}}><strong>⧉ Copy</strong> — copies the current command (with whichever framing is active) to your clipboard.</li>
<li style={{marginBottom: "0.2rem"}}><strong>$ cURL</strong> — a sample request against <code>localhost:30000</code> to confirm the server is up.</li>
<li style={{marginBottom: "0.2rem"}}><strong>⚙ Env</strong> — edits the placeholders (<code>HOST_IP</code>, <code>PORT</code>, <code>HF_TOKEN</code>, <code>NODE_RANK</code>, <code>NODE0_IP</code>) the command and cURL share. Persists in localStorage across cookbooks.</li>
<li><strong>Verified / Not Verified</strong> badge — green when the <code>(hw, variant, quant, strategy, nodes)</code> combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.</li>
</ul>
</div>
## Playground
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The base is read live from your Deploy selection — only your overrides change.
The knobs come in two flavors:
- **Built-in SGLang features** — parallelism overrides (TP / CP / DP-Attention — DP-Attention's value is the DP degree, with `off` to disable), MoE backend + EP, reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, HiCache tiers, and HiSparse hierarchical sparse attention (decode-role only — the card appears once PD-Disagg mode is set to decode).
- **DeepSeek-V4 specific features** — MegaMoE W4A8 / W4A4 fused kernel (Blackwell only; Hopper SM90 uses a separate all-FP8 MegaMoE path — see Configuration Tips below).
Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. When no override differs from the base cell, the playground inherits the base's **Verified** badge; any actual change flips it to **Not Verified** until the new configuration is run end-to-end and submitted back.
import { Playground } from "/src/snippets/_playground.jsx";
<Playground config={config} />
<div style={{fontSize: "0.85em", lineHeight: "1.55", color: "#6b7280", margin: "0.5rem 0 1rem 0"}}>
<p style={{margin: "0 0 0.3rem 0"}}><strong>Panel controls</strong> reuse <strong>Python / Docker</strong> · <strong>⧉ Copy</strong> · <strong>$ cURL</strong> · <strong>⚙ Env</strong> from the Deploy panel, plus one extra:</p>
<ul style={{margin: 0, paddingLeft: "1.25rem"}}>
<li><strong>Submit ↗</strong> — opens a pre-filled GitHub issue so you can land your override combo as a new verified cookbook cell. Shown only while the badge says <strong>Not Verified</strong>; click it once you've actually run the command on your hardware and confirmed it works.</li>
</ul>
</div>
## 1. Model Introduction
**DeepSeek-V4** is the next-generation Mixture-of-Experts model from DeepSeek, released 2026-04-24 under an **MIT License**. The 0731 Flash refresh adds a checkpoint with a bundled DSpark draft head:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "30%"}} />
<col style={{width: "15%"}} />
<col style={{width: "15%"}} />
<col style={{width: "40%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Variant</th>
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Total params</th>
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Active (MoE)</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash">DeepSeek-V4-Flash</a></strong></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>284B</strong></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>13B</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>single-node serving on B200 / B300 / GB200 / GB300 / H200 (TP=4); RTX PRO 6000 (TP=2); H100 (TP=8)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731">DeepSeek-V4-Flash-0731</a></strong></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>304</strong></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>13B</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Flash Official (0731), with a bundled DSpark draft head; verified on 8×B200, 4×GB300, and 4×H200</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro">DeepSeek-V4-Pro</a></strong></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>1.6T</strong></td>
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>49B</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>high-capacity: B200 / B300 (TP=8) · GB300 (TP=4) · H200 FP4 (TP=8) · GB200 (2-node, TP=8) · H200 FP8 (2-node, TP=16) · H100 (2-node, TP=16)</td>
</tr>
</tbody>
</table>
The Instruct checkpoints ship as **FP4 MoE experts + FP8 attention / dense** (one mixed-precision checkpoint covers every FP4-capable GPU). Matching `*-Base` repos ship pure FP8 mixed and are for further pre-training only — not for chat or tool calling.
**Highlights:** hybrid CSA + HCA attention (~27% inference FLOPs / ~10% KV cache vs DSv3.2 at 1M context), manifold-constrained hyper-connections (mHC), Muon optimizer, **1M-token context** (32T+ pre-training tokens), three reasoning modes (*Non-think* / *Think High* / *Think Max* — use ≥ 384K context for Think Max), and a dedicated `encoding_dsv4.encode_messages` Python encoder + DSML tool-call grammar.
**Recommended generation:** `temperature=1.0`, `top_p=1.0`.
**Resources:** HuggingFace · [Flash Official (0731)](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) · [Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) · [Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) · ModelScope · [Flash](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Flash) · [Pro](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Pro).
## 2. Configuration Tips
{/* TODO: expand this section as more recipes are validated end-to-end. */}
**Concurrency & DeepEP dispatch buffer**
Must hold: `max-running-requests × MTP_draft_tokens ≤ SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK`. Violating it blows DeepEP's dispatch buffer at steady-state load (`deep_ep.cpp:1105`). When tuning, move `--cuda-graph-max-bs-decode`, `--max-running-requests`, and the env together.
The generator currently picks values on the **conservative** side (mirroring an internal stress-test matrix). They run safely out of the box but likely leave throughput on the table — please tune them up toward your actual workload's peak concurrency and report findings back so the defaults can be revised.
**Speculative decoding**
The original Flash and Pro recipes use EAGLE. Flash Official (0731) uses the bundled DSpark draft head; see [DSpark](#3-4-dspark-speculative-decoding) for its launch and tuning notes.
For the original Flash and Pro checkpoints:
- `low-latency`: steps=3, draft-tokens=4 → largest win at bs=1.
- `balanced`: steps=1, draft-tokens=2 → gentler MTP, reduces throughput hit at higher batch.
- `high-throughput`: MTP disabled — at saturation the verify step costs more than it saves.
- MTP runs on the v2 speculative path.
**Compressed attention state dtype**
DeepSeek-V4 uses hybrid compressed attention for long-context efficiency. `SGLANG_DSV4_COMPRESS_STATE_DTYPE` controls the dtype of the C4 / C128 compressed attention state pools. Supported values are `float32` / `fp32` (default: `float32`) and `bfloat16` / `bf16`. For BF16 on the offline compression path:
```bash Command
SGLANG_DSV4_COMPRESS_STATE_DTYPE=bf16 \
sglang serve \
--model-path deepseek-ai/DeepSeek-V4-Flash \
<other args>
```
This BF16 setting applies only to the compressed attention state pools and reduces the GPU memory footprint of each compressed-state slot. It does not change model weight precision or the main KV cache dtype. With automatic pool sizing and no explicit capacity cap, the same memory budget holds more slots, and the startup log shows larger `c4_state` and `c128_state` pool sizes. Keep the default `float32` setting for the most conservative behavior.
**EPLB + Waterfill (Experimental)**
For recorded/static EPLB reproduction, first record an expert-distribution file by following
[Capture expert selection distribution in MoE models](../../../docs/basic_usage/native_api.mdx#capture-expert-selection-distribution-in-moe-models).
For reproduction runs, use the generated `expert_distribution_recorder_*.pt` as
the initial expert location. **Please checkout to latest main branch for this feature.**
For non-PD reproduction, use:
```bash Command
--moe-a2a-backend deepep \
--deepep-mode auto \
--init-expert-location /path/to/expert_distribution_recorder_*.pt \
--enable-waterfill
```
For PD-Disagg reproduction, use `normal` mode on the prefill server and
`low_latency` mode on the decode server. Add the same `--init-expert-location`
flag to both commands:
```bash Command
# prefill
--moe-a2a-backend deepep \
--deepep-mode normal \
--init-expert-location /path/to/expert_distribution_recorder_*.pt \
--enable-waterfill
# decode
--moe-a2a-backend deepep \
--deepep-mode low_latency \
--init-expert-location /path/to/expert_distribution_recorder_*.pt \
--enable-waterfill
```
You can also add `--ep-num-redundant-experts` and `--eplb-algorithm` to customize
EPLB placement.
Waterfill also supports MegaMOE. Use `--moe-a2a-backend megamoe
--enable-waterfill` to keep the MegaMOE backend while applying Waterfill to the
fused shared expert slot.
**FP4 Indexer (Experimental)**
DeepSeek-V4 uses the default indexer path unless `--enable-deepseek-v4-fp4-indexer` is set. Enable this flag to use the experimental FP4 C4 indexer on SM100 GPUs with DeepGEMM FP4 indexer support. This path is intended for decode-heavy long-context workloads where reducing indexer cache bandwidth is beneficial.
```bash Command
# Please use the latest main branch for this feature.
sglang serve \
--model-path deepseek-ai/DeepSeek-V4-Flash \
--tp 4 \
--moe-runner-backend flashinfer_mxfp4 \
--enable-deepseek-v4-fp4-indexer
```
**NVFP4 Hybrid Checkpoints**
The [`nvidia/DeepSeek-V4-Pro-NVFP4`](https://huggingface.co/nvidia/DeepSeek-V4-Pro-NVFP4) and
[`nvidia/DeepSeek-V4-Flash-NVFP4`](https://huggingface.co/nvidia/DeepSeek-V4-Flash-NVFP4) checkpoints
quantize MoE experts to **NVFP4** while keeping attention and dense layers in
**FP8**. It requires `--moe-runner-backend flashinfer_trtllm_routed` which will be automatically selected if not provided.
```bash Command
sglang serve \
--model-path nvidia/DeepSeek-V4-Pro-NVFP4 \
--tp 8
```
or
```bash Command
sglang serve \
--model-path nvidia/DeepSeek-V4-Flash-NVFP4 \
--tp 8
```
Requires Blackwell (SM100+). The MTP layer in this checkpoint stays
MXFP4-packed and is routed through the `Mxfp4FlashinferTrtllmMoEMethod` path
automatically.
<a id="hopper-note" />
**Hopper (H100 / H200) note**
Two options are available for running DeepSeek-V4 on Hopper:
- **Original FP4 checkpoints** — apply the W4A16 MoE kernels (Marlin) as the command generator picks for Hopper cells. This path works on both H100 and H200 and is the only option for H100 (no FP8 path). It is TP-only; on H200 the Pro variant fits on a single 8-GPU node, while H100 Pro needs 2 nodes (TP=16).
- **Converted FP8 checkpoints** (H100 and H200 only) — pre-repackaged FP8 weights at [`sgl-project/DeepSeek-V4-Flash-FP8`](https://huggingface.co/sgl-project/DeepSeek-V4-Flash-FP8) and [`sgl-project/DeepSeek-V4-Pro-FP8`](https://huggingface.co/sgl-project/DeepSeek-V4-Pro-FP8) unlock DP-attention + DeepEP and richer parallelism (e.g. Pro TP=16 across 2 nodes).
On these FP8 checkpoints you can additionally enable the all-FP8 **MegaMoE** path on SM90 for higher long-context / large-decode throughput — see the **SM90 (Hopper) FP8 MegaMoE** note in Configuration Tips below.
PD-Disagg recipes on H200 may require `docker run --privileged --ulimit memlock=-1`
(or `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK`) so mooncake
can discover the IB HCAs; without IB exposure mooncake silently falls back to
TCP, which can lead to garbled KV transfer on large checkpoints.
**RTX PRO 6000 (SM120 / Blackwell Desktop) note**
RTX PRO 6000 (96 GB) runs **Flash only** with the FlashInfer MXFP4 MoE runner.
V4-Pro doesn't fit on 8× 96 GB; the Deploy panel greys out unsupported recipes.
HiCache and MegaMoE are **not** supported on RTX PRO 6000.
**AMD (MI300X / MI355X) note**
- **Model checkpoints** — for correct accuracy, the FP4 model uses the stock `deepseek-ai/DeepSeek-V4-{Flash,Pro}`, and the FP8 model uses the repackaged `sgl-project/DeepSeek-V4-{Flash,Pro}-FP8`.
- **Supported models** — **MI300X** supports DeepSeek-V4-Flash in FP8; **MI355X** supports DeepSeek-V4-Flash / Pro in both FP4 and FP8. All recipes run single-node.
- **TP / DP setting** — both TP=4 and TP=8 are supported. At low concurrency we recommend **TP-only**; at high concurrency use **TP + DP**, which additionally needs `--dp 8 --enable-dp-attention --enable-prefill-delayer --prefill-delayer-max-delay-ms 5000`.
- **MTP** — speculative decoding is supported; add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`.
- **Kernels** — uses the Unified KV attention and the flydsl MoE.
**MegaMoE**
MegaMoE fuses expert dispatch + GEMM into a single kernel for higher throughput
on MoE layers. To enable it, use the **MegaMoE** chip in the Playground
below — the playground will swap `--moe-a2a-backend deepep` for
`--moe-a2a-backend megamoe` and add the relevant env vars automatically.
Two variants are exposed:
- **W4A8** — default MegaMoE kernel (FP4 weights, FP8 activations).
- **W4A4** — adds `SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1` and
`SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1` to run the custom W4A4
kernel (FP4 activations). Higher throughput with negligible accuracy drop
(~89.5 GPQA on Pro).
Notes:
- The W4A8 / W4A4 variants above are **Blackwell-only** (B200 / B300 / GB200 / GB300). On **Hopper (SM90, H100 / H200)** use the all-FP8 MegaMoE path described below instead.
- MegaMoE is **only wired into the `high-throughput` recipe** on Blackwell (per [sgl-project/sglang#26451](https://github.com/sgl-project/sglang/pull/26451)). The chip is hidden on `low-latency` and `balanced` — switch to `high-throughput` to expose it.
- When running MegaMoE, don't set `--moe-runner-backend` manually.
- Adjust `SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK` based on your workload and memory usage. Setting higher number of tokens for MegaMoE requires more HBM space (recommended: 8320 for high-throughput).
**SM90 (Hopper) FP8 MegaMoE (Experimental)**
On SM90 (Hopper, H100 / H200), the all-FP8 MegaMoE path routes MoE through the
DeepGEMM `mega_moe` runner for higher long-context / large-decode throughput on
the FP8 checkpoints. Unlike the Blackwell W4A8 / W4A4 variants above, experts
stay in **FP8** — keep `SGLANG_DSV4_FP4_EXPERTS=0`. It requires a `sgl-deep-gemm`
build with SM90 FP8 MegaMoE support. **Please use the latest image for this
feature.**
Enable the MegaMoE path with `--moe-a2a-backend megamoe` — or equivalently set
`SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1`, which auto-configures the same backend:
```bash Command
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096 \
SGLANG_DSV4_FP4_EXPERTS=0 \
sglang serve \
--model-path sgl-project/DeepSeek-V4-Flash-FP8 \
--tp 8 \
--moe-a2a-backend megamoe \
--chunked-prefill-size 4096
```
`SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK` caps the number of tokens
the MegaMoE path processes per rank (i.e. per GPU); the MegaMoE path is only used
for batches at or below this cap. The right value depends on your parallelism /
token-split scheme, and larger values reserve more HBM.
**GB300 PD-Disagg cross-pod MNNVL**
On some GB300 clusters with cross-pod KV transfer over NVLink, mooncake may
fail with `nvlink_transport.cpp:497 Requested address ... not found!`. If
this happens, prepend `MC_FORCE_MNNVL=1 NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1`
to both prefill and decode `sglang serve` commands.
## 3. Advanced Usage
### 3.1 Reasoning
Enable the `deepseek-v4` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer into `reasoning_content` vs `content`.
<Accordion title="Streaming with Thinking Process (Python)">
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V4-Flash",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
max_tokens=2048,
extra_body={"chat_template_kwargs": {"thinking": True}},
stream=True,
)
thinking_started = False
has_thinking = False
has_answer = False
for chunk in response:
if not chunk.choices:
continue
delta = chunk.choices[0].delta
if getattr(delta, "reasoning_content", None):
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
if delta.content:
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
</Accordion>
<Accordion title="Example Output">
```text Output
We are asked: "What is 15% of 240?" This is a simple percentage problem. I need to provide a step-by-step solution. The user wants the solution explained step by step. I'll calculate 15% of 240: 0.15 * 240 = 36. I'll break it down into steps: understand what percent means, convert percentage to decimal or fraction, then multiply. I'll present the answer clearly.</think>To find 15% of 240, follow these steps:
**Step 1: Understand the meaning of percent**
"Percent" means "per hundred," so 15% means 15 out of every100, or \( \frac{15}{100} \).
**Step2: Convert the percentage to a decimal or fraction**
\( 15\% = \frac{15}{100} = 0.15 \)
**Step3: Multiply by the given number**
Multiply the decimal form by 240:
\( 0.15 \times 240 \)
**Step4: Perform the multiplication**
\( 0.15 \times 240 = 36 \)
**Answer:** 15% of 240 is **36**.
```
</Accordion>
### 3.2 Tool Calling
Enable the `deepseekv4` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`.
<Accordion title="Python Example with Thinking Process">
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string", "description": "The city name"},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
},
"required": ["location"],
},
},
}
]
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-V4-Flash",
messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
tools=tools,
extra_body={"chat_template_kwargs": {"thinking": True}},
stream=True,
)
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}
for chunk in response:
if not chunk.choices:
continue
delta = chunk.choices[0].delta
if getattr(delta, "reasoning_content", None):
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
if getattr(delta, "tool_calls", None):
if has_thinking and thinking_started:
print("\n=============== Content =================\n", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {"name": None, "arguments": ""}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]["name"] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]["arguments"] += tool_call.function.arguments
if delta.content:
print(delta.content, end="", flush=True)
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
print()
```
</Accordion>
<Accordion title="Example Output">
```text Output
The user wants to know the weather in Beijing. I'll use the get_weather function with Beijing as the location. I don't need to specify a unit, so I'll just use the default.</think>
<｜DSML｜tool_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="location" string="true">Beijing</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜tool_calls>
```
</Accordion>
### 3.3 HiCache (Hierarchical KV Caching)
HiCache enables multi-tier KV cache offloading (GPU → CPU → Storage), significantly expanding effective context capacity for long-context and multi-turn scenarios. Combined with UnifiedRadixTree, it provides intelligent prefix caching across all tiers.
To enable HiCache, open the **HiCache** card in the [Playground above](#playground) and flip **Enable**:
- **L2 (GPU + CPU)** — leave Storage on `auto` (default). Cold KV pages spill to CPU pinned memory only.
- **L3 (GPU + CPU + Storage)** — pick a Storage backend (`file` / `mooncake` / `hf3fs` / `nixl`); the Playground emits the canonical `page_first_direct` mem-layout + `direct` IO backend + `wait_complete` prefetch policy, matching the [HiCache best-practices recipe](../../../docs/advanced_features/hicache_best_practices).
For AMD devices,
- **L2 (GPU + CPU)** — leave Storage on `auto` (default). Cold KV pages spill to CPU pinned memory only. Use `direct` IO backend + `page_first_direct` or `layer-first` mem-layout.
- **L3 (GPU + CPU + Storage)** — pick a Storage backend (`file`); the Playground emits the canonical `page_first_direct` mem-layout + `direct` IO backend + `wait_complete` prefetch policy, matching the [HiCache best-practices recipe](../../../docs/advanced_features/hicache_best_practices).
The Write policy knob defaults to `write_through` (the upstream default); switch to `write_back` / `write_through_selective` to trade durability for write speed when the storage tier is slow.
For more details, see the [HiCache documentation](../../../docs/advanced_features/hicache).
### 3.4 DSpark (Speculative Decoding)
Flash Official (0731) bundles a DSpark draft head in `deepseek-ai/DeepSeek-V4-Flash-0731`. The target and draft weights therefore come from the same checkpoint: enable DSpark with `--speculative-algorithm DSPARK` and do not set a separate `--speculative-draft-model-path`.
Unlike the EAGLE recipes for the original Flash and Pro checkpoints, this recipe omits `--speculative-num-steps`, `--speculative-eagle-topk`, and `--speculative-num-draft-tokens`. SGLang reads the DSpark shape from the checkpoint.
The verified 4×GB300 FP4 low-latency command is:
```bash Command
sglang serve \
--trust-remote-code \
--model-path deepseek-ai/DeepSeek-V4-Flash-0731 \
--tp 4 \
--moe-runner-backend flashinfer_mxfp4 \
--speculative-algorithm DSPARK \
--mem-fraction-static 0.90 \
--chunked-prefill-size 4096 \
--swa-full-tokens-ratio 0.1 \
--host 0.0.0.0 \
--port 30000
```
Keep `--mem-fraction-static 0.90` on this topology to leave enough headroom for the batch-256 verify graph. The first cold start can take 10–15 minutes while FlashInfer autotunes and SGLang captures the draft and verify graphs; later starts reuse the cache. This path is verified end-to-end on 4×GB300 with SGLang v0.5.16.
**Tune proposed draft tokens.** `--speculative-dspark-block-size N` asks DSpark to propose `N` tokens per step; the target verifies a window of `N + 1`. If the flag is omitted, SGLang reads the value from the checkpoint. The current 0731 checkpoint resolves to five proposed tokens, which is the verified default. Use the **DSpark Proposed Draft Tokens** slider in the [Playground](#playground) to sweep one through five.
Larger blocks can improve decode latency when acceptance stays high, but they also increase verification work and graph memory. Start from the checkpoint default, then sweep downward under the real prompt-length and concurrency distribution. The gain is usually largest for short interactive traffic and narrows as prefill dominates. Track P50/P99 TTFT and TPOT, total throughput, accepted length, GPU memory, and stop rate rather than choosing from acceptance alone.
For every candidate, compare with the same recipe without `--speculative-algorithm DSPARK`. Restart the server between the DSpark and non-speculative legs, keep the request corpus, sampling, concurrency, and warmup identical, and give each `bench_serving` leg its own `--flush-cache`. Leave `--speculative-draft-attention-backend` unset unless a separate profiling run justifies an override.
DSpark currently requires CUDA, `pp_size == 1`, and DP Attention disabled. It is not compatible with PD disaggregation on current SGLang releases; turn DSpark off before selecting a prefill or decode role. The DP-Attention and MI355X Flash Official recipes therefore run target-only. If a larger draft block or concurrency causes graph-capture OOM, lower `--mem-fraction-static`, the draft block size, or the configured maximum running requests, then rerun both performance and accuracy gates.