minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[Docs] Rename docs_new/ to docs/ (#32123)

Browse Source 
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
zijiexia
2026-08-03 16:51:00 -07:00
committed by GitHub
co-authored by Claude Opus 4.8
parent c949e91f18
commit b819d2fb5b
491 changed files with 122 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/Qwen/Qwen2.5-VL.mdx
+395
View File
@@ -0,0 +1,395 @@
---
title: Qwen2.5-VL
metatags:
description: "Deploy Qwen2.5-VL vision-language models with SGLang on AMD MI300X - available in 3B to 72B sizes with enhanced visual understanding."
---
import { Qwen25VLDeployment } from '/src/snippets/autoregressive/qwen25-vl-deployment.jsx';
## 1. Model Introduction
**[Qwen2.5-VL](https://huggingface.co/collections/Qwen/qwen25-vl)** is a vision-language model series from the Qwen team, offering significant improvements over its predecessor in understanding, reasoning, and multi-modal processing.
**Key Features:**
- **Understand things visually**: Proficient in recognizing common objects such as flowers, birds, fish, and insects, and it is highly capable of analyzing texts, charts, icons, graphics, and layouts within images.
- **More Agentic**: Play as a visual agent that can reason and dynamically direct tools, which is capable of computer use and phone use.
- **Understanding long videos and capturing events**: Supports comprehending videos of over 1 hour, and this time it has a new ability of capturing event by pinpointing the relevant video segments.
- **Capable of visual localization in different formats**: Accurately localize objects in an image by generating bounding boxes or points, and it can provide stable JSON outputs for coordinates and attributes.
- **Generating structured outputs**: Supports structured outputs of the contents, benefiting usages in finance, commerce, etc for data like scans of invoices, forms, tables, etc.
- **Dynamic Resolution and Frame Rate Training for Video Understanding**: Extend dynamic resolution to the temporal dimension by adopting dynamic FPS sampling, enabling the model to comprehend videos at various sampling rates. Accordingly, we update mRoPE in the time dimension with IDs and absolute time alignment, enabling the model to learn temporal sequence and speed, and ultimately acquire the ability to pinpoint specific moments.
- **Multiple Sizes**: Available in 3B, 7B, 32B, and 72B variants to suit different deployment needs.
- **ROCm Support**: Compatible with AMD MI300X, MI325X and MI355X GPUs via SGLang (verified).
For more details, please refer to the [official Qwen2.5-VL GitHub Repository](https://github.com/QwenLM/Qwen3-VL).
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides deployment configurations optimized for AMD MI300X, MI325X and MI355X as well as Intel Xeon CPU hardware platforms and different use cases.
### 3.1 Basic Configuration
The Qwen2.5-VL series offers models in various sizes. The following configurations have been verified on AMD MI300X, MI325X and MI355X GPUs as well as Intel Xeon CPUs.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model size.
<Qwen25VLDeployment />
### 3.2 Configuration Tips
* **Memory Management**: For the 72B model on MI300X/MI325X/MI355X, we have verified successful deployment with `--context-length 128000`. Smaller context lengths can be used to reduce memory usage if needed.
* **Multi-GPU Deployment**: Use Tensor Parallelism (`--tp`) to scale across multiple GPUs. For example, use `--tp 8` for the 72B model and `--tp 2` for the 32B model on MI300X/MI325X/MI355X.
* **Xeon CPU service configuration**: Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
### 4.2 Advanced Usage
#### 4.2.1 Multi-Modal Inputs
Qwen2.5-VL supports image inputs. Here's a basic example with single image input:
```python Example
import time
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:30000/v1",
timeout=3600
)
messages = [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
}
},
{
"type": "text",
"text": "Read all the text in the image."
}
]
}
]
start = time.time()
response = client.chat.completions.create(
model="Qwen/Qwen2.5-VL-7B-Instruct",
messages=messages,
max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```
**Example Output:**
```text Output
Response costs: 2.31s
Generated text: Auntie Anne's
CINNAMON SUGAR
1 x 17,000
SUB TOTAL
17,000
GRAND TOTAL
17,000
CASH IDR
20,000
CHANGE DUE
3,000
```
**Multi-Image Input Example:**
Qwen2.5-VL can process multiple images in a single request for comparison or analysis:
```python Example
import time
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:30000/v1",
timeout=3600
)
messages = [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
}
},
{
"type": "image_url",
"image_url": {
"url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
}
},
{
"type": "text",
"text": "Compare these two images and describe the differences in 100 words or less."
}
]
}
]
start = time.time()
response = client.chat.completions.create(
model="Qwen/Qwen2.5-VL-7B-Instruct",
messages=messages,
max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```
**Example Output:**
```text Output
Response costs: 13.79s
Generated text: The first image shows a single red taxi driving on a street with a few other taxis in the background. The second image shows a large number of taxis parked in a lot, with some appearing to be in various states of repair. The first image has a single taxi with a visible license plate, while the second image has multiple taxis with different license plates. The first image has a clear view of the street and surrounding area, while the second image is taken from an elevated perspective, showing a wider view of the parking lot and the surrounding area.
```
**Note:**
- You can also provide local file paths using `file://` protocol.
- For larger images, you may need more memory, adjust `--mem-fraction-static` accordingly.
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: AMD MI300X GPU (8x)
- Model: Qwen2.5-VL-72B-Instruct
- Tensor Parallelism: 8
- SGLang Version: 0.5.6
We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. To simulate real-world usage, you can specify different input and output lengths for each request. For example, each request can have 128 input tokens, two 720p images, and 1024 output tokens.
#### 5.1.1 Latency-Sensitive Benchmark
- Model Deployment Command:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen2.5-VL-72B-Instruct \
--tp 8 \
--host 0.0.0.0 \
--port 30000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang-oai-chat \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen2.5-VL-72B-Instruct \
--dataset-name image \
--image-count 2 \
--image-resolution 720p \
--random-input-len 128 \
--random-output-len 1024 \
--num-prompts 10 \
--max-concurrency 1
```
#### 5.1.2 Throughput-Sensitive Benchmark
- Model Deployment Command:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen2.5-VL-72B-Instruct \
--tp 8 \
--host 0.0.0.0 \
--port 30000
```
- Result:
```text Output
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 37.99
Total input tokens: 24781
Total input text tokens: 821
Total input vision tokens: 23960
Total generated tokens: 4220
Total generated tokens (retokenized): 2365
Request throughput (req/s): 0.26
Input token throughput (tok/s): 652.26
Output token throughput (tok/s): 111.07
Peak output token throughput (tok/s): 128.00
Peak concurrent requests: 2
Total token throughput (tok/s): 763.34
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 3797.61
Median E2E Latency (ms): 3140.90
P90 E2E Latency (ms): 6545.54
P99 E2E Latency (ms): 7939.56
---------------Time to First Token----------------
Mean TTFT (ms): 504.45
Median TTFT (ms): 510.93
P99 TTFT (ms): 521.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 7.82
Median TPOT (ms): 7.82
P99 TPOT (ms): 7.84
---------------Inter-Token Latency----------------
Mean ITL (ms): 10.07
Median ITL (ms): 7.90
P95 ITL (ms): 15.79
P99 ITL (ms): 15.93
Max ITL (ms): 23.60
==================================================
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang-oai-chat \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen2.5-VL-72B-Instruct \
--dataset-name image \
--image-count 2 \
--image-resolution 720p \
--random-input-len 128 \
--random-output-len 1024 \
--num-prompts 1000 \
--max-concurrency 100
```
```text Output
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 454.68
Total input tokens: 2481865
Total input text tokens: 85865
Total input vision tokens: 2396000
Total generated tokens: 510855
Total generated tokens (retokenized): 296466
Request throughput (req/s): 2.20
Input token throughput (tok/s): 5458.50
Output token throughput (tok/s): 1123.55
Peak output token throughput (tok/s): 5004.00
Peak concurrent requests: 106
Total token throughput (tok/s): 6582.05
Concurrency: 98.63
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 44844.92
Median E2E Latency (ms): 42866.15
P90 E2E Latency (ms): 82798.20
P99 E2E Latency (ms): 106306.30
---------------Time to First Token----------------
Mean TTFT (ms): 4507.79
Median TTFT (ms): 1180.83
P99 TTFT (ms): 39975.22
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 80.26
Median TPOT (ms): 82.38
P99 TPOT (ms): 152.89
---------------Inter-Token Latency----------------
Mean ITL (ms): 100.66
Median ITL (ms): 13.26
P95 ITL (ms): 428.45
P99 ITL (ms): 1393.35
Max ITL (ms): 31943.26
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 MMMU Benchmark
You can evaluate the model's accuracy using the MMMU dataset:
- Benchmark Command:
```shell Command
python3 benchmark/mmmu/bench_sglang.py \
--port 30000 \
--concurrency 64
```
```text Output
Benchmark time: 97.75084622902796
answers saved to: ./answer_sglang.json
Evaluating...
answers saved to: ./answer_sglang.json
{'Accounting': {'acc': 0.633, 'num': 30},
'Agriculture': {'acc': 0.5, 'num': 30},
'Architecture_and_Engineering': {'acc': 0.367, 'num': 30},
'Art': {'acc': 0.767, 'num': 30},
'Art_Theory': {'acc': 0.9, 'num': 30},
'Basic_Medical_Science': {'acc': 0.7, 'num': 30},
'Biology': {'acc': 0.467, 'num': 30},
'Chemistry': {'acc': 0.433, 'num': 30},
'Clinical_Medicine': {'acc': 0.733, 'num': 30},
'Computer_Science': {'acc': 0.567, 'num': 30},
'Design': {'acc': 0.833, 'num': 30},
'Diagnostics_and_Laboratory_Medicine': {'acc': 0.467, 'num': 30},
'Economics': {'acc': 0.767, 'num': 30},
'Electronics': {'acc': 0.433, 'num': 30},
'Energy_and_Power': {'acc': 0.467, 'num': 30},
'Finance': {'acc': 0.533, 'num': 30},
'Geography': {'acc': 0.633, 'num': 30},
'History': {'acc': 0.7, 'num': 30},
'Literature': {'acc': 0.867, 'num': 30},
'Manage': {'acc': 0.633, 'num': 30},
'Marketing': {'acc': 0.733, 'num': 30},
'Materials': {'acc': 0.333, 'num': 30},
'Math': {'acc': 0.533, 'num': 30},
'Mechanical_Engineering': {'acc': 0.433, 'num': 30},
'Music': {'acc': 0.367, 'num': 30},
'Overall': {'acc': 0.62, 'num': 900},
'Overall-Art and Design': {'acc': 0.717, 'num': 120},
'Overall-Business': {'acc': 0.66, 'num': 150},
'Overall-Health and Medicine': {'acc': 0.693, 'num': 150},
'Overall-Humanities and Social Science': {'acc': 0.775, 'num': 120},
'Overall-Science': {'acc': 0.553, 'num': 150},
'Overall-Tech and Engineering': {'acc': 0.443, 'num': 210},
'Pharmacy': {'acc': 0.833, 'num': 30},
'Physics': {'acc': 0.7, 'num': 30},
'Psychology': {'acc': 0.767, 'num': 30},
'Public_Health': {'acc': 0.733, 'num': 30},
'Sociology': {'acc': 0.767, 'num': 30}}
eval out saved to ./val_sglang.json
Overall accuracy: 0.62
```
docs/cookbook/autoregressive/Qwen/Qwen3-Coder-Next.mdx
+905
View File
@@ -0,0 +1,905 @@
---
title: Qwen3-Coder-Next
metatags:
description: "Deploy Qwen3-Coder-Next code-focused models with SGLang on AMD MI300X - available in 3B to 80B sizes with enhanced code understanding."
---
import { Qwen3CoderNextDeployment } from '/src/snippets/autoregressive/qwen3-coder-next-deployment.jsx';
## 1. Model Introduction
[Qwen3-Coder-Next](https://huggingface.co/Qwen/Qwen3-Coder-Next) is a cost-efficient code-focused language model from the Qwen team (Alibaba). With 80B total parameters but only 3B activated parameters, it achieves performance comparable to models with 10–20x more active parameters through its innovative hybrid architecture.
**Key Features:**
- **Hybrid Architecture**: Uses a 48-layer hybrid layout combining Gated DeltaNet and Gated Attention with Mixture-of-Experts (512 total experts, 10 activated, 1 shared), enabling exceptional efficiency.
- **Tool Calling Support**: Advanced agentic capabilities with native support for function calling and tool use via the `qwen3_coder` parser.
- **Extended Context Length**: Supports up to 256K tokens for processing large codebases and long documents.
- **Cost-Efficient Inference**: Only 3B parameters activated per token, making it ideal for local development and cost-effective deployment at scale.
- **IDE Integration**: Compatible with Claude Code, Qwen Code, Cline, and other IDE platforms.
For more details, please refer to the [Qwen3-Coder-Next model card](https://huggingface.co/Qwen/Qwen3-Coder-Next).
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
**Note:** Qwen3-Coder-Next requires SGLang v0.5.8 or later.
## 3. Model Deployment
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and deployment options.
<Qwen3CoderNextDeployment />
### 3.2 Configuration Tips
- **Context Length**: The model supports up to 256K tokens natively. If you encounter OOM issues, try `--context-length 32768`.
- **Tool Use**: To enable tool calling capabilities, use the `--tool-call-parser qwen3_coder` flag.
- **Sampling Parameters**: SGLang automatically applies the recommended sampling parameters from the model's `generation_config.json`. No manual configuration is needed.
- **Mamba Radix Cache**: Qwen3-Coder-Next's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`:
- **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage.
- **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64).
- **Xeon CPU service configuration**: Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation
**Deployment Command:**
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-Coder-Next \
--tp 2 \
--tool-call-parser qwen3_coder \
--host 0.0.0.0 \
--port 30000
```
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Advanced Usage
#### 4.2.1 Code Generation Example
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="Qwen/Qwen3-Coder-Next",
messages=[
{"role": "user", "content": "Write a Python function that implements binary search on a sorted list. Include type hints."}
],
max_tokens=2048
)
print(response.choices[0].message.content)
```
**Example Output:**
````text Output
Here's a Python function implementing binary search on a sorted list, with comprehensive type hints:
```python
from typing import Sequence, TypeVar, Optional
T = TypeVar('T')
def binary_search(sorted_list: Sequence[T], target: T) -> Optional[int]:
"""
Perform binary search on a sorted list to find the index of a target element.
Args:
sorted_list: A sequence (e.g., list, tuple) sorted in ascending order.
target: The element to search for in the list.
Returns:
The index of the target element if found, or None if not found.
Time Complexity: O(log n)
Space Complexity: O(1)
Note:
The function assumes the list is sorted in ascending order.
If the list contains duplicate elements, it returns the index of one of them.
"""
left = 0
right = len(sorted_list) - 1
while left <= right:
mid = (left + right) // 2
mid_val = sorted_list[mid]
if mid_val == target:
return mid
elif mid_val < target:
left = mid + 1
else:
right = mid - 1
return None
```
### Example usage:
```python
# Example 1: Finding an existing element
numbers = [1, 3, 5, 7, 9, 11]
print(binary_search(numbers, 7)) # Output: 3
# Example 2: Element not in the list
print(binary_search(numbers, 4)) # Output: None
# Example 3: Empty list
print(binary_search([], 5)) # Output: None
# Example 4: Single element
print(binary_search([1], 1)) # Output: 0
print(binary_search([1], 2)) # Output: None
```
### Key features:
- Uses `TypeVar` to support generic types (as long as comparison operations are defined)
- Returns `Optional[int]` to indicate either the index or no match found
- Uses `Sequence[T]` to accept any sequence type (list, tuple, etc.)
- Includes comprehensive docstring with time/space complexity
- Implements standard iterative binary search for O(1) space complexity
````
#### 4.2.2 Streaming Example
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="Qwen/Qwen3-Coder-Next",
messages=[
{"role": "user", "content": "Explain the difference between a stack and a queue in 3 sentences."}
],
max_tokens=512,
stream=True
)
for chunk in response:
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
print()
```
**Example Output:**
```text Output
A **stack** follows the **Last In, First Out (LIFO)** principle, meaning the last element added is the first one removed—operations like `push` (add) and `pop` (remove) occur at the same end, called the *top*. In contrast, a **queue** follows the **First In, First Out (FIFO)** principle, where elements are added at the *back* (enqueue) and removed from the *front* (dequeue), preserving the order of insertion. This structural difference makes stacks ideal for tasks like function call management and expression evaluation, while queues suit scheduling, buffering, and breadth-first traversal.
```
#### 4.2.3 Tool Calling Example
Qwen3-Coder-Next supports tool calling capabilities. Make sure `--tool-call-parser qwen3_coder` is included in the deployment command above.
**Python Example:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "execute_code",
"description": "Execute Python code and return the result",
"parameters": {
"type": "object",
"properties": {
"code": {
"type": "string",
"description": "The Python code to execute"
}
},
"required": ["code"]
}
}
}
]
response = client.chat.completions.create(
model="Qwen/Qwen3-Coder-Next",
messages=[
{"role": "user", "content": "Calculate the factorial of 10 using Python"}
],
tools=tools
)
# Check if the model wants to call a tool
if response.choices[0].message.tool_calls:
tool_call = response.choices[0].message.tool_calls[0]
print(f"Tool: {tool_call.function.name}")
print(f"Arguments: {tool_call.function.arguments}")
else:
print(response.choices[0].message.content)
```
**Example Output:**
```text Output
Tool: execute_code
Arguments: {"code": "import math\nmath.factorial(10)"}
```
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: NVIDIA B200 GPU (2x)
- Model: Qwen/Qwen3-Coder-Next
- Tensor Parallelism: 2
- sglang version: 0.5.8+
#### 5.1.1 Standard Scenario Benchmark
- Model Deployment Command:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-Coder-Next \
--tp 2 \
--host 0.0.0.0 \
--port 30000
```
##### 5.1.1.1 Low Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen3-Coder-Next \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1
```
- Result:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 27.86
Total input tokens: 6101
Total input text tokens: 6101
Total generated tokens: 4220
Total generated tokens (retokenized): 4218
Request throughput (req/s): 0.36
Input token throughput (tok/s): 219.00
Output token throughput (tok/s): 151.48
Peak output token throughput (tok/s): 166.00
Peak concurrent requests: 2
Total token throughput (tok/s): 370.48
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 2784.14
Median E2E Latency (ms): 2258.08
P90 E2E Latency (ms): 5044.43
P99 E2E Latency (ms): 6130.52
---------------Time to First Token----------------
Mean TTFT (ms): 161.68
Median TTFT (ms): 168.09
P99 TTFT (ms): 183.26
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 6.19
Median TPOT (ms): 6.23
P99 TPOT (ms): 6.32
---------------Inter-Token Latency----------------
Mean ITL (ms): 6.23
Median ITL (ms): 6.23
P95 ITL (ms): 6.51
P99 ITL (ms): 6.64
Max ITL (ms): 13.45
==================================================
```
##### 5.1.1.2 Medium Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen3-Coder-Next \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16
```
- Result:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 39.06
Total input tokens: 39668
Total input text tokens: 39668
Total generated tokens: 40805
Total generated tokens (retokenized): 40789
Request throughput (req/s): 2.05
Input token throughput (tok/s): 1015.62
Output token throughput (tok/s): 1044.73
Peak output token throughput (tok/s): 1664.00
Peak concurrent requests: 21
Total token throughput (tok/s): 2060.34
Concurrency: 14.16
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 6910.97
Median E2E Latency (ms): 7248.27
P90 E2E Latency (ms): 11612.63
P99 E2E Latency (ms): 13933.91
---------------Time to First Token----------------
Mean TTFT (ms): 183.48
Median TTFT (ms): 156.50
P99 TTFT (ms): 311.46
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 13.61
Median TPOT (ms): 13.59
P99 TPOT (ms): 21.11
---------------Inter-Token Latency----------------
Mean ITL (ms): 13.22
Median ITL (ms): 9.76
P95 ITL (ms): 10.43
P99 ITL (ms): 158.04
Max ITL (ms): 394.39
==================================================
```
##### 5.1.1.3 High Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen3-Coder-Next \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100
```
- Result:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 102.81
Total input tokens: 249831
Total input text tokens: 249831
Total generated tokens: 252662
Total generated tokens (retokenized): 252536
Request throughput (req/s): 4.86
Input token throughput (tok/s): 2429.99
Output token throughput (tok/s): 2457.53
Peak output token throughput (tok/s): 5299.00
Peak concurrent requests: 109
Total token throughput (tok/s): 4887.52
Concurrency: 94.28
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 19385.20
Median E2E Latency (ms): 17584.09
P90 E2E Latency (ms): 36762.15
P99 E2E Latency (ms): 42518.35
---------------Time to First Token----------------
Mean TTFT (ms): 270.62
Median TTFT (ms): 159.65
P99 TTFT (ms): 938.90
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 38.57
Median TPOT (ms): 41.78
P99 TPOT (ms): 53.28
---------------Inter-Token Latency----------------
Mean ITL (ms): 37.90
Median ITL (ms): 18.26
P95 ITL (ms): 167.82
P99 ITL (ms): 311.45
Max ITL (ms): 993.20
==================================================
```
#### 5.1.2 Reasoning Scenario Benchmark
- Model Deployment Command:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-Coder-Next \
--tp 2 \
--host 0.0.0.0 \
--port 30000
```
##### 5.1.2.1 Low Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen3-Coder-Next \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 10 \
--max-concurrency 1
```
- Result:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 285.02
Total input tokens: 6101
Total input text tokens: 6101
Total generated tokens: 44462
Total generated tokens (retokenized): 44432
Request throughput (req/s): 0.04
Input token throughput (tok/s): 21.41
Output token throughput (tok/s): 156.00
Peak output token throughput (tok/s): 173.00
Peak concurrent requests: 2
Total token throughput (tok/s): 177.40
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 28499.54
Median E2E Latency (ms): 30424.65
P90 E2E Latency (ms): 49132.26
P99 E2E Latency (ms): 51075.28
---------------Time to First Token----------------
Mean TTFT (ms): 95.51
Median TTFT (ms): 93.86
P99 TTFT (ms): 112.56
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 6.24
Median TPOT (ms): 6.30
P99 TPOT (ms): 6.60
---------------Inter-Token Latency----------------
Mean ITL (ms): 6.39
Median ITL (ms): 6.34
P95 ITL (ms): 7.16
P99 ITL (ms): 7.42
Max ITL (ms): 12.48
==================================================
```
##### 5.1.2.2 Medium Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen3-Coder-Next \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 80 \
--max-concurrency 16
```
- Result:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 237.77
Total input tokens: 39668
Total input text tokens: 39668
Total generated tokens: 318306
Total generated tokens (retokenized): 315646
Request throughput (req/s): 0.34
Input token throughput (tok/s): 166.83
Output token throughput (tok/s): 1338.72
Peak output token throughput (tok/s): 1727.00
Peak concurrent requests: 19
Total token throughput (tok/s): 1505.55
Concurrency: 13.88
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 41266.21
Median E2E Latency (ms): 41010.10
P90 E2E Latency (ms): 77574.22
P99 E2E Latency (ms): 82688.04
---------------Time to First Token----------------
Mean TTFT (ms): 140.73
Median TTFT (ms): 84.52
P99 TTFT (ms): 365.86
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 10.32
Median TPOT (ms): 10.38
P99 TPOT (ms): 10.87
---------------Inter-Token Latency----------------
Mean ITL (ms): 10.34
Median ITL (ms): 10.19
P95 ITL (ms): 10.75
P99 ITL (ms): 11.18
Max ITL (ms): 206.79
==================================================
```
##### 5.1.2.3 High Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen3-Coder-Next \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 320 \
--max-concurrency 64
```
- Result:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 384.82
Total input tokens: 158939
Total input text tokens: 158939
Total generated tokens: 1301025
Total generated tokens (retokenized): 1299908
Request throughput (req/s): 0.83
Input token throughput (tok/s): 413.02
Output token throughput (tok/s): 3380.83
Peak output token throughput (tok/s): 4317.00
Peak concurrent requests: 69
Total token throughput (tok/s): 3793.85
Concurrency: 56.42
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 67847.54
Median E2E Latency (ms): 70724.38
P90 E2E Latency (ms): 120888.83
P99 E2E Latency (ms): 133234.48
---------------Time to First Token----------------
Mean TTFT (ms): 212.24
Median TTFT (ms): 115.96
P99 TTFT (ms): 652.93
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 16.76
Median TPOT (ms): 16.99
P99 TPOT (ms): 18.18
---------------Inter-Token Latency----------------
Mean ITL (ms): 16.64
Median ITL (ms): 15.83
P95 ITL (ms): 31.64
P99 ITL (ms): 90.85
Max ITL (ms): 576.60
==================================================
```
#### 5.1.3 Summarization Scenario Benchmark
##### 5.1.3.1 Low Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen3-Coder-Next \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1
```
- Result:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 29.42
Total input tokens: 41941
Total input text tokens: 41941
Total generated tokens: 4220
Total generated tokens (retokenized): 4220
Request throughput (req/s): 0.34
Input token throughput (tok/s): 1425.35
Output token throughput (tok/s): 143.42
Peak output token throughput (tok/s): 169.00
Peak concurrent requests: 3
Total token throughput (tok/s): 1568.77
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 2941.19
Median E2E Latency (ms): 2411.84
P90 E2E Latency (ms): 5661.26
P99 E2E Latency (ms): 6497.45
---------------Time to First Token----------------
Mean TTFT (ms): 139.46
Median TTFT (ms): 160.33
P99 TTFT (ms): 184.30
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 6.56
Median TPOT (ms): 6.65
P99 TPOT (ms): 7.29
---------------Inter-Token Latency----------------
Mean ITL (ms): 6.65
Median ITL (ms): 6.68
P95 ITL (ms): 7.39
P99 ITL (ms): 7.51
Max ITL (ms): 16.34
==================================================
```
##### 5.1.3.2 Medium Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen3-Coder-Next \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16
```
- Result:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 41.62
Total input tokens: 300020
Total input text tokens: 300020
Total generated tokens: 41669
Total generated tokens (retokenized): 41664
Request throughput (req/s): 1.92
Input token throughput (tok/s): 7208.67
Output token throughput (tok/s): 1001.19
Peak output token throughput (tok/s): 1536.00
Peak concurrent requests: 21
Total token throughput (tok/s): 8209.86
Concurrency: 14.27
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 7421.29
Median E2E Latency (ms): 7985.77
P90 E2E Latency (ms): 12122.09
P99 E2E Latency (ms): 14595.05
---------------Time to First Token----------------
Mean TTFT (ms): 248.49
Median TTFT (ms): 179.25
P99 TTFT (ms): 915.90
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 14.13
Median TPOT (ms): 14.28
P99 TPOT (ms): 24.02
---------------Inter-Token Latency----------------
Mean ITL (ms): 13.80
Median ITL (ms): 10.46
P95 ITL (ms): 11.00
P99 ITL (ms): 173.14
Max ITL (ms): 823.32
==================================================
```
##### 5.1.3.3 High Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen3-Coder-Next \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 320 \
--max-concurrency 64
```
- Result:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 85.74
Total input tokens: 1273893
Total input text tokens: 1273893
Total generated tokens: 170000
Total generated tokens (retokenized): 169983
Request throughput (req/s): 3.73
Input token throughput (tok/s): 14858.12
Output token throughput (tok/s): 1982.80
Peak output token throughput (tok/s): 3734.00
Peak concurrent requests: 70
Total token throughput (tok/s): 16840.92
Concurrency: 59.75
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 16008.12
Median E2E Latency (ms): 15460.65
P90 E2E Latency (ms): 27705.81
P99 E2E Latency (ms): 32874.74
---------------Time to First Token----------------
Mean TTFT (ms): 476.99
Median TTFT (ms): 177.50
P99 TTFT (ms): 3014.39
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 29.81
Median TPOT (ms): 31.19
P99 TPOT (ms): 45.53
---------------Inter-Token Latency----------------
Mean ITL (ms): 29.29
Median ITL (ms): 15.75
P95 ITL (ms): 173.94
P99 ITL (ms): 202.00
Max ITL (ms): 2783.23
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 GSM8K Benchmark
- **Benchmark Command:**
```shell Command
python benchmark/gsm8k/bench_sglang.py --port 30000
```
- **Test Results:**
```text Output
Accuracy: 0.965
Invalid: 0.000
Latency: 26.407 s
Output throughput: 929.132 token/s
```
#### 5.2.2 MMLU Benchmark
- **Benchmark Command:**
```shell Command
cd benchmark/mmlu
bash download_data.sh
python3 bench_sglang.py --port 30000
```
- **Test Results:**
```text Output
subject: abstract_algebra, #q:100, acc: 0.780
subject: anatomy, #q:135, acc: 0.807
subject: astronomy, #q:152, acc: 0.921
subject: business_ethics, #q:100, acc: 0.820
subject: clinical_knowledge, #q:265, acc: 0.860
subject: college_biology, #q:144, acc: 0.944
subject: college_chemistry, #q:100, acc: 0.590
subject: college_computer_science, #q:100, acc: 0.820
subject: college_mathematics, #q:100, acc: 0.800
subject: college_medicine, #q:173, acc: 0.803
subject: college_physics, #q:102, acc: 0.775
subject: computer_security, #q:100, acc: 0.880
subject: conceptual_physics, #q:235, acc: 0.936
subject: econometrics, #q:114, acc: 0.807
subject: electrical_engineering, #q:145, acc: 0.834
subject: elementary_mathematics, #q:378, acc: 0.854
subject: formal_logic, #q:126, acc: 0.802
subject: global_facts, #q:100, acc: 0.610
subject: high_school_biology, #q:310, acc: 0.971
subject: high_school_chemistry, #q:203, acc: 0.803
subject: high_school_computer_science, #q:100, acc: 0.920
subject: high_school_european_history, #q:165, acc: 0.891
subject: high_school_geography, #q:198, acc: 0.929
subject: high_school_government_and_politics, #q:193, acc: 0.969
subject: high_school_macroeconomics, #q:390, acc: 0.903
subject: high_school_mathematics, #q:270, acc: 0.689
subject: high_school_microeconomics, #q:238, acc: 0.962
subject: high_school_physics, #q:151, acc: 0.854
subject: high_school_psychology, #q:545, acc: 0.947
subject: high_school_statistics, #q:216, acc: 0.815
subject: high_school_us_history, #q:204, acc: 0.907
subject: high_school_world_history, #q:237, acc: 0.937
subject: human_aging, #q:223, acc: 0.821
subject: human_sexuality, #q:131, acc: 0.840
subject: international_law, #q:121, acc: 0.934
subject: jurisprudence, #q:108, acc: 0.870
subject: logical_fallacies, #q:163, acc: 0.847
subject: machine_learning, #q:112, acc: 0.812
subject: management, #q:103, acc: 0.922
subject: marketing, #q:234, acc: 0.923
subject: medical_genetics, #q:100, acc: 0.970
subject: miscellaneous, #q:783, acc: 0.941
subject: moral_disputes, #q:346, acc: 0.850
subject: moral_scenarios, #q:895, acc: 0.726
subject: nutrition, #q:306, acc: 0.915
subject: philosophy, #q:311, acc: 0.859
subject: prehistory, #q:324, acc: 0.889
subject: professional_accounting, #q:282, acc: 0.723
subject: professional_law, #q:1534, acc: 0.648
subject: professional_medicine, #q:272, acc: 0.923
subject: professional_psychology, #q:612, acc: 0.845
subject: public_relations, #q:110, acc: 0.782
subject: security_studies, #q:245, acc: 0.796
subject: sociology, #q:201, acc: 0.925
subject: us_foreign_policy, #q:100, acc: 0.950
subject: virology, #q:166, acc: 0.572
subject: world_religions, #q:171, acc: 0.883
Total latency: 208.985
Average accuracy: 0.834
```
docs/cookbook/autoregressive/Qwen/Qwen3-Coder.mdx
+788
View File
@@ -0,0 +1,788 @@
---
title: Qwen3-Coder
metatags:
description: "Deploy Qwen3-Coder(480B, 30B) MoE coding model with SGLang on AMD MI300X (MI325X, MI355X)"
---
import { Qwen3CoderDeployment } from '/src/snippets/autoregressive/qwen3-coder-deployment.jsx';
## 1. Model Introduction
[Qwen3-Coder](https://huggingface.co/collections/Qwen/qwen3-coder) is the latest code-focused large language model series from the Qwen team. Built on the foundation of Qwen3, Qwen3-Coder delivers exceptional performance in code generation, understanding, and reasoning tasks.
**Key Features:**
- **State-of-the-art Coding Performance**: Achieves top-tier results on HumanEval, MBPP, LiveCodeBench, and other major coding benchmarks.
- **Tool Calling Support**: Native support for function calling and tool use, enabling seamless integration with external APIs and services.
- **Extended Context Length**: Supports up to 256K tokens for processing large codebases and long documents.
- **Multilingual Code Support**: Proficient in Python, JavaScript, TypeScript, Java, C++, Go, Rust, and many other programming languages.
- **MoE Architecture**: Efficient Mixture-of-Experts design for optimal performance-to-cost ratio.
- **ROCm Support**: Compatible with AMD MI300X, MI325X and MI355X GPUs via SGLang (verified).
- **NVIDIA GPU Support**: Compatible with NVIDIA GB200 and B200 GPUs via SGLang (verified).
For more details, please refer to the [official Qwen3-Coder GitHub Repository](https://github.com/QwenLM/Qwen3-Coder).
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides deployment configurations verified on AMD MI300X, MI325X, MI355X, NVIDIA B200, GB200, and Intel Xeon CPU hardware platforms.
### 3.1 Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, and quantization method.
<Qwen3CoderDeployment />
### 3.2 Configuration Tips
**AMD (MI300X/MI325X/MI355X):**
* **Memory Management**: We have verified successful deployment on MI300X/MI325X/MI355X with `--context-length 8192`. Larger context lengths may be supported but require additional memory.
* **Expert Parallelism**: For 480B-A35B with FP8 quantization, `--ep 2` is required to satisfy the dimension alignment requirement.
* **Page Size**: `--page-size 32` is recommended for MoE models to optimize memory usage.
* **Environment Variable**: If you encounter aiter-related issues, try setting `SGLANG_USE_AITER=0`.
**NVIDIA (B200/GB200):**
* **GB200 Parallelism**: Use `--tp 4 --ep 4` on GB200. B200 uses the default NVIDIA settings generated above.
* **NVFP4 Quantization**: Requires `--quantization modelopt_fp4` and uses a different model path (`nvidia/Qwen3-Coder-...`).
* **DP Attention**: NVFP4 configuration supports `--enable-dp-attention` for improved throughput.
**Intel Xeon CPU:**
* Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
**General:**
* **Tool Use**: To enable tool calling capabilities, add `--tool-call-parser qwen3_coder` to the launch command.
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Advanced Usage
#### 4.2.1 Code Generation Example
```python Example
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:30000/v1",
timeout=3600
)
messages = [
{
"role": "user",
"content": "Write a Python function that implements binary search on a sorted list. Include docstring and type hints."
}
]
response = client.chat.completions.create(
model="Qwen/Qwen3-Coder-480B-A35B-Instruct",
messages=messages,
max_tokens=2048,
temperature=0.7
)
print(response.choices[0].message.content)
```
**Example Output:**
````text Output
```python
from typing import List, Optional, TypeVar
T = TypeVar('T')
def binary_search(arr: List[T], target: T) -> Optional[int]:
"""
Perform binary search on a sorted list to find the index of a target element.
This function implements the binary search algorithm, which efficiently finds
a target value in a sorted array by repeatedly dividing the search interval
in half.
Args:
arr (List[T]): A sorted list of elements to search through.
target (T): The element to search for in the list.
Returns:
Optional[int]: The index of the target element if found, None otherwise.
Time Complexity:
O(log n) where n is the number of elements in the array.
Space Complexity:
O(1) - iterative implementation uses constant extra space.
Examples:
>>> binary_search([1, 2, 3, 4, 5], 3)
2
>>> binary_search([1, 2, 3, 4, 5], 6)
None
>>> binary_search(['a', 'b', 'c', 'd'], 'b')
1
>>> binary_search([], 1)
None
"""
if not arr:
return None
left: int = 0
right: int = len(arr) - 1
while left <= right:
mid: int = (left + right) // 2
if arr[mid] == target:
return mid
elif arr[mid] < target:
left = mid + 1
else:
right = mid - 1
return None
# Alternative recursive implementation
def binary_search_recursive(arr: List[T], target: T, left: int = 0, right: Optional[int] = None) -> Optional[int]:
"""
Perform binary search recursively on a sorted list to find the index of a target element.
Args:
arr (List[T]): A sorted list of elements to search through.
target (T): The element to search for in the list.
left (int): Left boundary of the search range (inclusive).
right (Optional[int]): Right boundary of the search range (inclusive).
Returns:
Optional[int]: The index of the target element if found, None otherwise.
Time Complexity:
O(log n) where n is the number of elements in the array.
Space Complexity:
O(log n) due to recursive call stack.
Examples:
>>> binary_search_recursive([1, 2, 3, 4, 5], 3)
2
>>> binary_search_recursive([1, 2, 3, 4, 5], 6)
None
"""
if not arr:
return None
if right is None:
right = len(arr) - 1
if left > right:
return None
mid: int = (left + right) // 2
if arr[mid] == target:
return mid
elif arr[mid] < target:
return binary_search_recursive(arr, target, mid + 1, right)
else:
return binary_search_recursive(arr, target, left, mid - 1)
```
This implementation provides:
1. **Main function** (`binary_search`): An iterative implementation that's more memory-efficient
2. **Alternative function** (`binary_search_recursive`): A recursive implementation for educational purposes
3. **Type hints**: Using generics (`TypeVar`) to work with any comparable type
4. **Comprehensive docstring**: Including description, parameters, return value, complexity analysis, and examples
5. **Edge case handling**: Empty lists, elements not found, etc.
6. **Clear variable names**: Self-documenting code
7. **Examples**: Doctest-style examples in the docstring
The function works with any sorted list of comparable elements (integers, strings, etc.) and returns the index of the target element if found, or `None` if not found.
````
#### 4.2.2 Tool Calling Example
Qwen3-Coder supports tool calling capabilities. Enable the tool call parser during deployment. The following example uses 30B-A3B model:
```shell Command
SGLANG_USE_AITER=0 python -m sglang.launch_server \
--model Qwen/Qwen3-Coder-30B-A3B-Instruct \
--tp 1 \
--context-length 8192 \
--page-size 32 \
--tool-call-parser qwen3_coder
```
**Python Example:**
```python Example
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:30000/v1",
timeout=3600
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "execute_code",
"description": "Execute Python code and return the result",
"parameters": {
"type": "object",
"properties": {
"code": {
"type": "string",
"description": "The Python code to execute"
}
},
"required": ["code"]
}
}
}
]
response = client.chat.completions.create(
model="Qwen/Qwen3-Coder-30B-A3B-Instruct",
messages=[
{"role": "user", "content": "Calculate the factorial of 10 using Python"}
],
tools=tools,
temperature=0.7
)
# Check if the model wants to call a tool
if response.choices[0].message.tool_calls:
tool_call = response.choices[0].message.tool_calls[0]
print(f"Tool: {tool_call.function.name}")
print(f"Arguments: {tool_call.function.arguments}")
else:
# Model may return tool call in content format
print(response.choices[0].message.content)
```
**Example Output:**
```text Output
Tool: execute_code
Arguments: {"code": "def factorial(n):\n if n == 0 or n == 1:\n return 1\n else:\n return n * factorial(n-1)\n\nresult = factorial(10)\nresult"}
```
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: AMD MI300X GPU (8x)
- Model: Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8
- Tensor Parallelism: 8
- Expert Parallelism: 2
- sglang version: 0.5.7
We use SGLang's built-in benchmarking tool to conduct performance evaluation with random dataset.
#### 5.1.1 AMD Standard Scenario Benchmark
- Model Deployment Command:
```shell Command
SGLANG_USE_AITER=0 python -m sglang.launch_server \
--model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
--tp 8 \
--ep 2 \
--context-length 8192 \
--page-size 32 \
--trust-remote-code
```
##### 5.1.1.1 Low Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 73.79
Total input tokens: 6101
Total input text tokens: 6101
Total generated tokens: 4220
Total generated tokens (retokenized): 4104
Request throughput (req/s): 0.14
Input token throughput (tok/s): 82.68
Output token throughput (tok/s): 57.19
Peak output token throughput (tok/s): 59.00
Peak concurrent requests: 2
Total token throughput (tok/s): 139.86
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 7376.26
Median E2E Latency (ms): 5851.51
P90 E2E Latency (ms): 13351.89
P99 E2E Latency (ms): 16908.32
---------------Time to First Token----------------
Mean TTFT (ms): 191.93
Median TTFT (ms): 126.06
P99 TTFT (ms): 662.15
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 17.06
Median TPOT (ms): 17.07
P99 TPOT (ms): 17.08
---------------Inter-Token Latency----------------
Mean ITL (ms): 17.06
Median ITL (ms): 17.06
P95 ITL (ms): 17.14
P99 ITL (ms): 17.19
Max ITL (ms): 18.53
==================================================
```
##### 5.1.1.2 Medium Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 87.04
Total input tokens: 39668
Total input text tokens: 39668
Total generated tokens: 40805
Total generated tokens (retokenized): 40364
Request throughput (req/s): 0.92
Input token throughput (tok/s): 455.77
Output token throughput (tok/s): 468.83
Peak output token throughput (tok/s): 608.00
Peak concurrent requests: 20
Total token throughput (tok/s): 924.59
Concurrency: 13.76
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 14966.88
Median E2E Latency (ms): 15871.93
P90 E2E Latency (ms): 24983.41
P99 E2E Latency (ms): 29504.85
---------------Time to First Token----------------
Mean TTFT (ms): 388.94
Median TTFT (ms): 157.49
P99 TTFT (ms): 1318.63
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 29.41
Median TPOT (ms): 29.22
P99 TPOT (ms): 43.48
---------------Inter-Token Latency----------------
Mean ITL (ms): 28.64
Median ITL (ms): 26.42
P95 ITL (ms): 27.51
P99 ITL (ms): 131.63
Max ITL (ms): 995.11
==================================================
```
##### 5.1.1.3 High Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 320 \
--max-concurrency 64
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 177.82
Total input tokens: 158939
Total input text tokens: 158939
Total generated tokens: 170134
Total generated tokens (retokenized): 168387
Request throughput (req/s): 1.80
Input token throughput (tok/s): 893.84
Output token throughput (tok/s): 956.80
Peak output token throughput (tok/s): 1728.00
Peak concurrent requests: 70
Total token throughput (tok/s): 1850.64
Concurrency: 58.88
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 32716.53
Median E2E Latency (ms): 30896.37
P90 E2E Latency (ms): 65605.24
P99 E2E Latency (ms): 80970.63
---------------Time to First Token----------------
Mean TTFT (ms): 372.97
Median TTFT (ms): 181.67
P99 TTFT (ms): 529.01
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 62.98
Median TPOT (ms): 50.44
P99 TPOT (ms): 204.24
---------------Inter-Token Latency----------------
Mean ITL (ms): 60.95
Median ITL (ms): 37.87
P95 ITL (ms): 143.98
P99 ITL (ms): 148.02
Max ITL (ms): 36863.32
==================================================
```
#### 5.1.2 NVIDIA (B200/GB200) Standard Scenario Benchmark
The following runs use the same random dataset benchmark client commands as the AMD section. On B200, launch the server with the following command:
```bash
sglang serve --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 --tp 8 --ep 8 --context-length 8192 --page-size 32 --trust-remote-code
##### 5.1.2.1 FP8 Model
- Low Concurrency:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 42.68
Total input tokens: 6101
Total input text tokens: 6101
Total generated tokens: 4220
Total generated tokens (retokenized): 4204
Request throughput (req/s): 0.23
Input token throughput (tok/s): 142.95
Output token throughput (tok/s): 98.88
Peak output token throughput (tok/s): 102.00
Peak concurrent requests: 2
Total token throughput (tok/s): 241.83
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 4266.06
Median E2E Latency (ms): 3420.24
P90 E2E Latency (ms): 7717.19
P99 E2E Latency (ms): 9504.50
---------------Time to First Token----------------
Mean TTFT (ms): 112.03
Median TTFT (ms): 112.70
P99 TTFT (ms): 115.35
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 9.87
Median TPOT (ms): 9.86
P99 TPOT (ms): 9.92
---------------Inter-Token Latency----------------
Mean ITL (ms): 9.87
Median ITL (ms): 9.87
P95 ITL (ms): 10.06
P99 ITL (ms): 10.18
Max ITL (ms): 14.80
==================================================
```
- Medium Concurrency:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 60.80
Total input tokens: 39668
Total input text tokens: 39668
Total generated tokens: 40805
Total generated tokens (retokenized): 40543
Request throughput (req/s): 1.32
Input token throughput (tok/s): 652.43
Output token throughput (tok/s): 671.13
Peak output token throughput (tok/s): 864.00
Peak concurrent requests: 20
Total token throughput (tok/s): 1323.57
Concurrency: 13.93
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 10587.26
Median E2E Latency (ms): 11486.18
P90 E2E Latency (ms): 17374.75
P99 E2E Latency (ms): 21107.18
---------------Time to First Token----------------
Mean TTFT (ms): 155.27
Median TTFT (ms): 121.57
P99 TTFT (ms): 294.31
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 20.77
Median TPOT (ms): 21.13
P99 TPOT (ms): 23.62
---------------Inter-Token Latency----------------
Mean ITL (ms): 20.49
Median ITL (ms): 18.73
P95 ITL (ms): 19.65
P99 ITL (ms): 98.85
Max ITL (ms): 536.87
==================================================
```
- High Concurrency:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 100.07
Total input tokens: 158939
Total input text tokens: 158939
Total generated tokens: 170134
Total generated tokens (retokenized): 169119
Request throughput (req/s): 3.20
Input token throughput (tok/s): 1588.32
Output token throughput (tok/s): 1700.19
Peak output token throughput (tok/s): 2303.00
Peak concurrent requests: 71
Total token throughput (tok/s): 3288.51
Concurrency: 57.93
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 18114.01
Median E2E Latency (ms): 18279.15
P90 E2E Latency (ms): 30557.22
P99 E2E Latency (ms): 35889.84
---------------Time to First Token----------------
Mean TTFT (ms): 346.40
Median TTFT (ms): 129.75
P99 TTFT (ms): 1370.20
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 33.76
Median TPOT (ms): 34.62
P99 TPOT (ms): 39.97
---------------Inter-Token Latency----------------
Mean ITL (ms): 33.48
Median ITL (ms): 25.70
P95 ITL (ms): 99.36
P99 ITL (ms): 132.30
Max ITL (ms): 1132.39
==================================================
```
##### 5.1.2.2 NVFP4 Model
- Low Concurrency:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 34.49
Total input tokens: 6101
Total input text tokens: 6101
Total generated tokens: 4220
Total generated tokens (retokenized): 4218
Request throughput (req/s): 0.29
Input token throughput (tok/s): 176.87
Output token throughput (tok/s): 122.34
Peak output token throughput (tok/s): 127.00
Peak concurrent requests: 2
Total token throughput (tok/s): 299.21
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 3448.01
Median E2E Latency (ms): 2768.11
P90 E2E Latency (ms): 6225.73
P99 E2E Latency (ms): 7668.26
---------------Time to First Token----------------
Mean TTFT (ms): 104.55
Median TTFT (ms): 105.38
P99 TTFT (ms): 105.63
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 7.94
Median TPOT (ms): 7.95
P99 TPOT (ms): 7.97
---------------Inter-Token Latency----------------
Mean ITL (ms): 7.94
Median ITL (ms): 7.94
P95 ITL (ms): 8.05
P99 ITL (ms): 8.11
Max ITL (ms): 24.64
==================================================
```
- Medium Concurrency:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 43.30
Total input tokens: 39668
Total input text tokens: 39668
Total generated tokens: 40805
Total generated tokens (retokenized): 39975
Request throughput (req/s): 1.85
Input token throughput (tok/s): 916.16
Output token throughput (tok/s): 942.42
Peak output token throughput (tok/s): 1264.00
Peak concurrent requests: 21
Total token throughput (tok/s): 1858.57
Concurrency: 13.90
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 7521.95
Median E2E Latency (ms): 8246.89
P90 E2E Latency (ms): 12370.93
P99 E2E Latency (ms): 15023.96
---------------Time to First Token----------------
Mean TTFT (ms): 137.27
Median TTFT (ms): 109.59
P99 TTFT (ms): 208.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 14.69
Median TPOT (ms): 14.87
P99 TPOT (ms): 17.63
---------------Inter-Token Latency----------------
Mean ITL (ms): 14.51
Median ITL (ms): 12.75
P95 ITL (ms): 13.33
P99 ITL (ms): 92.85
Max ITL (ms): 113.70
==================================================
```
- High Concurrency:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 73.93
Total input tokens: 158939
Total input text tokens: 158939
Total generated tokens: 170134
Total generated tokens (retokenized): 168841
Request throughput (req/s): 4.33
Input token throughput (tok/s): 2149.98
Output token throughput (tok/s): 2301.42
Peak output token throughput (tok/s): 3497.00
Peak concurrent requests: 71
Total token throughput (tok/s): 4451.40
Concurrency: 58.28
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 13463.58
Median E2E Latency (ms): 13498.74
P90 E2E Latency (ms): 22957.10
P99 E2E Latency (ms): 26656.95
---------------Time to First Token----------------
Mean TTFT (ms): 239.00
Median TTFT (ms): 113.42
P99 TTFT (ms): 713.87
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 25.13
Median TPOT (ms): 26.02
P99 TPOT (ms): 30.90
---------------Inter-Token Latency----------------
Mean ITL (ms): 24.92
Median ITL (ms): 16.68
P95 ITL (ms): 93.33
P99 ITL (ms): 119.26
Max ITL (ms): 548.82
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 GSM8K Benchmark
- **Benchmark Command:**
```shell Command
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
```
##### AMD (MI300X/MI325X/MI355X)
- **Results**:
- Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8
```
Accuracy: 0.965
Invalid: 0.000
Latency: 23.084 s
Output throughput: 1148.425 token/s
```
##### NVIDIA (B200/GB200)
For deployment commands, see [Section 3.1](#3-1-configuration).
- Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8
```
Accuracy: 0.965
Invalid: 0.000
Latency: 14.870 s
Output throughput: 1777.726 token/s
```
- nvidia/Qwen3-Coder-480B-A35B-Instruct-NVFP (NVFP4)
```
Accuracy: 0.960
Invalid: 0.000
Latency: 13.948 s
Output throughput: 1988.548 token/s
```
docs/cookbook/autoregressive/Qwen/Qwen3-Next.mdx
+794
View File
@@ -0,0 +1,794 @@
---
title: Qwen3-Next
metatags:
description: "Deploy Qwen3-Next with SGLang - hybrid attention architecture supporting 262K context, 80B MoE with 3B active parameters, and multi-token prediction."
---
import { Qwen3NextDeployment } from '/src/snippets/autoregressive/qwen3-next-deployment.jsx';
## 1. Model Introduction
[Qwen3-Next](https://huggingface.co/collections/Qwen/qwen3-next) is an advanced large language model architecture developed by Alibaba's Qwen team, designed to enhance efficiency and performance in handling extensive contexts and large-scale parameters. It features advanced capabilities in reasoning, function calling, and multilingual understanding.
Qwen3-Next introduces several groundbreaking innovations:
- **Hybrid Attention Mechanism**: Replaces standard attention with a combination of **Gated DeltaNet** (linear attention) and **Full Attention**, enabling efficient processing of context lengths up to 262,144 tokens. This hybrid approach makes it ideal for analyzing lengthy documents such as entire books or contracts.
- **Highly Sparse Mixture-of-Experts (MoE)**: Features an 80-billion parameter architecture where only 3 billion parameters are active during inference. This design reduces computational costs by up to 90% while maintaining high performance, drastically reducing FLOPs per token without compromising model capacity.
- **Multi-Token Prediction (MTP)**: Enables generation of multiple tokens per inference step, significantly reducing latency and enhancing user experience in real-time applications. This innovation boosts both pretraining performance and inference speed.
- **Multilingual Support**: Natively supports 119 languages, facilitating seamless cross-lingual tasks and making it versatile for global applications.
- **Enterprise-Ready Deployment**: Released under the Apache 2.0 license, offering flexible deployment options including on-premises, virtual private cloud (VPC), and private cloud environments, ensuring security and compliance for enterprise use.
- **Advanced Reasoning & Stability**: Demonstrates clear improvement in reasoning performance with support for tool use during inference. Includes stability optimizations such as **zero-centered** and **weight-decayed layernorm** for robust pre-training and post-training.
For more details, please refer to the [official Qwen3-Next blog](https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd&from=research.latest-advancements-list).
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
The Qwen3-Next series comes in only one size but offers different thinking modes. Recommended starting configurations vary depending on hardware.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.
<Qwen3NextDeployment />
### 3.2 Configuration Tips
- `--max-mamba-cache-size`: Adjust `--max-mamba-cache-size` to increase mamba cache space and max running requests capability. It will decrease KV cache space as a trade-off. You can adjust it according to workload.
- `--mamba-ssm-dtype`: `bfloat16` or `float32`, use `bfloat16` to save mamba cache size and `float32` to get more accurate results. The default setting is `float32`.
- `--mamba-full-memory-ratio`: Adjust `--mamba-full-memory-ratio` to set the ratio of mamba state memory to full kv cache memory. The default setting is `0.9`.
- **Mamba Radix Cache**: Qwen3-Next's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`:
- **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage.
- **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64).
- **Xeon CPU service configuration**: Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
1. **Streaming with Thinking Process:**
Qwen3-Next-80B-A3B-Thinking only supports thinking mode. Enable the reasoning parser during deployment to separate the thinking and the content sections.
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-Next-80B-A3B-Thinking \
--reasoning-parser qwen3 \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="Qwen/Qwen3-Next-80B-A3B-Thinking",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
Okay, let's see. I need to find 15% of 240. Hmm, percentages. Right, "percent" means per hundred, so 15% is 15 per 100, or 15/100. To find a percentage of a number, I think you multiply the number by the percentage as a decimal. So first, maybe convert 15% to a decimal. To convert a percentage to a decimal, you divide by 100. So 15 divided by 100 is 0.15. Then, multiply that by 240. Let me check that. So 0.15 times 240. Let's calculate that. Maybe break it down. 10% of 240 is 24, because 10% is just moving the decimal one place left, so 240 becomes 24. Then 5% would be half of 10%, so half of 24 is 12. So 10% + 5% = 15%, so 24 + 12 = 36. Oh, that's another way to do it. Let me verify with the multiplication. 0.15 * 240. Let's do 240 * 0.1 = 24, 240 * 0.05 = 12, so 24 + 12 = 36. Yep, that works. Alternatively, 240 * 15 = 3600, then divide by 100, which is 36. Because 15% of 240 is (15/100)*240 = (15*240)/100. 15*240: 10*240=2400, 5*240=1200, so 2400+1200=3600. Then 3600/100=36. So that's 36. So the answer should be 36. Let me make sure. 15% of 240. If I take 240 and multiply by 0.15, 240*0.15. Let's compute 240*0.1=24, 240*0.05=12, so 24+12=36. Yep, that's right. So 15% of 240 is 36.
=============== Content =================
To find **15% of 240**, follow these steps:
---
### **Step 1: Understand what "percent" means**
- "Percent" means **per hundred**, so **15% = 15/100 = 0.15** in decimal form.
---
### **Step 2: Multiply the number by the decimal**
- To find 15% of 240, multiply:
$$
240 \times 0.15
$$
---
### **Step 3: Break it down for clarity (optional but helpful)**
- **10% of 240** = $ 240 \times 0.1 = 24 $
- **5% of 240** = $ 240 \times 0.05 = 12 $
- Add them together:
$$
24 + 12 = 36
$$
---
### **Step 4: Confirm with direct multiplication**
- $ 240 \times 0.15 = 36 $
---
### ✅ Final Answer:
$$
\boxed{36}
$$
```
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
2. **Turn off Thinking:**
Qwen3-Next-80B-A3B-Instruct only supports instruct (non-thinking) mode.
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-Next-80B-A3B-Instruct \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Turn off thinking process
response = client.chat.completions.create(
model="Qwen/Qwen3-Next-80B-A3B-Instruct",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
stream=True,
extra_body={"chat_template_kwargs": {"enable_thinking": False}}
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
To find **15% of 240**, follow these steps:
---
### **Step 1: Understand what percentage means**
"Percent" means "per hundred," so **15%** is the same as **15 per 100**, or the fraction:
$$
\frac{15}{100}
$$
---
### **Step 2: Multiply the fraction by the number**
To find 15% of 240, multiply:
$$
\frac{15}{100} \times 240
$$
---
### **Step 3: Simplify the multiplication**
You can simplify this in a couple of ways.
#### **Option A: Multiply first, then divide**
$$
15 \times 240 = 3600
$$
Then divide by 100:
$$
\frac{3600}{100} = 36
$$
#### **Option B: Simplify the fraction first**
$$
\frac{15}{100} = \frac{3}{20} \quad \text{(divided numerator and denominator by 5)}
$$
Now multiply:
$$
\frac{3}{20} \times 240 = \frac{3 \times 240}{20} = \frac{720}{20} = 36
$$
---
### **Step 4: Final Answer**
$$
\boxed{36}
$$
So, **15% of 240 is 36**.
```
#### 4.2.2 Tool Calling
Qwen/Qwen3-Next-80B-A3B-Instruct | Qwen/Qwen3-Next-80B-A3B-Thinking both support tool calling capabilities. Enable the tool call parser:
**Python Example (without Thinking Process):**
Start sglang server:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-Next-80B-A3B-Instruct \
--tool-call-parser qwen \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="Qwen/Qwen3-Next-80B-A3B-Instruct",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
if tool_call.function:
print(f"🔧 Tool Call: {tool_call.function.name}")
print(f" Arguments: {tool_call.function.arguments}")
# Print content
if delta.content:
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
<tool_call>
{"name": "get_weather", "arguments": {"location": "Beijing"}}
</tool_call>
```
**Python Example (with Thinking Process):**
Start sglang server:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-Next-80B-A3B-Thinking \
--reasoning-parser qwen3 \
--tool-call-parser qwen \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="Qwen/Qwen3-Next-80B-A3B-Thinking",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
if tool_call.function:
print(f"🔧 Tool Call: {tool_call.function.name}")
print(f" Arguments: {tool_call.function.arguments}")
# Print content
if delta.content:
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
Okay, the user is asking for the weather in Beijing. Let me check the available tools. There's a get_weather function that requires location and optionally unit. The location is needed, so I need to provide Beijing as the location. The unit is optional, but the user didn't specify Celsius or Fahrenheit. Since the default might be Celsius, but maybe I should check if the parameters require unit. Wait, the required field is only location, so unit is optional. So I can just call get_weather with location "Beijing" and not include the unit. Let me confirm the parameters. The parameters for get_weather have location as required, and unit is an enum with celsius or fahrenheit, but not required. So the correct call is to send location as Beijing, and omit unit. So the tool call should be {"name": "get_weather", "arguments": {"location": "Beijing"}}.
<tool_call>
{"name": "get_weather", "arguments": {"location": "Beijing"}}
</tool_call>
```
**Note:**
- The reasoning parser shows how the model decides to use a tool
- Tool calls are clearly marked with the function name and arguments
- You can then execute the function and send the result back to continue the conversation
**Handling Tool Call Results:**
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="Qwen/Qwen3-Next-80B-A3B-Thinking",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```
#### 4.2.3 Processing Ultra-Long Texts
Qwen3-Next natively supports context lengths of up to 262,144 tokens. For conversations where the total length (including both input and output) significantly exceeds this limit, we recommend using RoPE scaling techniques to handle long texts effectively. We have validated the model's performance on context lengths of up to 1 million tokens using the YaRN method.
**Qwen3-Next-80B-A3B-Instruct**
```shell Command
SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python -m sglang.launch_server --model Qwen/Qwen3-Next-80B-A3B-Instruct --tp 8 --host 0.0.0.0 --port 8000 --json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144}}' --context-length 1010000
```
**Qwen3-Next-80B-A3B-Thinking**
```shell Command
SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python -m sglang.launch_server --model Qwen/Qwen3-Next-80B-A3B-Thinking --reasoning-parser qwen3 --tp 8 --host 0.0.0.0 --port 8000 --json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144}}' --context-length 1010000
```
#### 4.2.4 Multi-Token Prediction (NEXTN Speculative Decoding)
Qwen3-Next ships built-in Multi-Token Prediction (MTP) layers and supports [EAGLE-style speculative decoding](../../../docs/advanced_features/speculative_decoding#eagle-decoding) through the `NEXTN` algorithm. The MTP weights are bundled in the main checkpoint, so no separate draft model is required.
```shell Command
python3 -m sglang.launch_server \
--model Qwen/Qwen3-Next-80B-A3B-Instruct \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--tp 4
```
Tune `--speculative-num-steps`, `--speculative-eagle-topk`, and `--speculative-num-draft-tokens` for your workload with [bench_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py). See [PR #10233](https://github.com/sgl-project/sglang/pull/10233) for implementation details.
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: NVIDIA B200 GPU (8x)
- Tensor Parallelism: 8
- Model: Qwen/Qwen3-Next-80B-A3B-Instruct
- sglang version: 0.5.6
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.
#### 5.1.1 Latency-Sensitive Benchmark
- Server Command:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-Next-80B-A3B-Instruct \
--tp 8
```
- Test Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--num-prompt 100 \
--max-concurrency 1
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 100
Benchmark duration (s): 146.52
Total input tokens: 33839
Total input text tokens: 33839
Total input vision tokens: 0
Total generated tokens: 21640
Total generated tokens (retokenized): 21619
Request throughput (req/s): 0.68
Input token throughput (tok/s): 230.95
Output token throughput (tok/s): 147.70
Peak output token throughput (tok/s): 164.00
Peak concurrent requests: 6
Total token throughput (tok/s): 378.65
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 1464.81
Median E2E Latency (ms): 1077.48
---------------Time to First Token----------------
Mean TTFT (ms): 127.88
Median TTFT (ms): 132.88
P99 TTFT (ms): 212.85
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 6.19
Median TPOT (ms): 6.17
P99 TPOT (ms): 6.64
---------------Inter-Token Latency----------------
Mean ITL (ms): 6.21
Median ITL (ms): 6.16
P95 ITL (ms): 6.51
P99 ITL (ms): 6.71
Max ITL (ms): 10.07
==================================================
```
#### 5.1.2 Throughput-Sensitive Benchmark
- Server Command:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-Next-80B-A3B-Instruct \
--tp 8 \
```
- Test Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--num-prompt 1000 \
--max-concurrency 100
```
**Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 100.32
Total input tokens: 302118
Total input text tokens: 302118
Total input vision tokens: 0
Total generated tokens: 195775
Total generated tokens (retokenized): 195016
Request throughput (req/s): 9.97
Input token throughput (tok/s): 3011.69
Output token throughput (tok/s): 1951.60
Peak output token throughput (tok/s): 5909.00
Peak concurrent requests: 120
Total token throughput (tok/s): 4963.29
Concurrency: 93.05
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 9333.98
Median E2E Latency (ms): 6054.12
---------------Time to First Token----------------
Mean TTFT (ms): 161.77
Median TTFT (ms): 137.94
P99 TTFT (ms): 503.29
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 50.87
Median TPOT (ms): 50.28
P99 TPOT (ms): 122.87
---------------Inter-Token Latency----------------
Mean ITL (ms): 47.11
Median ITL (ms): 13.84
P95 ITL (ms): 195.33
P99 ITL (ms): 289.56
Max ITL (ms): 486.38
==================================================
```
### 5.2 Accuracy Benchmark
### 5.2.1 GSM8K Benchmark
- **Benchmark Command:**
```shell Command
python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000
```
- **Results**:
- Qwen3-Next-80B-A3B-Instruct
```
Accuracy: 0.960
Invalid: 0.000
Latency: 12.673 s
Output throughput: 2538.255 token/s
```
- Qwen3-Next-80B-A3B-Thinking
```
Accuracy: 0.935
Invalid: 0.000
Latency: 9.912 s
Output throughput: 3288.737 token/s
```
### 5.2.2 MMLU Benchmark
- **Benchmark Command:**
```shell Command
cd sglang
bash benchmark/mmlu/download_data.sh
python3 benchmark/mmlu/bench_sglang.py --nsub 10
```
- **Results**:
- Qwen3-Next-80B-A3B-Instruct
```
subject: abstract_algebra, #q:100, acc: 0.800
subject: anatomy, #q:135, acc: 0.807
subject: astronomy, #q:152, acc: 0.947
subject: business_ethics, #q:100, acc: 0.810
subject: clinical_knowledge, #q:265, acc: 0.894
subject: college_biology, #q:144, acc: 0.972
subject: college_chemistry, #q:100, acc: 0.680
subject: college_computer_science, #q:100, acc: 0.860
subject: college_mathematics, #q:100, acc: 0.780
subject: college_medicine, #q:173, acc: 0.861
Total latency: 10.098
Average accuracy: 0.856
```
- Qwen3-Next-80B-A3B-Thinking
```
subject: abstract_algebra, #q:100, acc: 0.780
subject: anatomy, #q:135, acc: 0.815
subject: astronomy, #q:152, acc: 0.941
subject: business_ethics, #q:100, acc: 0.870
subject: clinical_knowledge, #q:265, acc: 0.894
subject: college_biology, #q:144, acc: 0.965
subject: college_chemistry, #q:100, acc: 0.670
subject: college_computer_science, #q:100, acc: 0.840
subject: college_mathematics, #q:100, acc: 0.770
subject: college_medicine, #q:173, acc: 0.861
Total latency: 10.236
Average accuracy: 0.855
```
docs/cookbook/autoregressive/Qwen/Qwen3-VL.mdx
+809
View File
@@ -0,0 +1,809 @@
---
title: Qwen3-VL
metatags:
description: "Deploy Qwen3-VL vision-language models with SGLang - open model for text, 262K context, enhanced visual reasoning and agent capabilities."
---
## 1. Model Introduction
[Qwen3-VL series](https://github.com/QwenLM/Qwen3-VL) are the most powerful vision-language models in the Qwen series to date, featuring advanced capabilities in multi-modal understanding, reasoning, and agentic applications.
This generation delivers comprehensive upgrades across the board:
- **Superior text understanding & generation**: Qwen3-VL-235B-A22B-Instruct was ranked as the [#1 open model for text on lmarena.ai](https://x.com/arena/status/1973151703563460942)
- **Deeper visual perception & reasoning**: Enhanced image and video understanding capabilities.
- **Extended context length**: Supports up to 262K tokens for processing long documents and videos.
- **Enhanced spatial and video dynamics comprehension**: Better understanding of spatial relationships and temporal dynamics.
- **Stronger agent interaction capabilities**: Improved tool use and search-based agent performance.
- **Flexible deployment options**: Available in Dense and MoE architectures that scale from edge to cloud, with Instruct and reasoning-enhanced Thinking editions.
For more details, please refer to the [official Qwen3-VL GitHub Repository](https://github.com/QwenLM/Qwen3-VL).
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
The Qwen3-VL series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA and AMD GPUs, as well as Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.
import { Qwen3VLDeployment } from "/src/snippets/autoregressive/qwen3-vl-deployment.jsx";
<Qwen3VLDeployment />
### 3.2 Configuration Tips
* **Multimodal attention backend** : Usually, `--mm-attention-backend` is default to `fa3` on H100/H200/A100 for better performance, but it is default to `triton_attn` on B200 for compatibility.
* **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size * number of images in current running requests.)
* **Memory Management** : Set lower `--context-length` to conserve memory. A value of `128000` is sufficient for most scenarios, down from the default 262K.
* **Expert Parallelism** : SGLang supports Expert Parallelism (EP) via `--ep`, allowing experts in MoE models to be deployed on separate GPUs for better throughput. One thing to note is that, for quantized models, you need to set `--ep` to a value that satisfies the requirement: `(moe_intermediate_size / moe_tp_size) % weight_block_size_n == 0, where moe_tp_size is equal to tp_size divided by ep_size.` Note that EP may perform worse in low concurrency scenarios due to additional communication overhead. Check out [Expert Parallelism Deployment](../../../docs/advanced_features/expert_parallelism) for more details.
* **Kernel Tuning** : For MoE Triton kernel tuning on your specific hardware, refer to [fused_moe_triton](https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton).
**Hardware-specific notes:**
- **H100 (FP8):** Use the `Qwen/Qwen3-VL-235B-A22B-Instruct-FP8` checkpoint for best memory efficiency.
- **A100 / H100 (BF16):** Use standard multimodal parameters to manage throughput and GPU memory usage.
- **H200 / B200:** Runs out of the box, supporting full context length plus concurrent image + video processing.
**Additional multimodal server parameters:**
- `--keep-mm-feature-on-device`: Retain multimodal feature tensors on GPU after processing to avoid device-to-host memory copies, improving performance for high-frequency inference.
**Example with full multimodal optimizations:**
```bash Command
SGLANG_USE_CUDA_IPC_TRANSPORT=1 \
SGLANG_VLM_CACHE_SIZE_MB=0 \
python -m sglang.launch_server \
--model-path Qwen/Qwen3-VL-235B-A22B-Instruct \
--host 0.0.0.0 \
--port 30000 \
--trust-remote-code \
--tp-size 8 \
--enable-cache-report \
--log-level info \
--max-running-requests 64 \
--mem-fraction-static 0.65 \
--chunked-prefill-size 8192 \
--attention-backend fa3 \
--mm-attention-backend fa3 \
--enable-metrics
```
* **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
### 4.2 Advanced Usage
#### 4.2.1 Multi-Modal Inputs
Qwen3-VL supports both image and video inputs. Here's a basic example with image input:
```python Example
import time
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:30000/v1",
timeout=3600
)
messages = [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
}
},
{
"type": "text",
"text": "Read all the text in the image."
}
]
}
]
start = time.time()
response = client.chat.completions.create(
model="Qwen/Qwen3-VL-235B-A22B-Instruct",
messages=messages,
max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```
**Example Output:**
```text Output
Response costs: 3.37s
Generated text: Auntie Anne's
CINNAMON SUGAR
1 x 17,000 17,000
SUB TOTAL 17,000
GRAND TOTAL 17,000
CASH IDR 20,000
CHANGE DUE 3,000
```
**Multi-Image Input Example:**
Qwen3-VL can process multiple images in a single request for comparison or analysis:
```python Example
import time
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:30000/v1",
timeout=3600
)
messages = [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
}
},
{
"type": "image_url",
"image_url": {
"url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
}
},
{
"type": "text",
"text": "Compare these two images and describe the differences in 100 words or less. Focus on the key visual elements, colors, textures, and any notable contrasts between the two scenes. Be specific about what you see in each image."
}
]
}
]
start = time.time()
response = client.chat.completions.create(
model="Qwen/Qwen3-VL-235B-A22B-Instruct",
messages=messages,
max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```
**Example Output:**
```text Output
Response costs: 10.18s
Generated text: The two images present starkly different portrayals of Hong Kong’s iconic red taxis, contrasting a dynamic street-level moment with a static, large-scale gathering.
The first image is a close-up, eye-level shot capturing a single red Toyota Crown taxi (license plate RX 5004) in motion or paused at an urban intersection. Its glossy red paint gleams under daylight, reflecting the vibrant, cluttered backdrop of a Hong Kong street — neon signs, glass-fronted shops displaying sunglasses, and Chinese characters. The taxi’s chrome grille, clear headlights, and black trim provide visual contrast. A green “4 SEATS” sticker and a “的士 TAXI” sign on the side reinforce its identity. The composition is intimate, focusing on the vehicle’s details — the texture of its paint, the slight reflections on the windows, and the crispness of its license plate. Other red taxis flank it, suggesting a bustling city rhythm, but the central taxi dominates the frame, conveying movement and immediacy.
In contrast, the second image is an elevated, wide-angle shot of dozens of red taxis — along with a few green ones — parked in neat, grid-like rows on what appears to be a highway or staging area. The scene is static, almost ceremonial. Many taxis have their hoods open, suggesting maintenance, inspection, or protest. People are scattered among the vehicles, some inspecting engines, others conversing — adding a human, documentary element. The dominant color remains red, but the repetition creates a visual pattern rather than individual focus. The green taxis offer a subtle color contrast, hinting at different service zones (green for New Territories, red for urban areas). The setting is more utilitarian — concrete barriers, metal railings, and sparse vegetation — with an overpass looming in the background. The texture here is less about polished paint and more about the collective mass of vehicles, the asphalt, and the functional layout.
Key contrasts emerge: the first image is kinetic and personal, emphasizing the taxi as a working vehicle in the city’s daily flow; the second is static and collective, portraying the taxis as a fleet, possibly for logistical or political purposes. The lighting in both is bright daylight, but the first has richer color saturation and depth due to its proximity and urban backdrop, while the second feels flatter, more documentary in tone. The first image invites you into the city’s pulse; the second invites you to observe a system — organized, perhaps even paused — from a distance.
In essence, the first image celebrates the individual taxi in its natural habitat; the second reveals the scale and structure behind the fleet, transforming the familiar red icon into a symbol of coordination, maintenance, or collective action. Both are quintessentially Hong Kong, yet they offer vastly different narratives — one of motion and commerce, the other of assembly and purpose.
```
**Video Input Example:**
Qwen3-VL supports video understanding by processing video URLs:
```python Example
import time
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:30000/v1",
timeout=3600
)
messages = [
{
"role": "user",
"content": [
{
"type": "video_url",
"video_url": {
"url": "https://videos.pexels.com/video-files/4114797/4114797-uhd_3840_2160_25fps.mp4"
}
},
{
"type": "text",
"text": "Describe what happens in this video."
}
]
}
]
start = time.time()
response = client.chat.completions.create(
model="Qwen/Qwen3-VL-235B-A22B-Instruct",
messages=messages,
max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```
**Note:**
- For video processing, ensure you have sufficient context length configured (up to 262K tokens)
- Video processing may require more memory; adjust `--mem-fraction-static` accordingly
- You can also provide local file paths using `file://` protocol
**Example Output:**
```text Output
Response costs: 3.89s
Generated text: A person wearing blue gloves is using a microscope. They are adjusting the focus knob with one hand while holding a pipette with the other, suggesting they are preparing or examining a sample on the slide beneath the objective lens. The microscope's 40x objective lens is positioned over the slide, indicating a high-magnification observation. The person carefully manipulates the slide and the microscope controls, likely to achieve a clear view of the specimen.
```
#### 4.2.2 Reasoning Parser
Qwen3-VL-Thinking supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-VL-235B-A22B-Thinking \
--reasoning-parser qwen3 \
--tp 8 \
--host 0.0.0.0 \
--port 30000
```
**Streaming with Thinking Process:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="Qwen/Qwen3-VL-235B-A22B-Thinking",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
To solve this problem, I need to calculate 15% of 240.
Step 1: Convert 15% to decimal: 15% = 0.15
Step 2: Multiply 240 by 0.15
Step 3: 240 × 0.15 = 36
=============== Content =================
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
```
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
#### 4.2.3 Tool Calling
Qwen3-VL supports tool calling capabilities. Enable the tool call parser:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-VL-235B-A22B-Thinking \
--reasoning-parser qwen3 \
--tool-call-parser qwen \
--tp 8 \
--host 0.0.0.0 \
--port 30000
```
**Python Example (with Thinking Process):**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="Qwen/Qwen3-VL-235B-A22B-Thinking",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Accumulate tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================\n", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {
'name': None,
'arguments': ''
}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]['name'] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
# Print content
if delta.content:
print(delta.content, end="", flush=True)
# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"🔧 Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
print()
```
**Output Example:**
```text Output
=============== Thinking =================
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
I should call the function with location="Beijing".
=============== Content =================
🔧 Tool Call: get_weather
Arguments: {"location": "Beijing", "unit": "celsius"}
```
**Note:**
- The reasoning parser shows how the model decides to use a tool
- Tool calls are clearly marked with the function name and arguments
- You can then execute the function and send the result back to continue the conversation
**Handling Tool Call Results:**
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="Qwen/Qwen3-VL-235B-A22B-Thinking",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: NVIDIA B200 GPU (8x)
- Model: Qwen3-VL-235B-A22B-Instruct
- Tensor Parallelism: 8
- sglang version: 0.5.6
We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. To simulate real-world usage, you can specify different input and output lengths for each request. For example, each request can have 128 input tokens, two 720p images, and 1024 output tokens.
#### 5.1.1 Latency-Sensitive Benchmark
- Model Deployment Command:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
--tp 8 \
--host 0.0.0.0 \
--port 30000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang-oai-chat \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
--dataset-name image \
--image-count 2 \
--image-resolution 720p \
--random-input-len 128 \
--random-output-len 1024 \
--num-prompts 10 \
--max-concurrency 1
```
- **Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 45.97
Total input tokens: 18348
Total input text tokens: 708
Total input vision tokens: 17640
Total generated tokens: 4220
Total generated tokens (retokenized): 3423
Request throughput (req/s): 0.22
Input token throughput (tok/s): 399.17
Output token throughput (tok/s): 91.81
Peak output token throughput (tok/s): 96.00
Peak concurrent requests: 2
Total token throughput (tok/s): 490.98
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 4594.52
Median E2E Latency (ms): 3725.04
---------------Time to First Token----------------
Mean TTFT (ms): 193.35
Median TTFT (ms): 196.32
P99 TTFT (ms): 222.75
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 10.44
Median TPOT (ms): 10.44
P99 TPOT (ms): 10.47
---------------Inter-Token Latency----------------
Mean ITL (ms): 11.78
Median ITL (ms): 10.48
P95 ITL (ms): 21.01
P99 ITL (ms): 31.40
Max ITL (ms): 31.92
==================================================
```
**Optimized Results (with CUDA IPC Transport):**
For further TTFT optimization, enable CUDA IPC Transport for multimodal features by setting `SGLANG_USE_CUDA_IPC_TRANSPORT=1`. This significantly reduces TTFT by using CUDA IPC for transferring multimodal features.
- Model Deployment Command:
```shell Command
SGLANG_USE_CUDA_IPC_TRANSPORT=1 python -m sglang.launch_server \
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
--tp 8 \
--host 0.0.0.0 \
--port 30000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang-oai-chat \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
--dataset-name image \
--image-count 2 \
--image-resolution 720p \
--random-input-len 128 \
--random-output-len 1024 \
--num-prompts 100 \
--max-concurrency 1
```
- **Test Results:**
With `SGLANG_USE_CUDA_IPC_TRANSPORT=1`, TTFT improves significantly:
```text Output
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 100
Benchmark duration (s): 566.84
Total input tokens: 183667
Total input text tokens: 7267
Total input vision tokens: 176400
Total generated tokens: 52444
Total generated tokens (retokenized): 28702
Request throughput (req/s): 0.18
Input token throughput (tok/s): 324.02
Output token throughput (tok/s): 92.52
Peak output token throughput (tok/s): 96.00
Peak concurrent requests: 3
Total token throughput (tok/s): 416.54
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 5667.50
Median E2E Latency (ms): 5830.00
---------------Time to First Token----------------
Mean TTFT (ms): 191.16
Median TTFT (ms): 182.58
P99 TTFT (ms): 244.58
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 10.46
Median TPOT (ms): 10.46
P99 TPOT (ms): 10.48
---------------Inter-Token Latency----------------
Mean ITL (ms): 13.91
Median ITL (ms): 10.56
P95 ITL (ms): 21.35
P99 ITL (ms): 31.55
Max ITL (ms): 42.36
==================================================
```
#### 5.1.2 Throughput-Sensitive Benchmark
- Model Deployment Command:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
--tp 8 \
--host 0.0.0.0 \
--port 30000
```
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang-oai-chat \
--host 127.0.0.1 \
--port 30000 \
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
--dataset-name image \
--image-count 2 \
--image-resolution 720p \
--random-input-len 128 \
--random-output-len 1024 \
--num-prompts 1000 \
--max-concurrency 100
```
- **Test Results:**
```text Output
============ Serving Benchmark Result ============
Backend: sglang-oai-chat
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 1000
Benchmark duration (s): 584.65
Total input tokens: 1839015
Total input text tokens: 75015
Total input vision tokens: 1764000
Total generated tokens: 510855
Total generated tokens (retokenized): 284284
Request throughput (req/s): 1.71
Input token throughput (tok/s): 3145.50
Output token throughput (tok/s): 873.78
Peak output token throughput (tok/s): 2855.00
Peak concurrent requests: 107
Total token throughput (tok/s): 4019.29
Concurrency: 98.35
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 57502.05
Median E2E Latency (ms): 54301.08
---------------Time to First Token----------------
Mean TTFT (ms): 5802.23
Median TTFT (ms): 1444.75
P99 TTFT (ms): 46675.92
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 100.22
Median TPOT (ms): 105.43
P99 TPOT (ms): 144.37
---------------Inter-Token Latency----------------
Mean ITL (ms): 134.20
Median ITL (ms): 25.57
P95 ITL (ms): 558.14
P99 ITL (ms): 1449.01
Max ITL (ms): 33453.23
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 MMMU Benchmark
You can evaluate the model's accuracy using the MMMU dataset with `lmms_eval`:
- Benchmark Command:
```shell Command
uv pip install lmms_eval
python3 -m lmms_eval \
--model openai_compatible \
--model_args "model=Qwen/Qwen3-VL-235B-A22B-Instruct,api_key=EMPTY,base_url=http://127.0.0.1:30000/v1/" \
--tasks mmmu_val \
--batch_size 128 \
--log_samples \
--log_samples_suffix "openai_compatible" \
--output_path ./logs \
--gen_kwargs "max_new_tokens=4096"
```
- **Test Results:**
```text Output
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "12%"}} />
<col style={{width: "11%"}} />
<col style={{width: "11%"}} />
<col style={{width: "11%"}} />
<col style={{width: "11%"}} />
<col style={{width: "11%"}} />
<col style={{width: "11%"}} />
<col style={{width: "11%"}} />
<col style={{width: "11%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Tasks</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Version</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Filter</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>n-shot</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Metric</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}></th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Value</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}></th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Stderr</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>mmmu_val</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>none</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>mmmu_acc</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>↑</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.6567</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>±</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
</tr>
</tbody>
</table>
```
docs/cookbook/autoregressive/Qwen/Qwen3.5.mdx
+1038
View File
File diff suppressed because it is too large Load Diff
docs/cookbook/autoregressive/Qwen/Qwen3.6.mdx
+518
View File
@@ -0,0 +1,518 @@
---
title: Qwen3.6
metatags:
description: "Deploy Qwen3.6 with SGLang - open-weight multimodal series with a 35B MoE (3B active) variant and a 27B dense variant, hybrid reasoning, tool calling, MTP, and long-context support."
tag: NEW
---
import { Qwen36Deployment } from '/src/snippets/autoregressive/qwen36-deployment.jsx';
## 1. Model Introduction
The Qwen3.6 series is developed by Alibaba. Built on direct feedback from the community, Qwen3.6 prioritizes stability and real-world utility, delivering substantial upgrades in agentic coding and thinking preservation. Two size/sparsity variants are released:
- [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) — **Sparse MoE** (35B total, 3B active) on a Gated Delta Networks backbone.
- [Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) — **Dense** hybrid GDN; smaller weights footprint, single-GPU friendly.
Both variants share the same hybrid reasoning, tool-calling, and multimodal interface and natively handle context lengths of up to 262,144 tokens, extensible to over 1M tokens.
**Key Features:**
- **Agentic Coding**: Handles frontend workflows and repository-level reasoning with greater fluency and precision
- **Thinking Preservation**: New option to retain reasoning context from historical messages, streamlining iterative development
- **Efficient Hybrid Architecture**: Gated Delta Networks backbone; sparse MoE (35B / 3B active) or dense 27B variant
- **Hybrid Reasoning**: Thinking mode enabled by default with step-by-step reasoning, can be disabled for direct responses
- **Tool Calling**: Built-in tool calling support with `qwen3_coder` parser
- **Multi-Token Prediction (MTP)**: Speculative decoding support for lower latency; both MoE and Dense variants ship `mtp.safetensors`
- **Multimodal**: Unified vision-language model supporting text, image, and video inputs
**Available Models:**
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<thead>
<tr>
<th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>Model</th>
<th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>Architecture</th>
<th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>Weights</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3.6-35B-A3B (BF16)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>MoE 35B / 3B active</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3.6-35B-A3B (FP8)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MoE 35B / 3B active</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[Qwen/Qwen3.6-35B-A3B-FP8](https://huggingface.co/Qwen/Qwen3.6-35B-A3B-FP8)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3.6-35B-A3B (NVFP4)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>MoE 35B / 3B active (Blackwell)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[nvidia/Qwen3.6-35B-A3B-NVFP4](https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3.6-27B (BF16)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Dense 27B</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3.6-27B (FP8)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Dense 27B</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[Qwen/Qwen3.6-27B-FP8](https://huggingface.co/Qwen/Qwen3.6-27B-FP8)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3.6-27B (NVFP4)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Dense 27B (Blackwell)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[nvidia/Qwen3.6-27B-NVFP4](https://huggingface.co/nvidia/Qwen3.6-27B-NVFP4)</td>
</tr>
</tbody>
</table>
**License:** Apache 2.0
## 2. SGLang Installation
SGLang `>=0.5.10` is required for Qwen3.6. You can install from PyPI, from source, or use a Docker image:
```bash Command
# Install from PyPI
uv pip install sglang
# Or install from source
uv pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
# Or use Docker (NVIDIA GPUs; also serves the NVFP4 variants)
docker pull lmsysorg/sglang:latest
```
For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and capabilities.
<Qwen36Deployment />
### 3.2 Configuration Tips
- Speculative decoding (MTP) can significantly reduce latency for interactive use cases.
- **Mamba Radix Cache**: Qwen3.6's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`:
- **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage.
- **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend (NVIDIA GPUs only). Trades higher mamba state memory for better throughput.
- The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload.
- Context length defaults to 262,144 tokens. If you encounter OOM errors, consider reducing it, but maintain at least 128K to preserve thinking capabilities.
- **CUDA IPC Transport**: Add `SGLANG_USE_CUDA_IPC_TRANSPORT=1` as an environment variable to use CUDA IPC for transferring multimodal features, significantly improving TTFT (Time To First Token). Note: this consumes additional memory proportional to image size, so you may need to lower `--mem-fraction-static` or `--max-running-requests`.
- **Multimodal Attention Backend**: Use `--mm-attention-backend fa3` on H100/H200 for better vision performance, or `--mm-attention-backend fa4` on B200/B300.
- For processing large images or videos, you may need to lower `--mem-fraction-static` to leave room for image feature tensors.
- Hardware requirements:
- **35B-A3B BF16**: ~70GB for weights. TP=1 fits on all supported hardware.
- **35B-A3B FP8**: ~35GB for weights. TP=1 fits on all supported hardware.
- **35B-A3B NVFP4**: ~23GB for weights. TP=1 fits on B200/B300.
- **27B BF16**: ~54GB for weights. TP=1 fits on all supported hardware.
- **27B FP8**: ~27GB for weights. TP=1 fits on all supported hardware.
- **27B NVFP4**: ~22GB for weights. TP=1 fits on B200/B300.
All Qwen3.6 variants (MoE 35B-A3B and Dense 27B) fit on a single supported GPU. NVFP4 is available on B200/B300:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<thead>
<tr>
<th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>Hardware</th>
<th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>Memory</th>
<th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>BF16 TP</th>
<th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>FP8 TP</th>
<th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>NVFP4 TP</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H100</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>80GB</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.05)"}}>H200</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>141GB</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>B200</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>183GB</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.05)"}}>B300</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>275GB</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
</tr>
</tbody>
</table>
- **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation
Deploy Qwen3.6 with the following command (H200, all features enabled). Swap `--model-path` to `Qwen/Qwen3.6-27B-FP8` for the dense 27B variant — all other flags carry over:
```shell Command
sglang serve \
--model-path Qwen/Qwen3.6-35B-A3B-FP8 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--mem-fraction-static 0.8 \
--host 0.0.0.0 \
--port 30000
```
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
### 4.2 Vision Input
Qwen3.6 supports image and video inputs as a unified vision-language model.
**Image Input Example:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="Qwen/Qwen3.6-35B-A3B-FP8",
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/CI_Demo/mathv-1327.jpg"
}
},
{
"type": "text",
"text": "Describe this image in detail."
}
]
}
],
max_tokens=2048,
stream=True
)
thinking_started = False
has_thinking = False
has_answer = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
if delta.content:
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Video Input Example:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="Qwen/Qwen3.6-35B-A3B-FP8",
messages=[
{
"role": "user",
"content": [
{
"type": "video_url",
"video_url": {
"url": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/video/N1cdUjctpG8.mp4"
}
},
{
"type": "text",
"text": "Describe what happens in this video."
}
]
}
],
max_tokens=2048,
stream=True
)
thinking_started = False
has_thinking = False
has_answer = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
if delta.content:
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
### 4.3 Advanced Usage
#### 4.3.1 Reasoning Parser
Qwen3.6 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response.
To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time:
- **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed.
- **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process.
**Example 1: Thinking Mode (Default)**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="Qwen/Qwen3.6-35B-A3B-FP8",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
max_tokens=2048,
stream=True
)
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
if delta.content:
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Example 2: Instruct Mode (Thinking Off)**
To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="Qwen/Qwen3.6-35B-A3B-FP8",
messages=[
{"role": "user", "content": "What is 15% of 240?"}
],
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
max_tokens=2048,
stream=True
)
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if delta.content:
print(delta.content, end="", flush=True)
print()
```
#### 4.3.2 Thinking Preservation
Qwen3.6 has been trained to preserve and leverage thinking traces from historical messages. Enable this for agent scenarios where maintaining full reasoning context improves decision consistency:
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
response = client.chat.completions.create(
model="Qwen/Qwen3.6-35B-A3B-FP8",
messages=[
{"role": "user", "content": "Help me plan a web app architecture."}
],
extra_body={"chat_template_kwargs": {"preserve_thinking": True}},
max_tokens=2048,
stream=True
)
thinking_started = False
has_thinking = False
has_answer = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
if delta.content:
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
#### 4.3.3 Tool Calling
Qwen3.6 supports tool calling capabilities. Enable the tool call parser during deployment.
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:30000/v1",
api_key="EMPTY"
)
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
response = client.chat.completions.create(
model="Qwen/Qwen3.6-35B-A3B-FP8",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
stream=True
)
thinking_started = False
has_thinking = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
if hasattr(delta, 'tool_calls') and delta.tool_calls:
if has_thinking and thinking_started:
print("\n=============== Content =================", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
if tool_call.function:
print(f"Tool Call: {tool_call.function.name}")
print(f" Arguments: {tool_call.function.arguments}")
if delta.content:
print(delta.content, end="", flush=True)
print()
```
docs/cookbook/autoregressive/Qwen/Qwen3.mdx
+887
View File
@@ -0,0 +1,887 @@
---
title: Qwen3
metatags:
description: "Deploy Qwen3 series models with SGLang - featuring advanced reasoning, 256K context, and flexible Dense/MoE architectures for edge to cloud."
---
## 1. Model Introduction
[Qwen3 series](https://github.com/QwenLM/Qwen3) are the most powerful vision-language models in the Qwen series to date, featuring advanced capabilities in multi-modal understanding, reasoning, and agentic applications.
This generation delivers comprehensive upgrades across the board:
- **Stronger general intelligence**: Significant improvements in instruction following, logical reasoning, text comprehension, mathematics, science, coding, and tool usage.
- **Broader multilingual knowledge**: Substantial gains in long-tail knowledge coverage across multiple languages.
- **More helpful & aligned responses**: Markedly better alignment with user preferences in subjective and open-ended tasks, enabling higher-quality, more useful text generation.
- **Extended context length**: Enhanced capabilities in understanding and reasoning over 256K-token long contexts.
- **Stronger agent interaction capabilities**: Improved tool use and search-based agent performance.
- **Flexible deployment options**: Available in Dense and MoE architectures that scale from edge to cloud, with Instruct and reasoning-enhanced Thinking editions.
For more details, please refer to the [official Qwen3 GitHub Repository](https://github.com/QwenLM/Qwen3).
## 2. SGLang Installation
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
The Qwen3 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, and Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.
import { Qwen3Deployment } from "/src/snippets/autoregressive/qwen3-deployment.jsx";
<Qwen3Deployment />
### 3.2 Configuration Tips
- **Memory Management:** Set lower `--context-length` to conserve memory. A value of `128000` is sufficient for most scenarios, down from the default 262K.
- **Expert Parallelism:** SGLang supports Expert Parallelism (EP) via `--ep`, allowing experts in MoE models to be deployed on separate GPUs for better throughput. One thing to note is that, for quantized models, you need to set `--ep` to a value that satisfies the requirement: `(moe_intermediate_size / moe_tp_size) % weight_block_size_n == 0, where moe_tp_size is equal to tp_size divided by ep_size.` Note that EP may perform worse in low concurrency scenarios due to additional communication overhead. Check out [Expert Parallelism Deployment](../../../docs/advanced_features/expert_parallelism) for more details.
- **Kernel Tuning:** For MoE Triton kernel tuning on your specific hardware, refer to [fused_moe_triton](https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton).
- **Speculative Decoding:** Using Speculative Decoding for latency-sensitive scenarios.
- `--speculative-algorithm EAGLE3`: Speculative decoding algorithm
- `--speculative-num-steps 3`: Number of speculative verification rounds
- `--speculative-eagle-topk 1`: Top-k sampling for draft tokens
- `--speculative-num-draft-tokens 4`: Number of draft tokens per step
- `--speculative-draft-model-path`: The path of the draft model weights. This can be a local folder or a Hugging Face repo ID such as [`lmsys/SGLang-EAGLE3-Qwen3-235B-A22B-Instruct-2507-SpecForge-Meituan`](https://huggingface.co/lmsys/SGLang-EAGLE3-Qwen3-235B-A22B-Instruct-2507-SpecForge-Meituan).
- **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
## 4. Model Invocation
### 4.1 Basic Usage
For basic API usage and request examples, please refer to:
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
### 4.2 Advanced Usage
#### 4.2.1 Reasoning Parser
Qwen3-235B-A22B supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-235B-A22B-Thinking-2507 \
--reasoning-parser qwen3 \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
**Streaming with Thinking Process:**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
model="Qwen/Qwen3-235B-A22B-Thinking-2507",
messages=[
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
],
temperature=0.7,
max_tokens=2048,
stream=True
)
# Process the stream
has_thinking = False
has_answer = False
thinking_started = False
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Print answer content
if delta.content:
# Close thinking section and add content header
if has_thinking and not has_answer:
print("\n=============== Content =================", flush=True)
has_answer = True
print(delta.content, end="", flush=True)
print()
```
**Output Example:**
```text Output
=============== Thinking =================
Okay, so I need to figure out what 15% of 240 is. Hmm, percentages can sometimes trip me up, but I think I remember some basics. Let me start by recalling that "percent" means "per hundred," so 15% is the same as 15 per 100, or 15/100. So, maybe I can convert 15% into a decimal first? Yeah, I think that's a common method.
...
So conclusion: The answer is 36.
=============== Content =================
To determine what 15% of 240 is, we can follow a systematic approach that involves converting the percentage to a decimal and then performing multiplication. Here's a step-by-step breakdown of the solution:
....
### Final Answer:
$$
\boxed{36}
$$
Thus, 15% of 240 is **36**.
```
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
#### 4.2.3 Tool Calling
Qwen3 supports tool calling capabilities. Enable the tool call parser:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-235B-A22B-Thinking-2507 \
--reasoning-parser qwen3 \
--tool-call-parser qwen25 \
--tp 8 \
--host 0.0.0.0 \
--port 8000
```
**Python Example (with Thinking Process):**
```python Example
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY"
)
# Define available tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit"
}
},
"required": ["location"]
}
}
}
]
# Make request with streaming to see thinking process
response = client.chat.completions.create(
model="Qwen/Qwen3-235B-A22B-Thinking-2507",
messages=[
{"role": "user", "content": "What's the weather in Beijing?"}
],
tools=tools,
temperature=0.7,
stream=True
)
# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}
for chunk in response:
if chunk.choices and len(chunk.choices) > 0:
delta = chunk.choices[0].delta
# Print thinking process
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
if not thinking_started:
print("=============== Thinking =================", flush=True)
thinking_started = True
has_thinking = True
print(delta.reasoning_content, end="", flush=True)
# Accumulate tool calls
if hasattr(delta, 'tool_calls') and delta.tool_calls:
# Close thinking section if needed
if has_thinking and thinking_started:
print("\n=============== Content =================\n", flush=True)
thinking_started = False
for tool_call in delta.tool_calls:
index = tool_call.index
if index not in tool_calls_accumulator:
tool_calls_accumulator[index] = {
'name': None,
'arguments': ''
}
if tool_call.function:
if tool_call.function.name:
tool_calls_accumulator[index]['name'] = tool_call.function.name
if tool_call.function.arguments:
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
# Print content
if delta.content:
print(delta.content, end="", flush=True)
# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
print(f"🔧 Tool Call: {tool_call['name']}")
print(f" Arguments: {tool_call['arguments']}")
print()
```
**Output Example:**
```text Output
=============== Thinking =================
Okay, the user is asking for the weather in Beijing. Let me check the tools available. There's a function called get_weather that takes location and unit parameters. The location is required, so I need to specify Beijing as the location. The unit is optional and can be either celsius or fahrenheit. Since the user didn't specify the unit, maybe I should default to a common one. In China, they usually use celsius, so I'll set unit to celsius. I'll call the get_weather function with location: Beijing and unit: celsius. That should get the current weather for them.
=============== Content =================
🔧 Tool Call: get_weather
Arguments: {"location": "Beijing", "unit": "celsius"}
```
**Note:**
- The reasoning parser shows how the model decides to use a tool
- Tool calls are clearly marked with the function name and arguments
- You can then execute the function and send the result back to continue the conversation
**Handling Tool Call Results:**
```python Example
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
# Your actual weather API call here
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
# Send tool result back to the model
messages = [
{"role": "user", "content": "What's the weather in Beijing?"},
{
"role": "assistant",
"content": None,
"tool_calls": [{
"id": "call_123",
"type": "function",
"function": {
"name": "get_weather",
"arguments": '{"location": "Beijing", "unit": "celsius"}'
}
}]
},
{
"role": "tool",
"tool_call_id": "call_123",
"content": get_weather("Beijing", "celsius")
}
]
final_response = client.chat.completions.create(
model="Qwen/Qwen3-235B-A22B-Thinking-2507",
messages=messages,
temperature=0.7
)
print(final_response.choices[0].message.content)
# Output: "The current weather in Beijing is **22°C** and **sunny**. A perfect day to enjoy outdoor activities! 🌞"
```
## 5. Benchmark
### 5.1 Speed Benchmark
**Test Environment:**
- Hardware: NVIDIA B200 GPU (8x)
- Model: Qwen3-235B-A22B-Instruct-2507
- Tensor Parallelism: 8
- sglang version: 0.5.6
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.
#### 5.1.1 Standard Scenario Benchmark
- Model Deployment Command:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
--tp 8
```
##### 5.1.1.1 Low Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 43.56
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 4210
Total generated tokens (retokenized): 4206
Request throughput (req/s): 0.23
Input token throughput (tok/s): 140.07
Output token throughput (tok/s): 96.65
Peak output token throughput (tok/s): 100.00
Peak concurrent requests: 2
Total token throughput (tok/s): 236.72
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 4353.63
Median E2E Latency (ms): 3475.79
---------------Time to First Token----------------
Mean TTFT (ms): 99.03
Median TTFT (ms): 92.18
P99 TTFT (ms): 166.05
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 10.12
Median TPOT (ms): 10.12
P99 TPOT (ms): 10.15
---------------Inter-Token Latency----------------
Mean ITL (ms): 10.13
Median ITL (ms): 10.12
P95 ITL (ms): 10.49
P99 ITL (ms): 10.70
Max ITL (ms): 13.45
==================================================
```
##### 5.1.1.2 Medium Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 48.95
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 40725
Total generated tokens (retokenized): 40716
Request throughput (req/s): 1.63
Input token throughput (tok/s): 810.44
Output token throughput (tok/s): 832.04
Peak output token throughput (tok/s): 1151.00
Peak concurrent requests: 21
Total token throughput (tok/s): 1642.48
Concurrency: 13.61
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 8326.72
Median E2E Latency (ms): 8827.86
---------------Time to First Token----------------
Mean TTFT (ms): 215.70
Median TTFT (ms): 88.82
P99 TTFT (ms): 727.08
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 16.36
Median TPOT (ms): 16.12
P99 TPOT (ms): 24.09
---------------Inter-Token Latency----------------
Mean ITL (ms): 15.96
Median ITL (ms): 14.52
P95 ITL (ms): 16.04
P99 ITL (ms): 67.69
Max ITL (ms): 457.52
==================================================
```
##### 5.1.1.3 High Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 1000 \
--num-prompts 500 \
--max-concurrency 100
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 100
Successful requests: 500
Benchmark duration (s): 92.07
Total input tokens: 249831
Total input text tokens: 249831
Total input vision tokens: 0
Total generated tokens: 252162
Total generated tokens (retokenized): 251124
Request throughput (req/s): 5.43
Input token throughput (tok/s): 2713.46
Output token throughput (tok/s): 2738.78
Peak output token throughput (tok/s): 4400.00
Peak concurrent requests: 110
Total token throughput (tok/s): 5452.24
Concurrency: 90.50
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 16665.09
Median E2E Latency (ms): 16060.10
---------------Time to First Token----------------
Mean TTFT (ms): 260.55
Median TTFT (ms): 122.68
P99 TTFT (ms): 863.11
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 32.94
Median TPOT (ms): 34.04
P99 TPOT (ms): 41.19
---------------Inter-Token Latency----------------
Mean ITL (ms): 32.59
Median ITL (ms): 23.54
P95 ITL (ms): 69.79
P99 ITL (ms): 119.09
Max ITL (ms): 577.70
==================================================
```
#### 5.1.2 Reasoning Scenario Benchmark
- Model Deployment Command:
```shell Command
python -m sglang.launch_server \
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
--tp 8
```
##### 5.1.2.1 Low Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 10 \
--max-concurrency 1
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 457.45
Total input tokens: 6101
Total input text tokens: 6101
Total input vision tokens: 0
Total generated tokens: 44452
Total generated tokens (retokenized): 44059
Request throughput (req/s): 0.02
Input token throughput (tok/s): 13.34
Output token throughput (tok/s): 97.17
Peak output token throughput (tok/s): 100.00
Peak concurrent requests: 2
Total token throughput (tok/s): 110.51
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 45742.42
Median E2E Latency (ms): 49266.87
---------------Time to First Token----------------
Mean TTFT (ms): 110.60
Median TTFT (ms): 109.36
P99 TTFT (ms): 167.43
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 10.23
Median TPOT (ms): 10.24
P99 TPOT (ms): 10.32
---------------Inter-Token Latency----------------
Mean ITL (ms): 10.27
Median ITL (ms): 10.26
P95 ITL (ms): 10.71
P99 ITL (ms): 10.97
Max ITL (ms): 15.79
==================================================
```
##### 5.1.2.2 Medium Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 80 \
--max-concurrency 16
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 340.17
Total input tokens: 39668
Total input text tokens: 39668
Total input vision tokens: 0
Total generated tokens: 318226
Total generated tokens (retokenized): 318104
Request throughput (req/s): 0.24
Input token throughput (tok/s): 116.61
Output token throughput (tok/s): 935.49
Peak output token throughput (tok/s): 1120.00
Peak concurrent requests: 19
Total token throughput (tok/s): 1052.10
Concurrency: 13.85
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 58885.30
Median E2E Latency (ms): 59238.70
---------------Time to First Token----------------
Mean TTFT (ms): 169.71
Median TTFT (ms): 101.61
P99 TTFT (ms): 455.71
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 14.82
Median TPOT (ms): 14.91
P99 TPOT (ms): 15.20
---------------Inter-Token Latency----------------
Mean ITL (ms): 14.76
Median ITL (ms): 14.63
P95 ITL (ms): 15.46
P99 ITL (ms): 16.62
Max ITL (ms): 104.94
==================================================
```
##### 5.1.2.3 High Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
--dataset-name random \
--random-input-len 1000 \
--random-output-len 8000 \
--num-prompts 320 \
--max-concurrency 64
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 544.83
Total input tokens: 158939
Total input text tokens: 158939
Total input vision tokens: 0
Total generated tokens: 1300705
Total generated tokens (retokenized): 1293015
Request throughput (req/s): 0.59
Input token throughput (tok/s): 291.72
Output token throughput (tok/s): 2387.34
Peak output token throughput (tok/s): 3008.00
Peak concurrent requests: 68
Total token throughput (tok/s): 2679.06
Concurrency: 56.35
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 95937.70
Median E2E Latency (ms): 99362.32
---------------Time to First Token----------------
Mean TTFT (ms): 265.03
Median TTFT (ms): 129.11
P99 TTFT (ms): 823.85
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 23.66
Median TPOT (ms): 24.07
P99 TPOT (ms): 24.97
---------------Inter-Token Latency----------------
Mean ITL (ms): 23.54
Median ITL (ms): 23.07
P95 ITL (ms): 25.92
P99 ITL (ms): 63.87
Max ITL (ms): 408.30
==================================================
```
#### 5.1.3 Summarization Scenario Benchmark
##### 5.1.3.1 Low Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 10 \
--max-concurrency 1
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 1
Successful requests: 10
Benchmark duration (s): 44.82
Total input tokens: 41941
Total input text tokens: 41941
Total input vision tokens: 0
Total generated tokens: 4210
Total generated tokens (retokenized): 4210
Request throughput (req/s): 0.22
Input token throughput (tok/s): 935.86
Output token throughput (tok/s): 93.94
Peak output token throughput (tok/s): 99.00
Peak concurrent requests: 2
Total token throughput (tok/s): 1029.80
Concurrency: 1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 4479.60
Median E2E Latency (ms): 3622.99
---------------Time to First Token----------------
Mean TTFT (ms): 139.90
Median TTFT (ms): 114.85
P99 TTFT (ms): 225.17
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 10.31
Median TPOT (ms): 10.33
P99 TPOT (ms): 10.51
---------------Inter-Token Latency----------------
Mean ITL (ms): 10.33
Median ITL (ms): 10.33
P95 ITL (ms): 10.73
P99 ITL (ms): 10.93
Max ITL (ms): 14.48
==================================================
```
##### 5.1.3.2 Medium Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 80 \
--max-concurrency 16
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 80
Benchmark duration (s): 50.68
Total input tokens: 300020
Total input text tokens: 300020
Total input vision tokens: 0
Total generated tokens: 41589
Total generated tokens (retokenized): 41578
Request throughput (req/s): 1.58
Input token throughput (tok/s): 5920.41
Output token throughput (tok/s): 820.69
Peak output token throughput (tok/s): 1200.00
Peak concurrent requests: 20
Total token throughput (tok/s): 6741.10
Concurrency: 13.90
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 8805.54
Median E2E Latency (ms): 9368.79
---------------Time to First Token----------------
Mean TTFT (ms): 284.29
Median TTFT (ms): 168.48
P99 TTFT (ms): 1027.21
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 16.81
Median TPOT (ms): 16.66
P99 TPOT (ms): 27.18
---------------Inter-Token Latency----------------
Mean ITL (ms): 16.42
Median ITL (ms): 13.68
P95 ITL (ms): 17.23
P99 ITL (ms): 90.75
Max ITL (ms): 574.64
==================================================
```
##### 5.1.3.3 High Concurrency
- Benchmark Command:
```shell Command
python3 -m sglang.bench_serving \
--backend sglang \
--model Qwen/Qwen3-235B-A22B-Instruct-2507 \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 320 \
--max-concurrency 64
```
- Test Results:
```text Output
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 320
Benchmark duration (s): 94.77
Total input tokens: 1273893
Total input text tokens: 1273893
Total input vision tokens: 0
Total generated tokens: 169680
Total generated tokens (retokenized): 169640
Request throughput (req/s): 3.38
Input token throughput (tok/s): 13441.86
Output token throughput (tok/s): 1790.43
Peak output token throughput (tok/s): 2687.00
Peak concurrent requests: 70
Total token throughput (tok/s): 15232.28
Concurrency: 58.63
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 17364.14
Median E2E Latency (ms): 17495.95
---------------Time to First Token----------------
Mean TTFT (ms): 238.22
Median TTFT (ms): 203.27
P99 TTFT (ms): 510.48
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 32.50
Median TPOT (ms): 34.27
P99 TPOT (ms): 40.59
---------------Inter-Token Latency----------------
Mean ITL (ms): 32.36
Median ITL (ms): 22.50
P95 ITL (ms): 97.81
P99 ITL (ms): 151.55
Max ITL (ms): 352.79
==================================================
```
### 5.2 Accuracy Benchmark
#### 5.2.1 GSM8K Benchmark
- **Benchmark Command:**
```shell Command
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
```
- **Results**:
- Qwen/Qwen3-235B-A22B-Instruct-2507
```text Output
Accuracy: 0.945
Invalid: 0.000
Latency: 11.980 s
Output throughput: 2358.105 token/s
```