810 lines
31 KiB
Plaintext
810 lines
31 KiB
Plaintext
---
|
||
title: Qwen3-VL
|
||
metatags:
|
||
description: "Deploy Qwen3-VL vision-language models with SGLang - open model for text, 262K context, enhanced visual reasoning and agent capabilities."
|
||
---
|
||
|
||
|
||
## 1. Model Introduction
|
||
|
||
[Qwen3-VL series](https://github.com/QwenLM/Qwen3-VL) are the most powerful vision-language models in the Qwen series to date, featuring advanced capabilities in multi-modal understanding, reasoning, and agentic applications.
|
||
|
||
This generation delivers comprehensive upgrades across the board:
|
||
|
||
- **Superior text understanding & generation**: Qwen3-VL-235B-A22B-Instruct was ranked as the [#1 open model for text on lmarena.ai](https://x.com/arena/status/1973151703563460942)
|
||
- **Deeper visual perception & reasoning**: Enhanced image and video understanding capabilities.
|
||
- **Extended context length**: Supports up to 262K tokens for processing long documents and videos.
|
||
- **Enhanced spatial and video dynamics comprehension**: Better understanding of spatial relationships and temporal dynamics.
|
||
- **Stronger agent interaction capabilities**: Improved tool use and search-based agent performance.
|
||
- **Flexible deployment options**: Available in Dense and MoE architectures that scale from edge to cloud, with Instruct and reasoning-enhanced Thinking editions.
|
||
|
||
For more details, please refer to the [official Qwen3-VL GitHub Repository](https://github.com/QwenLM/Qwen3-VL).
|
||
|
||
## 2. SGLang Installation
|
||
|
||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||
|
||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||
|
||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||
|
||
## 3. Model Deployment
|
||
|
||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||
|
||
### 3.1 Basic Configuration
|
||
|
||
The Qwen3-VL series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA and AMD GPUs, as well as Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size.
|
||
|
||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.
|
||
|
||
import { Qwen3VLDeployment } from "/src/snippets/autoregressive/qwen3-vl-deployment.jsx";
|
||
|
||
<Qwen3VLDeployment />
|
||
|
||
### 3.2 Configuration Tips
|
||
|
||
* **Multimodal attention backend** : Usually, `--mm-attention-backend` is default to `fa3` on H100/H200/A100 for better performance, but it is default to `triton_attn` on B200 for compatibility.
|
||
* **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size * number of images in current running requests.)
|
||
* **Memory Management** : Set lower `--context-length` to conserve memory. A value of `128000` is sufficient for most scenarios, down from the default 262K.
|
||
* **Expert Parallelism** : SGLang supports Expert Parallelism (EP) via `--ep`, allowing experts in MoE models to be deployed on separate GPUs for better throughput. One thing to note is that, for quantized models, you need to set `--ep` to a value that satisfies the requirement: `(moe_intermediate_size / moe_tp_size) % weight_block_size_n == 0, where moe_tp_size is equal to tp_size divided by ep_size.` Note that EP may perform worse in low concurrency scenarios due to additional communication overhead. Check out [Expert Parallelism Deployment](../../../docs/advanced_features/expert_parallelism) for more details.
|
||
* **Kernel Tuning** : For MoE Triton kernel tuning on your specific hardware, refer to [fused_moe_triton](https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton).
|
||
|
||
**Hardware-specific notes:**
|
||
- **H100 (FP8):** Use the `Qwen/Qwen3-VL-235B-A22B-Instruct-FP8` checkpoint for best memory efficiency.
|
||
- **A100 / H100 (BF16):** Use standard multimodal parameters to manage throughput and GPU memory usage.
|
||
- **H200 / B200:** Runs out of the box, supporting full context length plus concurrent image + video processing.
|
||
|
||
**Additional multimodal server parameters:**
|
||
- `--keep-mm-feature-on-device`: Retain multimodal feature tensors on GPU after processing to avoid device-to-host memory copies, improving performance for high-frequency inference.
|
||
|
||
**Example with full multimodal optimizations:**
|
||
```bash Command
|
||
SGLANG_USE_CUDA_IPC_TRANSPORT=1 \
|
||
SGLANG_VLM_CACHE_SIZE_MB=0 \
|
||
python -m sglang.launch_server \
|
||
--model-path Qwen/Qwen3-VL-235B-A22B-Instruct \
|
||
--host 0.0.0.0 \
|
||
--port 30000 \
|
||
--trust-remote-code \
|
||
--tp-size 8 \
|
||
--enable-cache-report \
|
||
--log-level info \
|
||
--max-running-requests 64 \
|
||
--mem-fraction-static 0.65 \
|
||
--chunked-prefill-size 8192 \
|
||
--attention-backend fa3 \
|
||
--mm-attention-backend fa3 \
|
||
--enable-metrics
|
||
```
|
||
|
||
* **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||
|
||
## 4. Model Invocation
|
||
|
||
### 4.1 Basic Usage
|
||
|
||
For basic API usage and request examples, please refer to:
|
||
|
||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
|
||
|
||
### 4.2 Advanced Usage
|
||
|
||
#### 4.2.1 Multi-Modal Inputs
|
||
|
||
Qwen3-VL supports both image and video inputs. Here's a basic example with image input:
|
||
|
||
```python Example
|
||
import time
|
||
from openai import OpenAI
|
||
|
||
client = OpenAI(
|
||
api_key="EMPTY",
|
||
base_url="http://localhost:30000/v1",
|
||
timeout=3600
|
||
)
|
||
|
||
messages = [
|
||
{
|
||
"role": "user",
|
||
"content": [
|
||
{
|
||
"type": "image_url",
|
||
"image_url": {
|
||
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
|
||
}
|
||
},
|
||
{
|
||
"type": "text",
|
||
"text": "Read all the text in the image."
|
||
}
|
||
]
|
||
}
|
||
]
|
||
|
||
start = time.time()
|
||
response = client.chat.completions.create(
|
||
model="Qwen/Qwen3-VL-235B-A22B-Instruct",
|
||
messages=messages,
|
||
max_tokens=2048
|
||
)
|
||
print(f"Response costs: {time.time() - start:.2f}s")
|
||
print(f"Generated text: {response.choices[0].message.content}")
|
||
```
|
||
|
||
**Example Output:**
|
||
|
||
```text Output
|
||
Response costs: 3.37s
|
||
Generated text: Auntie Anne's
|
||
|
||
CINNAMON SUGAR
|
||
1 x 17,000 17,000
|
||
|
||
SUB TOTAL 17,000
|
||
|
||
GRAND TOTAL 17,000
|
||
|
||
CASH IDR 20,000
|
||
|
||
CHANGE DUE 3,000
|
||
```
|
||
|
||
**Multi-Image Input Example:**
|
||
|
||
Qwen3-VL can process multiple images in a single request for comparison or analysis:
|
||
|
||
```python Example
|
||
import time
|
||
from openai import OpenAI
|
||
|
||
client = OpenAI(
|
||
api_key="EMPTY",
|
||
base_url="http://localhost:30000/v1",
|
||
timeout=3600
|
||
)
|
||
|
||
messages = [
|
||
{
|
||
"role": "user",
|
||
"content": [
|
||
{
|
||
"type": "image_url",
|
||
"image_url": {
|
||
"url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
|
||
}
|
||
},
|
||
{
|
||
"type": "image_url",
|
||
"image_url": {
|
||
"url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
|
||
}
|
||
},
|
||
{
|
||
"type": "text",
|
||
"text": "Compare these two images and describe the differences in 100 words or less. Focus on the key visual elements, colors, textures, and any notable contrasts between the two scenes. Be specific about what you see in each image."
|
||
}
|
||
]
|
||
}
|
||
]
|
||
|
||
start = time.time()
|
||
response = client.chat.completions.create(
|
||
model="Qwen/Qwen3-VL-235B-A22B-Instruct",
|
||
messages=messages,
|
||
max_tokens=2048
|
||
)
|
||
print(f"Response costs: {time.time() - start:.2f}s")
|
||
print(f"Generated text: {response.choices[0].message.content}")
|
||
```
|
||
|
||
**Example Output:**
|
||
|
||
```text Output
|
||
Response costs: 10.18s
|
||
Generated text: The two images present starkly different portrayals of Hong Kong’s iconic red taxis, contrasting a dynamic street-level moment with a static, large-scale gathering.
|
||
|
||
The first image is a close-up, eye-level shot capturing a single red Toyota Crown taxi (license plate RX 5004) in motion or paused at an urban intersection. Its glossy red paint gleams under daylight, reflecting the vibrant, cluttered backdrop of a Hong Kong street — neon signs, glass-fronted shops displaying sunglasses, and Chinese characters. The taxi’s chrome grille, clear headlights, and black trim provide visual contrast. A green “4 SEATS” sticker and a “的士 TAXI” sign on the side reinforce its identity. The composition is intimate, focusing on the vehicle’s details — the texture of its paint, the slight reflections on the windows, and the crispness of its license plate. Other red taxis flank it, suggesting a bustling city rhythm, but the central taxi dominates the frame, conveying movement and immediacy.
|
||
|
||
In contrast, the second image is an elevated, wide-angle shot of dozens of red taxis — along with a few green ones — parked in neat, grid-like rows on what appears to be a highway or staging area. The scene is static, almost ceremonial. Many taxis have their hoods open, suggesting maintenance, inspection, or protest. People are scattered among the vehicles, some inspecting engines, others conversing — adding a human, documentary element. The dominant color remains red, but the repetition creates a visual pattern rather than individual focus. The green taxis offer a subtle color contrast, hinting at different service zones (green for New Territories, red for urban areas). The setting is more utilitarian — concrete barriers, metal railings, and sparse vegetation — with an overpass looming in the background. The texture here is less about polished paint and more about the collective mass of vehicles, the asphalt, and the functional layout.
|
||
|
||
Key contrasts emerge: the first image is kinetic and personal, emphasizing the taxi as a working vehicle in the city’s daily flow; the second is static and collective, portraying the taxis as a fleet, possibly for logistical or political purposes. The lighting in both is bright daylight, but the first has richer color saturation and depth due to its proximity and urban backdrop, while the second feels flatter, more documentary in tone. The first image invites you into the city’s pulse; the second invites you to observe a system — organized, perhaps even paused — from a distance.
|
||
|
||
In essence, the first image celebrates the individual taxi in its natural habitat; the second reveals the scale and structure behind the fleet, transforming the familiar red icon into a symbol of coordination, maintenance, or collective action. Both are quintessentially Hong Kong, yet they offer vastly different narratives — one of motion and commerce, the other of assembly and purpose.
|
||
```
|
||
|
||
**Video Input Example:**
|
||
|
||
Qwen3-VL supports video understanding by processing video URLs:
|
||
|
||
```python Example
|
||
import time
|
||
from openai import OpenAI
|
||
|
||
client = OpenAI(
|
||
api_key="EMPTY",
|
||
base_url="http://localhost:30000/v1",
|
||
timeout=3600
|
||
)
|
||
|
||
messages = [
|
||
{
|
||
"role": "user",
|
||
"content": [
|
||
{
|
||
"type": "video_url",
|
||
"video_url": {
|
||
"url": "https://videos.pexels.com/video-files/4114797/4114797-uhd_3840_2160_25fps.mp4"
|
||
}
|
||
},
|
||
{
|
||
"type": "text",
|
||
"text": "Describe what happens in this video."
|
||
}
|
||
]
|
||
}
|
||
]
|
||
|
||
start = time.time()
|
||
response = client.chat.completions.create(
|
||
model="Qwen/Qwen3-VL-235B-A22B-Instruct",
|
||
messages=messages,
|
||
max_tokens=2048
|
||
)
|
||
print(f"Response costs: {time.time() - start:.2f}s")
|
||
print(f"Generated text: {response.choices[0].message.content}")
|
||
```
|
||
|
||
**Note:**
|
||
|
||
- For video processing, ensure you have sufficient context length configured (up to 262K tokens)
|
||
- Video processing may require more memory; adjust `--mem-fraction-static` accordingly
|
||
- You can also provide local file paths using `file://` protocol
|
||
|
||
**Example Output:**
|
||
|
||
```text Output
|
||
Response costs: 3.89s
|
||
Generated text: A person wearing blue gloves is using a microscope. They are adjusting the focus knob with one hand while holding a pipette with the other, suggesting they are preparing or examining a sample on the slide beneath the objective lens. The microscope's 40x objective lens is positioned over the slide, indicating a high-magnification observation. The person carefully manipulates the slide and the microscope controls, likely to achieve a clear view of the specimen.
|
||
```
|
||
|
||
#### 4.2.2 Reasoning Parser
|
||
|
||
Qwen3-VL-Thinking supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
|
||
|
||
```shell Command
|
||
python -m sglang.launch_server \
|
||
--model Qwen/Qwen3-VL-235B-A22B-Thinking \
|
||
--reasoning-parser qwen3 \
|
||
--tp 8 \
|
||
--host 0.0.0.0 \
|
||
--port 30000
|
||
```
|
||
|
||
**Streaming with Thinking Process:**
|
||
|
||
```python Example
|
||
from openai import OpenAI
|
||
|
||
client = OpenAI(
|
||
base_url="http://localhost:30000/v1",
|
||
api_key="EMPTY"
|
||
)
|
||
|
||
# Enable streaming to see the thinking process in real-time
|
||
response = client.chat.completions.create(
|
||
model="Qwen/Qwen3-VL-235B-A22B-Thinking",
|
||
messages=[
|
||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||
],
|
||
temperature=0.7,
|
||
max_tokens=2048,
|
||
stream=True
|
||
)
|
||
|
||
# Process the stream
|
||
has_thinking = False
|
||
has_answer = False
|
||
thinking_started = False
|
||
|
||
for chunk in response:
|
||
if chunk.choices and len(chunk.choices) > 0:
|
||
delta = chunk.choices[0].delta
|
||
|
||
# Print thinking process
|
||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||
if not thinking_started:
|
||
print("=============== Thinking =================", flush=True)
|
||
thinking_started = True
|
||
has_thinking = True
|
||
print(delta.reasoning_content, end="", flush=True)
|
||
|
||
# Print answer content
|
||
if delta.content:
|
||
# Close thinking section and add content header
|
||
if has_thinking and not has_answer:
|
||
print("\n=============== Content =================", flush=True)
|
||
has_answer = True
|
||
print(delta.content, end="", flush=True)
|
||
|
||
print()
|
||
```
|
||
|
||
**Output Example:**
|
||
|
||
```text Output
|
||
=============== Thinking =================
|
||
To solve this problem, I need to calculate 15% of 240.
|
||
Step 1: Convert 15% to decimal: 15% = 0.15
|
||
Step 2: Multiply 240 by 0.15
|
||
Step 3: 240 × 0.15 = 36
|
||
=============== Content =================
|
||
|
||
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
|
||
```
|
||
|
||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||
|
||
#### 4.2.3 Tool Calling
|
||
|
||
Qwen3-VL supports tool calling capabilities. Enable the tool call parser:
|
||
|
||
```shell Command
|
||
python -m sglang.launch_server \
|
||
--model Qwen/Qwen3-VL-235B-A22B-Thinking \
|
||
--reasoning-parser qwen3 \
|
||
--tool-call-parser qwen \
|
||
--tp 8 \
|
||
--host 0.0.0.0 \
|
||
--port 30000
|
||
```
|
||
|
||
**Python Example (with Thinking Process):**
|
||
|
||
```python Example
|
||
from openai import OpenAI
|
||
|
||
client = OpenAI(
|
||
base_url="http://localhost:30000/v1",
|
||
api_key="EMPTY"
|
||
)
|
||
|
||
# Define available tools
|
||
tools = [
|
||
{
|
||
"type": "function",
|
||
"function": {
|
||
"name": "get_weather",
|
||
"description": "Get the current weather for a location",
|
||
"parameters": {
|
||
"type": "object",
|
||
"properties": {
|
||
"location": {
|
||
"type": "string",
|
||
"description": "The city name"
|
||
},
|
||
"unit": {
|
||
"type": "string",
|
||
"enum": ["celsius", "fahrenheit"],
|
||
"description": "Temperature unit"
|
||
}
|
||
},
|
||
"required": ["location"]
|
||
}
|
||
}
|
||
}
|
||
]
|
||
|
||
# Make request with streaming to see thinking process
|
||
response = client.chat.completions.create(
|
||
model="Qwen/Qwen3-VL-235B-A22B-Thinking",
|
||
messages=[
|
||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||
],
|
||
tools=tools,
|
||
temperature=0.7,
|
||
stream=True
|
||
)
|
||
|
||
# Process streaming response
|
||
thinking_started = False
|
||
has_thinking = False
|
||
tool_calls_accumulator = {}
|
||
|
||
for chunk in response:
|
||
if chunk.choices and len(chunk.choices) > 0:
|
||
delta = chunk.choices[0].delta
|
||
|
||
# Print thinking process
|
||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||
if not thinking_started:
|
||
print("=============== Thinking =================", flush=True)
|
||
thinking_started = True
|
||
has_thinking = True
|
||
print(delta.reasoning_content, end="", flush=True)
|
||
|
||
# Accumulate tool calls
|
||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||
# Close thinking section if needed
|
||
if has_thinking and thinking_started:
|
||
print("\n=============== Content =================\n", flush=True)
|
||
thinking_started = False
|
||
|
||
for tool_call in delta.tool_calls:
|
||
index = tool_call.index
|
||
if index not in tool_calls_accumulator:
|
||
tool_calls_accumulator[index] = {
|
||
'name': None,
|
||
'arguments': ''
|
||
}
|
||
|
||
if tool_call.function:
|
||
if tool_call.function.name:
|
||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||
if tool_call.function.arguments:
|
||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||
|
||
# Print content
|
||
if delta.content:
|
||
print(delta.content, end="", flush=True)
|
||
|
||
# Print accumulated tool calls
|
||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||
print(f"🔧 Tool Call: {tool_call['name']}")
|
||
print(f" Arguments: {tool_call['arguments']}")
|
||
|
||
print()
|
||
```
|
||
|
||
**Output Example:**
|
||
|
||
```text Output
|
||
=============== Thinking =================
|
||
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
|
||
I should call the function with location="Beijing".
|
||
=============== Content =================
|
||
|
||
🔧 Tool Call: get_weather
|
||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||
```
|
||
|
||
**Note:**
|
||
|
||
- The reasoning parser shows how the model decides to use a tool
|
||
- Tool calls are clearly marked with the function name and arguments
|
||
- You can then execute the function and send the result back to continue the conversation
|
||
|
||
**Handling Tool Call Results:**
|
||
|
||
```python Example
|
||
# After getting the tool call, execute the function
|
||
def get_weather(location, unit="celsius"):
|
||
# Your actual weather API call here
|
||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||
|
||
# Send tool result back to the model
|
||
messages = [
|
||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||
{
|
||
"role": "assistant",
|
||
"content": None,
|
||
"tool_calls": [{
|
||
"id": "call_123",
|
||
"type": "function",
|
||
"function": {
|
||
"name": "get_weather",
|
||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||
}
|
||
}]
|
||
},
|
||
{
|
||
"role": "tool",
|
||
"tool_call_id": "call_123",
|
||
"content": get_weather("Beijing", "celsius")
|
||
}
|
||
]
|
||
|
||
final_response = client.chat.completions.create(
|
||
model="Qwen/Qwen3-VL-235B-A22B-Thinking",
|
||
messages=messages,
|
||
temperature=0.7
|
||
)
|
||
|
||
print(final_response.choices[0].message.content)
|
||
# Output: "The weather in Beijing is currently 22°C and sunny."
|
||
```
|
||
|
||
## 5. Benchmark
|
||
|
||
### 5.1 Speed Benchmark
|
||
|
||
**Test Environment:**
|
||
|
||
- Hardware: NVIDIA B200 GPU (8x)
|
||
- Model: Qwen3-VL-235B-A22B-Instruct
|
||
- Tensor Parallelism: 8
|
||
- sglang version: 0.5.6
|
||
|
||
We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. To simulate real-world usage, you can specify different input and output lengths for each request. For example, each request can have 128 input tokens, two 720p images, and 1024 output tokens.
|
||
|
||
#### 5.1.1 Latency-Sensitive Benchmark
|
||
|
||
- Model Deployment Command:
|
||
|
||
```shell Command
|
||
python -m sglang.launch_server \
|
||
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
|
||
--tp 8 \
|
||
--host 0.0.0.0 \
|
||
--port 30000
|
||
```
|
||
|
||
- Benchmark Command:
|
||
|
||
```shell Command
|
||
python3 -m sglang.bench_serving \
|
||
--backend sglang-oai-chat \
|
||
--host 127.0.0.1 \
|
||
--port 30000 \
|
||
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
|
||
--dataset-name image \
|
||
--image-count 2 \
|
||
--image-resolution 720p \
|
||
--random-input-len 128 \
|
||
--random-output-len 1024 \
|
||
--num-prompts 10 \
|
||
--max-concurrency 1
|
||
```
|
||
|
||
- **Test Results:**
|
||
|
||
```text Output
|
||
============ Serving Benchmark Result ============
|
||
Backend: sglang-oai-chat
|
||
Traffic request rate: inf
|
||
Max request concurrency: 1
|
||
Successful requests: 10
|
||
Benchmark duration (s): 45.97
|
||
Total input tokens: 18348
|
||
Total input text tokens: 708
|
||
Total input vision tokens: 17640
|
||
Total generated tokens: 4220
|
||
Total generated tokens (retokenized): 3423
|
||
Request throughput (req/s): 0.22
|
||
Input token throughput (tok/s): 399.17
|
||
Output token throughput (tok/s): 91.81
|
||
Peak output token throughput (tok/s): 96.00
|
||
Peak concurrent requests: 2
|
||
Total token throughput (tok/s): 490.98
|
||
Concurrency: 1.00
|
||
----------------End-to-End Latency----------------
|
||
Mean E2E Latency (ms): 4594.52
|
||
Median E2E Latency (ms): 3725.04
|
||
---------------Time to First Token----------------
|
||
Mean TTFT (ms): 193.35
|
||
Median TTFT (ms): 196.32
|
||
P99 TTFT (ms): 222.75
|
||
-----Time per Output Token (excl. 1st token)------
|
||
Mean TPOT (ms): 10.44
|
||
Median TPOT (ms): 10.44
|
||
P99 TPOT (ms): 10.47
|
||
---------------Inter-Token Latency----------------
|
||
Mean ITL (ms): 11.78
|
||
Median ITL (ms): 10.48
|
||
P95 ITL (ms): 21.01
|
||
P99 ITL (ms): 31.40
|
||
Max ITL (ms): 31.92
|
||
==================================================
|
||
```
|
||
|
||
**Optimized Results (with CUDA IPC Transport):**
|
||
|
||
For further TTFT optimization, enable CUDA IPC Transport for multimodal features by setting `SGLANG_USE_CUDA_IPC_TRANSPORT=1`. This significantly reduces TTFT by using CUDA IPC for transferring multimodal features.
|
||
|
||
- Model Deployment Command:
|
||
|
||
```shell Command
|
||
SGLANG_USE_CUDA_IPC_TRANSPORT=1 python -m sglang.launch_server \
|
||
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
|
||
--tp 8 \
|
||
--host 0.0.0.0 \
|
||
--port 30000
|
||
```
|
||
|
||
- Benchmark Command:
|
||
|
||
```shell Command
|
||
python3 -m sglang.bench_serving \
|
||
--backend sglang-oai-chat \
|
||
--host 127.0.0.1 \
|
||
--port 30000 \
|
||
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
|
||
--dataset-name image \
|
||
--image-count 2 \
|
||
--image-resolution 720p \
|
||
--random-input-len 128 \
|
||
--random-output-len 1024 \
|
||
--num-prompts 100 \
|
||
--max-concurrency 1
|
||
```
|
||
|
||
- **Test Results:**
|
||
|
||
With `SGLANG_USE_CUDA_IPC_TRANSPORT=1`, TTFT improves significantly:
|
||
|
||
```text Output
|
||
============ Serving Benchmark Result ============
|
||
Backend: sglang-oai-chat
|
||
Traffic request rate: inf
|
||
Max request concurrency: 1
|
||
Successful requests: 100
|
||
Benchmark duration (s): 566.84
|
||
Total input tokens: 183667
|
||
Total input text tokens: 7267
|
||
Total input vision tokens: 176400
|
||
Total generated tokens: 52444
|
||
Total generated tokens (retokenized): 28702
|
||
Request throughput (req/s): 0.18
|
||
Input token throughput (tok/s): 324.02
|
||
Output token throughput (tok/s): 92.52
|
||
Peak output token throughput (tok/s): 96.00
|
||
Peak concurrent requests: 3
|
||
Total token throughput (tok/s): 416.54
|
||
Concurrency: 1.00
|
||
----------------End-to-End Latency----------------
|
||
Mean E2E Latency (ms): 5667.50
|
||
Median E2E Latency (ms): 5830.00
|
||
---------------Time to First Token----------------
|
||
Mean TTFT (ms): 191.16
|
||
Median TTFT (ms): 182.58
|
||
P99 TTFT (ms): 244.58
|
||
-----Time per Output Token (excl. 1st token)------
|
||
Mean TPOT (ms): 10.46
|
||
Median TPOT (ms): 10.46
|
||
P99 TPOT (ms): 10.48
|
||
---------------Inter-Token Latency----------------
|
||
Mean ITL (ms): 13.91
|
||
Median ITL (ms): 10.56
|
||
P95 ITL (ms): 21.35
|
||
P99 ITL (ms): 31.55
|
||
Max ITL (ms): 42.36
|
||
==================================================
|
||
```
|
||
|
||
#### 5.1.2 Throughput-Sensitive Benchmark
|
||
|
||
- Model Deployment Command:
|
||
|
||
```shell Command
|
||
python -m sglang.launch_server \
|
||
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
|
||
--tp 8 \
|
||
--host 0.0.0.0 \
|
||
--port 30000
|
||
```
|
||
|
||
- Benchmark Command:
|
||
|
||
```shell Command
|
||
python3 -m sglang.bench_serving \
|
||
--backend sglang-oai-chat \
|
||
--host 127.0.0.1 \
|
||
--port 30000 \
|
||
--model Qwen/Qwen3-VL-235B-A22B-Instruct \
|
||
--dataset-name image \
|
||
--image-count 2 \
|
||
--image-resolution 720p \
|
||
--random-input-len 128 \
|
||
--random-output-len 1024 \
|
||
--num-prompts 1000 \
|
||
--max-concurrency 100
|
||
```
|
||
|
||
- **Test Results:**
|
||
|
||
```text Output
|
||
============ Serving Benchmark Result ============
|
||
Backend: sglang-oai-chat
|
||
Traffic request rate: inf
|
||
Max request concurrency: 100
|
||
Successful requests: 1000
|
||
Benchmark duration (s): 584.65
|
||
Total input tokens: 1839015
|
||
Total input text tokens: 75015
|
||
Total input vision tokens: 1764000
|
||
Total generated tokens: 510855
|
||
Total generated tokens (retokenized): 284284
|
||
Request throughput (req/s): 1.71
|
||
Input token throughput (tok/s): 3145.50
|
||
Output token throughput (tok/s): 873.78
|
||
Peak output token throughput (tok/s): 2855.00
|
||
Peak concurrent requests: 107
|
||
Total token throughput (tok/s): 4019.29
|
||
Concurrency: 98.35
|
||
----------------End-to-End Latency----------------
|
||
Mean E2E Latency (ms): 57502.05
|
||
Median E2E Latency (ms): 54301.08
|
||
---------------Time to First Token----------------
|
||
Mean TTFT (ms): 5802.23
|
||
Median TTFT (ms): 1444.75
|
||
P99 TTFT (ms): 46675.92
|
||
-----Time per Output Token (excl. 1st token)------
|
||
Mean TPOT (ms): 100.22
|
||
Median TPOT (ms): 105.43
|
||
P99 TPOT (ms): 144.37
|
||
---------------Inter-Token Latency----------------
|
||
Mean ITL (ms): 134.20
|
||
Median ITL (ms): 25.57
|
||
P95 ITL (ms): 558.14
|
||
P99 ITL (ms): 1449.01
|
||
Max ITL (ms): 33453.23
|
||
==================================================
|
||
```
|
||
|
||
### 5.2 Accuracy Benchmark
|
||
|
||
#### 5.2.1 MMMU Benchmark
|
||
|
||
You can evaluate the model's accuracy using the MMMU dataset with `lmms_eval`:
|
||
|
||
- Benchmark Command:
|
||
|
||
```shell Command
|
||
uv pip install lmms_eval
|
||
|
||
python3 -m lmms_eval \
|
||
--model openai_compatible \
|
||
--model_args "model=Qwen/Qwen3-VL-235B-A22B-Instruct,api_key=EMPTY,base_url=http://127.0.0.1:30000/v1/" \
|
||
--tasks mmmu_val \
|
||
--batch_size 128 \
|
||
--log_samples \
|
||
--log_samples_suffix "openai_compatible" \
|
||
--output_path ./logs \
|
||
--gen_kwargs "max_new_tokens=4096"
|
||
```
|
||
|
||
- **Test Results:**
|
||
|
||
```text Output
|
||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||
<colgroup>
|
||
<col style={{width: "12%"}} />
|
||
<col style={{width: "11%"}} />
|
||
<col style={{width: "11%"}} />
|
||
<col style={{width: "11%"}} />
|
||
<col style={{width: "11%"}} />
|
||
<col style={{width: "11%"}} />
|
||
<col style={{width: "11%"}} />
|
||
<col style={{width: "11%"}} />
|
||
<col style={{width: "11%"}} />
|
||
</colgroup>
|
||
<thead>
|
||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Tasks</th>
|
||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Version</th>
|
||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Filter</th>
|
||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>n-shot</th>
|
||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Metric</th>
|
||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}></th>
|
||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Value</th>
|
||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}></th>
|
||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Stderr</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr>
|
||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>mmmu_val</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>none</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>mmmu_acc</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>↑</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.6567</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>±</td>
|
||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
```
|