docs(cookbook): port popular model usage guides into cookbook pages (#25813)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com> Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
This commit is contained in:
Prabhu Prakash Kagitha
co-authored by
Claude Sonnet 4.6
zijiexia
Zijie Xia
Claude Sonnet 4.6
zijiexia
Zijie Xia
parent
5e63200064
commit
98a1b58c47
@@ -1,58 +0,0 @@
|
||||
---
|
||||
title: "DeepSeek OCR (OCR-1 / OCR-2)"
|
||||
metatags:
|
||||
description: "DeepSeek OCR models are multimodal (image + text) models for OCR and document understanding."
|
||||
---
|
||||
|
||||
DeepSeek OCR models are multimodal (image + text) models for OCR and document understanding.
|
||||
|
||||
## Launch server
|
||||
|
||||
```shell
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-OCR-2 \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
> You can replace `deepseek-ai/DeepSeek-OCR-2` with `deepseek-ai/DeepSeek-OCR`.
|
||||
|
||||
## Prompt examples
|
||||
|
||||
Recommended prompts from the model card:
|
||||
|
||||
```
|
||||
<image>
|
||||
<|grounding|>Convert the document to markdown.
|
||||
```
|
||||
|
||||
```
|
||||
<image>
|
||||
Free OCR.
|
||||
```
|
||||
|
||||
## OpenAI-compatible request example
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
url = "http://localhost:30000/v1/chat/completions"
|
||||
|
||||
data = {
|
||||
"model": "deepseek-ai/DeepSeek-OCR-2",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "<image>\n<|grounding|>Convert the document to markdown."},
|
||||
{"type": "image_url", "image_url": {"url": "https://example.com/your_image.jpg"}},
|
||||
],
|
||||
}
|
||||
],
|
||||
"max_tokens": 512,
|
||||
}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
print(response.text)
|
||||
```
|
||||
@@ -1,375 +0,0 @@
|
||||
---
|
||||
title: "DeepSeek V3/V3.1/R1 Usage"
|
||||
metatags:
|
||||
description: "Deploy DeepSeek V3/R1 with SGLang: MLA optimization, FP8 quantization, multi-node TP, DP attention, MTP speculative decoding. Supports H200, B200, MI300X, A100."
|
||||
---
|
||||
SGLang provides many optimizations specifically designed for the DeepSeek models, making it the inference engine recommended by the official [DeepSeek team](https://github.com/deepseek-ai/DeepSeek-V3/tree/main?tab=readme-ov-file#62-inference-with-sglang-recommended) from Day 0.
|
||||
|
||||
This document outlines current optimizations for DeepSeek.
|
||||
For an overview of the implemented features see the completed [Roadmap](https://github.com/sgl-project/sglang/issues/2591).
|
||||
|
||||
## Launch DeepSeek V3.1/V3/R1 with SGLang
|
||||
|
||||
To run DeepSeek V3.1/V3/R1 models, the recommended settings are as follows:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "50%"}} />
|
||||
<col style={{width: "50%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Weight Type</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Configuration</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}} rowSpan={5}><strong>Full precision <a href="https://huggingface.co/deepseek-ai/DeepSeek-R1-0528">FP8</a></strong><br>*(recommended)*</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8 x H200</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8 x B200</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8 x MI300X</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>2 x 8 x H100/800/20</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Xeon 6980P CPU</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}} rowSpan={4}><strong>Full precision (<a href="https://huggingface.co/unsloth/DeepSeek-R1-0528-BF16">BF16</a>)</strong> (upcast from original FP8)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>2 x 8 x H200</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>2 x 8 x MI300X</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>4 x 8 x H100/800/20</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>4 x 8 x A100/A800</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}} rowSpan={4}><strong>Quantized weights (<a href="https://huggingface.co/meituan/DeepSeek-R1-Channel-INT8">INT8</a>)</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16 x A100/800</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>32 x L40S</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Xeon 6980P CPU</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>4 x Atlas 800I A3</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Quantized weights (<a href="https://huggingface.co/novita/Deepseek-R1-0528-W4AFP8">W4A8</a>)</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8 x H20/100, 4 x H200</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}} rowSpan={2}><strong>Quantized weights (<a href="https://huggingface.co/QuixiAI/DeepSeek-R1-0528-AWQ">AWQ</a>)</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8 x H100/800/20</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8 x A100/A800</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Quantized weights (<a href="https://huggingface.co/amd/DeepSeek-R1-MXFP4-Preview">MXFP4</a>)</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8, 4 x MI355X/350X</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Quantized weights (<a href="https://huggingface.co/nvidia/DeepSeek-R1-0528-NVFP4-v2">NVFP4</a>)</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8, 4 x B200</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<style>
|
||||
.md-typeset__table {
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
.md-typeset__table table {
|
||||
border-collapse: collapse;
|
||||
margin: 1em 0;
|
||||
border: 2px solid var(--md-typeset-table-color);
|
||||
table-layout: fixed;
|
||||
}
|
||||
|
||||
.md-typeset__table th {
|
||||
border: 1px solid var(--md-typeset-table-color);
|
||||
border-bottom: 2px solid var(--md-typeset-table-color);
|
||||
background-color: var(--md-default-bg-color--lighter);
|
||||
padding: 12px;
|
||||
}
|
||||
|
||||
.md-typeset__table td {
|
||||
border: 1px solid var(--md-typeset-table-color);
|
||||
padding: 12px;
|
||||
}
|
||||
|
||||
.md-typeset__table tr:nth-child(2n) {
|
||||
background-color: var(--md-default-bg-color--lightest);
|
||||
}
|
||||
</style>
|
||||
|
||||
<Warning>
|
||||
The official DeepSeek V3 is already in FP8 format, so you should not run it with any quantization arguments like `--quantization fp8`.
|
||||
</Warning>
|
||||
|
||||
Detailed commands for reference:
|
||||
|
||||
- [8 x H200](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#using-docker-recommended)
|
||||
- [4 x B200, 8 x B200](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-one-b200-node)
|
||||
- [8 x MI300X](../hardware-platforms/amd_gpu#running-deepseek-v3)
|
||||
- [2 x 8 x H200](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-two-h2008-nodes-and-docker)
|
||||
- [4 x 8 x A100](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-four-a1008-nodes)
|
||||
- [8 x A100 (AWQ)](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-8-a100a800-with-awq-quantization)
|
||||
- [16 x A100 (INT8)](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-16-a100a800-with-int8-quantization)
|
||||
- [32 x L40S (INT8)](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-32-l40s-with-int8-quantization)
|
||||
- [Xeon 6980P CPU](../hardware-platforms/cpu_server#example-running-deepseek-r1)
|
||||
- [4 x Atlas 800I A3 (int8)](../hardware-platforms/ascend-npus/ascend_npu_deepseek_example#running-deepseek-with-pd-disaggregation-on-4-x-atlas-800i-a3)
|
||||
|
||||
### Download Weights
|
||||
If you encounter errors when starting the server, ensure the weights have finished downloading. It's recommended to download them beforehand or restart multiple times until all weights are downloaded. Please refer to [DeepSeek V3](https://huggingface.co/deepseek-ai/DeepSeek-V3-Base#61-inference-with-deepseek-infer-demo-example-only) official guide to download the weights.
|
||||
|
||||
### Launch with one node of 8 x H200
|
||||
Please refer to [the example](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#installation--launch).
|
||||
|
||||
### Running examples on Multi-Node
|
||||
|
||||
- [Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP](https://lmsys.org/blog/2025-06-16-gb200-part-1/) ([Part I](https://lmsys.org/blog/2025-06-16-gb200-part-1/), [Part II](https://lmsys.org/blog/2025-09-25-gb200-part-2/)) - Comprehensive guide on GB200 optimizations.
|
||||
|
||||
- [Deploying DeepSeek with PD Disaggregation and Large-Scale Expert Parallelism on 96 H100 GPUs](https://lmsys.org/blog/2025-05-05-large-scale-ep/) - Guide on PD disaggregation and large-scale EP.
|
||||
|
||||
- [Serving with two H20*8 nodes](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-two-h208-nodes).
|
||||
|
||||
- [Best Practices for Serving DeepSeek-R1 on H20](https://lmsys.org/blog/2025-09-26-sglang-ant-group/) - Comprehensive guide on H20 optimizations, deployment and performance.
|
||||
|
||||
- [Serving with two H200*8 nodes and docker](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-two-h2008-nodes-and-docker).
|
||||
|
||||
- [Serving with four A100*8 nodes](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-four-a1008-nodes).
|
||||
|
||||
## Optimizations
|
||||
|
||||
### Multi-head Latent Attention (MLA) Throughput Optimizations
|
||||
|
||||
**Description**: [MLA](https://arxiv.org/pdf/2405.04434) is an innovative attention mechanism introduced by the DeepSeek team, aimed at improving inference efficiency. SGLang has implemented specific optimizations for this, including:
|
||||
|
||||
- **Weight Absorption**: By applying the associative law of matrix multiplication to reorder computation steps, this method balances computation and memory access and improves efficiency in the decoding phase.
|
||||
|
||||
- **MLA Attention Backends**: Currently SGLang supports different optimized MLA attention backends, including [FlashAttention3](https://github.com/Dao-AILab/flash-attention), [Flashinfer](https://docs.flashinfer.ai/api/attention.html#flashinfer-mla), [FlashMLA](https://github.com/deepseek-ai/FlashMLA), [CutlassMLA](https://github.com/sgl-project/sglang/pull/5390), **TRTLLM MLA** (optimized for Blackwell architecture), and [Triton](https://github.com/triton-lang/triton) backends. The default FA3 provides good performance across wide workloads.
|
||||
|
||||
- **FP8 Quantization**: W8A8 FP8 and KV Cache FP8 quantization enables efficient FP8 inference. Additionally, we have implemented Batched Matrix Multiplication (BMM) operator to facilitate FP8 inference in MLA with weight absorption.
|
||||
|
||||
- **CUDA Graph & Torch.compile**: Both MLA and Mixture of Experts (MoE) are compatible with CUDA Graph and Torch.compile, which reduces latency and accelerates decoding speed for small batch sizes.
|
||||
|
||||
- **Chunked Prefix Cache**: Chunked prefix cache optimization can increase throughput by cutting prefix cache into chunks, processing them with multi-head attention and merging their states. Its improvement can be significant when doing chunked prefill on long sequences. Currently this optimization is only available for FlashAttention3 backend.
|
||||
|
||||
Overall, with these optimizations, we have achieved up to **7x** acceleration in output throughput compared to the previous version.
|
||||
|
||||
<p align="center">
|
||||
<img src="https://lmsys.org/images/blog/sglang_v0_3/deepseek_mla.svg" alt="Multi-head Latent Attention for DeepSeek Series Models" />
|
||||
</p>
|
||||
|
||||
**Usage**: MLA optimization is enabled by default.
|
||||
|
||||
**Reference**: Check [Blog](https://lmsys.org/blog/2024-09-04-sglang-v0-3/#deepseek-multi-head-latent-attention-mla-throughput-optimizations) and [Slides](https://github.com/sgl-project/sgl-learning-materials/blob/main/slides/lmsys_1st_meetup_deepseek_mla.pdf) for more details.
|
||||
|
||||
### Data Parallelism Attention
|
||||
|
||||
**Description**: This optimization involves data parallelism (DP) for the MLA attention mechanism of DeepSeek Series Models, which allows for a significant reduction in the KV cache size, enabling larger batch sizes. Each DP worker independently handles different types of batches (prefill, decode, idle), which are then synchronized before and after processing through the Mixture-of-Experts (MoE) layer. If you do not use DP attention, KV cache will be duplicated among all TP ranks.
|
||||
|
||||
<p align="center">
|
||||
<img src="https://lmsys.org/images/blog/sglang_v0_4/dp_attention.svg" alt="Data Parallelism Attention for DeepSeek Series Models" />
|
||||
</p>
|
||||
|
||||
With data parallelism attention enabled, we have achieved up to **1.9x** decoding throughput improvement compared to the previous version.
|
||||
|
||||
<p align="center">
|
||||
<img src="https://lmsys.org/images/blog/sglang_v0_4/deepseek_coder_v2.svg" alt="Data Parallelism Attention Performance Comparison" />
|
||||
</p>
|
||||
|
||||
**Usage**:
|
||||
- Append `--enable-dp-attention --tp 8 --dp 8` to the server arguments when using 8 H200 GPUs. This optimization improves peak throughput in high batch size scenarios where the server is limited by KV cache capacity.
|
||||
- DP and TP attention can be flexibly combined. For example, to deploy DeepSeek-V3/R1 on 2 nodes with 8 H100 GPUs each, you can specify `--enable-dp-attention --tp 16 --dp 2`. This configuration runs attention with 2 DP groups, each containing 8 TP GPUs.
|
||||
|
||||
<Warning>
|
||||
Data parallelism attention is not recommended for low-latency, small-batch use cases. It is optimized for high-throughput scenarios with large batch sizes.
|
||||
</Warning>
|
||||
|
||||
**Reference**: Check [Blog](https://lmsys.org/blog/2024-12-04-sglang-v0-4/#data-parallelism-attention-for-deepseek-models).
|
||||
|
||||
### Multi-Node Tensor Parallelism
|
||||
|
||||
**Description**: For users with limited memory on a single node, SGLang supports serving DeepSeek Series Models, including DeepSeek V3, across multiple nodes using tensor parallelism. This approach partitions the model parameters across multiple GPUs or nodes to handle models that are too large for one node's memory.
|
||||
|
||||
**Usage**: Check [here](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-two-h2008-nodes-and-docker) for usage examples.
|
||||
|
||||
### Block-wise FP8
|
||||
|
||||
**Description**: SGLang implements block-wise FP8 quantization with two key optimizations:
|
||||
|
||||
- **Activation**: E4M3 format using per-token-per-128-channel sub-vector scales with online casting.
|
||||
|
||||
- **Weight**: Per-128x128-block quantization for better numerical stability.
|
||||
|
||||
- **DeepGEMM**: The [DeepGEMM](https://github.com/deepseek-ai/DeepGEMM) kernel library optimized for FP8 matrix multiplications.
|
||||
|
||||
**Usage**: The activation and weight optimization above are turned on by default for DeepSeek V3 models. DeepGEMM is enabled by default on NVIDIA Hopper/Blackwell GPUs and disabled by default on other devices. DeepGEMM can also be manually turned off by setting the environment variable `SGLANG_ENABLE_JIT_DEEPGEMM=0`.
|
||||
|
||||
<Tip>
|
||||
Before serving the DeepSeek model, precompile the DeepGEMM kernels to improve first-run performance. The precompilation process typically takes around 10 minutes to complete.
|
||||
</Tip>
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-V3 --tp 8 --trust-remote-code
|
||||
```
|
||||
|
||||
### Multi-token Prediction
|
||||
**Description**: SGLang implements DeepSeek V3 Multi-Token Prediction (MTP) based on [EAGLE speculative decoding](../advanced_features/speculative_decoding#EAGLE-Decoding). With this optimization, the decoding speed can be improved by **1.8x** for batch size 1 and **1.5x** for batch size 32 respectively on H200 TP8 setting.
|
||||
|
||||
**Usage**:
|
||||
Add `--speculative-algorithm EAGLE`. Other flags, like `--speculative-num-steps`, `--speculative-eagle-topk` and `--speculative-num-draft-tokens` are optional. For example:
|
||||
```text Output
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3-0324 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--trust-remote-code \
|
||||
--tp 8
|
||||
```
|
||||
- The default configuration for DeepSeek models is `--speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The best configuration for `--speculative-num-steps`, `--speculative-eagle-topk` and `--speculative-num-draft-tokens` can be searched with [bench_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py) script for given batch size. The minimum configuration is `--speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2`, which can achieve speedup for larger batch sizes.
|
||||
- Most MLA attention backends fully support MTP usage. See [MLA Backends](../advanced_features/attention_backend#mla-backends) for details.
|
||||
|
||||
<Note>
|
||||
To enable DeepSeek MTP for large batch sizes (>48), you need to adjust some parameters (Reference [this discussion](https://github.com/sgl-project/sglang/issues/4543#issuecomment-2737413756)):
|
||||
- Adjust `--max-running-requests` to a larger number. The default value is `48` for MTP. For larger batch sizes, you should increase this value beyond the default value.
|
||||
- Set `--cuda-graph-bs`. It's a list of batch sizes for cuda graph capture. The [default captured batch sizes for speculative decoding](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/server_args.py#L888-L895) is 48. You can customize this by including more batch sizes.
|
||||
</Note>
|
||||
|
||||
<Tip>
|
||||
To enable the experimental overlap scheduler for EAGLE speculative decoding, set the environment variable `SGLANG_ENABLE_SPEC_V2=1`. This can improve performance by enabling overlap scheduling between draft and verification stages.
|
||||
</Tip>
|
||||
|
||||
|
||||
### Reasoning Content for DeepSeek R1 & V3.1
|
||||
|
||||
See [Reasoning Parser](../advanced_features/separate_reasoning) and [Thinking Parameter for DeepSeek V3.1](./openai_api_completions#Example:-DeepSeek-V3-Models).
|
||||
|
||||
|
||||
### Function calling for DeepSeek Models
|
||||
|
||||
Add arguments `--tool-call-parser deepseekv3` and `--chat-template ./examples/chat_template/tool_chat_template_deepseekv3.jinja`(recommended) to enable this feature. For example (running on 1 * H20 node):
|
||||
|
||||
```text Output
|
||||
python3 -m sglang.launch_server \
|
||||
--model deepseek-ai/DeepSeek-V3-0324 \
|
||||
--tp 8 \
|
||||
--port 30000 \
|
||||
--host 0.0.0.0 \
|
||||
--mem-fraction-static 0.9 \
|
||||
--tool-call-parser deepseekv3 \
|
||||
--chat-template ./examples/chat_template/tool_chat_template_deepseekv3.jinja
|
||||
```
|
||||
|
||||
Sample Request:
|
||||
|
||||
```
|
||||
curl "http://127.0.0.1:30000/v1/chat/completions" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"temperature": 0, "max_tokens": 100, "model": "deepseek-ai/DeepSeek-V3-0324", "tools": [{"type": "function", "function": {"name": "query_weather", "description": "Get weather of a city, the user should supply a city first", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "The city, e.g. Beijing"}}, "required": ["city"]}}}], "messages": [{"role": "user", "content": "How'\''s the weather like in Qingdao today"}]}'
|
||||
```
|
||||
|
||||
Expected Response
|
||||
|
||||
```text Output
|
||||
{"id":"6501ef8e2d874006bf555bc80cddc7c5","object":"chat.completion","created":1745993638,"model":"deepseek-ai/DeepSeek-V3-0324","choices":[{"index":0,"message":{"role":"assistant","content":null,"reasoning_content":null,"tool_calls":[{"id":"0","index":null,"type":"function","function":{"name":"query_weather","arguments":"{\"city\": \"Qingdao\"}"}}]},"logprobs":null,"finish_reason":"tool_calls","matched_stop":null}],"usage":{"prompt_tokens":116,"total_tokens":138,"completion_tokens":22,"prompt_tokens_details":null}}
|
||||
|
||||
```
|
||||
Sample Streaming Request:
|
||||
```
|
||||
curl "http://127.0.0.1:30000/v1/chat/completions" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"temperature": 0, "max_tokens": 100, "model": "deepseek-ai/DeepSeek-V3-0324","stream":true,"tools": [{"type": "function", "function": {"name": "query_weather", "description": "Get weather of a city, the user should supply a city first", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "The city, e.g. Beijing"}}, "required": ["city"]}}}], "messages": [{"role": "user", "content": "How'\''s the weather like in Qingdao today"}]}'
|
||||
```
|
||||
Expected Streamed Chunks (simplified for clarity):
|
||||
```text Output
|
||||
data: {"choices":[{"delta":{"tool_calls":[{"function":{"arguments":"{\""}}]}}]}
|
||||
data: {"choices":[{"delta":{"tool_calls":[{"function":{"arguments":"city"}}]}}]}
|
||||
data: {"choices":[{"delta":{"tool_calls":[{"function":{"arguments":"\":\""}}]}}]}
|
||||
data: {"choices":[{"delta":{"tool_calls":[{"function":{"arguments":"Q"}}]}}]}
|
||||
data: {"choices":[{"delta":{"tool_calls":[{"function":{"arguments":"ing"}}]}}]}
|
||||
data: {"choices":[{"delta":{"tool_calls":[{"function":{"arguments":"dao"}}]}}]}
|
||||
data: {"choices":[{"delta":{"tool_calls":[{"function":{"arguments":"\"}"}}]}}]}
|
||||
data: {"choices":[{"delta":{"tool_calls":null}}], "finish_reason": "tool_calls"}
|
||||
data: [DONE]
|
||||
```
|
||||
The client needs to concatenate all arguments fragments to reconstruct the complete tool call:
|
||||
```text Output
|
||||
{"city": "Qingdao"}
|
||||
```
|
||||
|
||||
<Warning>
|
||||
1. Use a lower `"temperature"` value for better results.
|
||||
2. To receive more consistent tool call results, it is recommended to use `--chat-template examples/chat_template/tool_chat_template_deepseekv3.jinja`. It provides an improved unified prompt.
|
||||
</Warning>
|
||||
|
||||
|
||||
### Thinking Budget for DeepSeek R1
|
||||
|
||||
In SGLang, we can implement thinking budget with `CustomLogitProcessor`.
|
||||
|
||||
Launch a server with `--enable-custom-logit-processor` flag on.
|
||||
|
||||
```text Output
|
||||
python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-R1 --tp 8 --port 30000 --host 0.0.0.0 --mem-fraction-static 0.9 --disable-cuda-graph --reasoning-parser deepseek-r1 --enable-custom-logit-processor
|
||||
```
|
||||
|
||||
Sample Request:
|
||||
|
||||
```python Sample Request
|
||||
import openai
|
||||
from rich.pretty import pprint
|
||||
from sglang.srt.sampling.custom_logit_processor import DeepSeekR1ThinkingBudgetLogitProcessor
|
||||
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-R1",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Question: Is Paris the Capital of France?",
|
||||
}
|
||||
],
|
||||
max_tokens=1024,
|
||||
extra_body={
|
||||
"custom_logit_processor": DeepSeekR1ThinkingBudgetLogitProcessor().to_str(),
|
||||
"custom_params": {
|
||||
"thinking_budget": 512,
|
||||
},
|
||||
},
|
||||
)
|
||||
pprint(response)
|
||||
```
|
||||
|
||||
## FAQ
|
||||
|
||||
**Q: Model loading is taking too long, and I'm encountering an NCCL timeout. What should I do?**
|
||||
|
||||
A: If you're experiencing extended model loading times and an NCCL timeout, you can try increasing the timeout duration. Add the argument `--dist-timeout 3600` when launching your model. This will set the timeout to one hour, which often resolves the issue.
|
||||
@@ -1,601 +0,0 @@
|
||||
---
|
||||
title: "DeepSeek V3.2/GLM-5 Usage"
|
||||
metatags:
|
||||
description: "Deploy DeepSeek V3.2/GLM-5 with SGLang: DeepSeek Sparse Attention (DSA), long-context optimization, MTP speculative decoding, function calling. Supports H200, B200, MI300X, MI350."
|
||||
---
|
||||
DeepSeek-V3.2 model family equips DeepSeek-V3.1-Terminus with DeepSeek Sparse Attention (DSA) through continued training. With DSA, a fine-grained sparse attention mechanism powered by a lightning indexer, DeepSeek-V3.2 achieves efficiency improvements in long-context scenarios.
|
||||
|
||||
|
||||
Note: This document is originally written for the usage of [DeepSeek-V3.2-Exp](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp) model. The usage of [DeepSeek-V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2) or [DeepSeek-V3.2-Speciale](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Speciale) is the same as DeepSeek-V3.2-Exp except for the tool call parser. [GLM-5](https://huggingface.co/zai-org/GLM-5) model also applies DSA (DeepSeek Sparse Attention) structure, so it can share most of the usage here, except for the reasoning parser and tool call parser.
|
||||
|
||||
|
||||
## Installation
|
||||
|
||||
### Docker
|
||||
|
||||
```bash Command
|
||||
# H200/B200
|
||||
docker pull lmsysorg/sglang:latest
|
||||
|
||||
# MI350/MI355
|
||||
docker pull lmsysorg/sglang:v0.5.8-rocm700-mi35x
|
||||
|
||||
# MI300
|
||||
# v0.5.8-rocm700-mi30x does not include PR #17504. Prefer the newest MI30x ROCm
|
||||
# image tag from Docker Hub when available, or build from source (below).
|
||||
docker pull lmsysorg/sglang:v0.5.8-rocm700-mi30x
|
||||
|
||||
|
||||
# NPUs
|
||||
docker pull lmsysorg/sglang:dsv32-a2
|
||||
docker pull lmsysorg/sglang:dsv32-a3
|
||||
```
|
||||
|
||||
### Build From Source
|
||||
|
||||
```bash Command
|
||||
# Install SGLang
|
||||
git clone https://github.com/sgl-project/sglang
|
||||
cd sglang
|
||||
pip3 install pip --upgrade
|
||||
pip3 install -e "python"
|
||||
```
|
||||
|
||||
## Launch DeepSeek V3.2/GLM-5 with SGLang
|
||||
|
||||
To serve [DeepSeek-V3.2-Exp](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp) on 8xH200/B200 GPUs:
|
||||
|
||||
```bash Command
|
||||
# Launch with TP + DP (Recommended)
|
||||
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --dp 8 --enable-dp-attention
|
||||
|
||||
# Launch with EP + DP
|
||||
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --ep 8 --dp 8 --enable-dp-attention
|
||||
|
||||
# Launch with Pure TP
|
||||
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8
|
||||
|
||||
# Launch with TP on MI30x/MI35x
|
||||
python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --dsa-prefill-backend tilelang --dsa-decode-backend tilelang
|
||||
```
|
||||
|
||||
To serve GLM-5, just replace the `--model` argument with `zai-org/GLM-5-FP8`.
|
||||
|
||||
### Configuration Tips
|
||||
- **DP Attention**: To enable [DP Attention](../advanced_features/dp_dpa_smg_guide), please include `--enable-dp-attention --dp <dp-size>` in command. DP Attention is better for large concurrency scenarios.
|
||||
- **TP Attention**: Launching with TP attention is also supported. TP attention is better for low latency scenarios.
|
||||
- **Short-sequence MHA prefill (adaptive)**: For short prefill sequences (default threshold: **2048 tokens**), the DSA backend uses standard MHA automatically (no extra flags). On H200 (SM90) this path uses the FlashAttention variable-length kernel; on B200 (SM100) it uses TRT-LLM ragged MHA. MHA uses `MHA_ONE_SHOT` for best performance, which computes multi-head attention over all tokens (both cached prefix and newly extended tokens) in a single kernel invocation, avoiding the overhead of chunked KV cache processing. This achieves optimal throughput for short sequences where total sequence length fits within the chunk capacity limit.
|
||||
- **MHA prefill threshold relaxation**: To apply MHA attention to requests longer than 2048 tokens, please set the flag `SGLANG_DSA_PREFILL_DENSE_ATTN_KV_LEN_THRESHOLD` to a value larger than 2048 (`SGLANG_NSA_PREFILL_DENSE_ATTN_KV_LEN_THRESHOLD` is a deprecated alias). As threshold grows larger, the prefill performance can be improved, but at the cost of potential accuracy drop.
|
||||
- **Choices of Attention Kernels**: The attention backend is automatically set to `dsa` attention backend for DeepSeek V3.2 model. In this backend, different kernels for sparse prefilling/decoding are implemented, which can be specified by `--dsa-prefill-backend` and `--dsa-decode-backend` server arguments. The choices of dsa prefill/decode attention kernels include:
|
||||
- `flashmla_sparse`: `flash_mla_sparse_fwd` kernel from `flash_mla` library. Can run on both Hopper and Blackwell GPUs. It requires bf16 q, kv inputs.
|
||||
- `flashmla_kv`: `flash_mla_with_kvcache` kernel from `flash_mla` library. Can run on both Hopper and Blackwell GPUs. It requires bf16 q, fp8 k_cache inputs.
|
||||
- `flashmla_auto`: enables automatic selection of either `flashmla_sparse` or `flashmla_kv` kernel for prefill based on KV cache dtype, hardware, and heuristics. With BF16 KV cache, `flashmla_sparse` is always used on both Hopper and Blackwell. With FP8 KV cache: On Hopper (SM90), it unconditionally uses `flashmla_kv`; On Blackwell (SM100), it uses `flashmla_sparse` when `total_kv_tokens < total_q_tokens * 512`, otherwise falls back to `flashmla_kv`. The heuristics may need to be tuned if the performance of either kernel changes significantly.
|
||||
- `fa3`: `flash_attn_with_kvcache` kernel from `flash_attn` library. Can only run on Hopper GPUs. It requires bf16 q, kv inputs.
|
||||
- `tilelang`: `tilelang` implementation that can run on GPU, HPU and NPU.
|
||||
- `aiter`: Aiter kernel on AMD HPUs. Can only be used as decode kernel.
|
||||
- `trtllm`: `trtllm-mla` sparse kernel from flashinfer library. Only run on blackwell GPUs. It requires q,k,v to be uniformly bf16 or fp8_e4m3 format.
|
||||
- On the basis of performance benchmarks, the default configuration of DSA kernels on Hopper and Blackwell are set as follows :
|
||||
- Bfloat 16 kv cache: On Hopper, `flashmla_sparse` prefill attention, `fa3` decode attention; On Blackwell, `flashmla_sparse` prefill attention, `trtllm` decode attention
|
||||
- Float8_e4m3fn KV cache: On Hopper, `flashmla_kv` prefill attention, `flashmla_kv` decode attention; On Blackwell, `trtllm` prefill attention and `trtllm` decode attention.
|
||||
- **Index Cache**: Introduce in [this paper](https://arxiv.org/abs/2603.12201), IndexCache improves speed by reusing the result of indexer across different layers, only at cost of negligible accuracy loss. For **GLM-5** model, we recommend appending `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` to command for better tradeoff between speedup and performance.
|
||||
|
||||
## Multi-token Prediction
|
||||
SGLang implements Multi-Token Prediction (MTP) for DeepSeek V3.2 based on [EAGLE speculative decoding](../advanced_features/speculative_decoding#EAGLE-Decoding). With this optimization, the decoding speed can be improved significantly on small batch sizes. Please look at [this PR](https://github.com/sgl-project/sglang/pull/11652) for more information.
|
||||
|
||||
Example usage with DP Attention:
|
||||
```bash Command
|
||||
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --dp 8 --enable-dp-attention --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
|
||||
```
|
||||
|
||||
Example usage with Pure TP:
|
||||
```bash Command
|
||||
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
|
||||
```
|
||||
|
||||
- The best configuration for `--speculative-num-steps`, `--speculative-eagle-topk` and `--speculative-num-draft-tokens` can be searched with [bench_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py) script for given batch size. The minimum configuration is `--speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2`, which can achieve speedup for larger batch sizes.
|
||||
- The default value of `--max-running-requests` is set to `48` for MTP. For larger batch sizes, this value should be increased beyond the default value.
|
||||
|
||||
<Tip>
|
||||
To enable overlap scheduler for EAGLE speculative decoding, we recommend setting the environment variable `SGLANG_ENABLE_SPEC_V2=1`. This can improve performance by enabling overlap scheduling between draft and verification stages.
|
||||
</Tip>
|
||||
|
||||
|
||||
## Function Calling and Reasoning Parser
|
||||
The usage of function calling and reasoning parser is the same as DeepSeek V3.1. Please refer to [Reasoning Parser](../advanced_features/separate_reasoning) and [Tool Parser](../advanced_features/tool_parser) documents.
|
||||
|
||||
To launch `DeepSeek-V3.2-Exp` with function calling and reasoning parser:
|
||||
> Note: It is recommended to specify the chat-template, ensuring that you are within the sglang's root directory.
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3.2-Exp \
|
||||
--trust-remote-code \
|
||||
--tp-size 8 --dp-size 8 --enable-dp-attention \
|
||||
--tool-call-parser deepseekv31 \
|
||||
--reasoning-parser deepseek-v3 \
|
||||
--chat-template ./examples/chat_template/tool_chat_template_deepseekv32.jinja
|
||||
```
|
||||
|
||||
To launch `DeepSeek-V3.2` with function calling and reasoning parser:
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3.2 \
|
||||
--trust-remote-code \
|
||||
--tp-size 8 --dp-size 8 --enable-dp-attention \
|
||||
--tool-call-parser deepseekv32 \
|
||||
--reasoning-parser deepseek-v3
|
||||
```
|
||||
|
||||
`DeepSeek-V3.2-Speciale` does not support tool calling, so it can only be launched with the reasoning parser:
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3.2-Speciale \
|
||||
--trust-remote-code \
|
||||
--tp-size 8 --dp-size 8 --enable-dp-attention \
|
||||
--reasoning-parser deepseek-v3
|
||||
```
|
||||
|
||||
To launch `GLM-5` with function calling and reasoning parser:
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-5-FP8 \
|
||||
--tp-size 8 --dp-size 8 --enable-dp-attention \
|
||||
--tool-call-parser glm47 \
|
||||
--reasoning-parser glm45 \
|
||||
```
|
||||
|
||||
## NVFP4 Checkpoint
|
||||
|
||||
To launch deepseek v3.2 [NVFP4 checkpoint](https://huggingface.co/nvidia/DeepSeek-V3.2-NVFP4) on Blackwell devices, the user needs to specify the quantization method as `modelopt_fp4`, and moe runner backend as one of `flashinfer_trtllm`(recommended), `flashinfer_cutlass` and `flashinfer_cutedsl`. Any other usage (parallelism, reasoning parser, ...) is the same as FP8 checkpoint.
|
||||
|
||||
An example launching command can be:
|
||||
```bash Command
|
||||
python -m sglang.launch_server --model nvidia/DeepSeek-V3.2-NVFP4 --tp 4 --quantization modelopt_fp4 --moe-runner-backend flashinfer_trtllm --tool-call-parser deepseekv32 --reasoning-parser deepseek-v3
|
||||
```
|
||||
|
||||
## PD Disaggregation
|
||||
|
||||
Prefill Command:
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3.2-Exp \
|
||||
--disaggregation-mode prefill \
|
||||
--host $LOCAL_IP \
|
||||
--port $PORT \
|
||||
--tp 8 \
|
||||
--dp 8 \
|
||||
--enable-dp-attention \
|
||||
--dist-init-addr ${HOST}:${DIST_PORT} \
|
||||
--trust-remote-code \
|
||||
--disaggregation-bootstrap-port 8998 \
|
||||
--mem-fraction-static 0.9 \
|
||||
```
|
||||
|
||||
Decode command:
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3.2-Exp \
|
||||
--disaggregation-mode decode \
|
||||
--host $LOCAL_IP \
|
||||
--port $PORT \
|
||||
--tp 8 \
|
||||
--dp 8 \
|
||||
--enable-dp-attention \
|
||||
--dist-init-addr ${HOST}:${DIST_PORT} \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.9 \
|
||||
```
|
||||
|
||||
Router command:
|
||||
```bash Command
|
||||
python -m sglang_router.launch_router --pd-disaggregation \
|
||||
--prefill $PREFILL_ADDR 8998 \
|
||||
--decode $DECODE_ADDR \
|
||||
--host 127.0.0.1 \
|
||||
--port 8000 \
|
||||
```
|
||||
|
||||
If you need more advanced deployment methods or production-ready deployment methods, such as RBG or LWS-based deployment, please refer to [references/multi_node_deployment/rbg_pd/deepseekv32_pd.md](../references/multi_node_deployment/rbg_pd/deepseekv32_pd). Additionally, you can also find startup commands for DeepEP-based EP parallelism in the aforementioned documentation.
|
||||
|
||||
|
||||
## Benchmarking Results
|
||||
|
||||
### Accuracy Test with `gsm8k`
|
||||
A simple accuracy benchmark can be tested with `gsm8k` dataset:
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1319 --parallel 1319
|
||||
```
|
||||
|
||||
The result is 0.956, which matches our expectation:
|
||||
```bash Command
|
||||
Accuracy: 0.956
|
||||
Invalid: 0.000
|
||||
Latency: 25.109 s
|
||||
Output throughput: 5226.235 token/s
|
||||
```
|
||||
|
||||
To test long-context accuracy, run gsm8k with `--num-shots 20`. The results are very close to the 8 shots results:
|
||||
```text Output
|
||||
Accuracy: 0.956
|
||||
Invalid: 0.000
|
||||
Latency: 29.545 s
|
||||
Output throughput: 4418.617 token/s
|
||||
```
|
||||
|
||||
|
||||
### Accuracy Test with `gpqa-diamond`
|
||||
|
||||
Accuracy benchmark on long context can be tested on GPQA-diamond dataset with long output tokens and thinking enabled:
|
||||
```bash Command
|
||||
python3 -m sglang.test.run_eval --port 30000 --eval-name gpqa --num-examples 198 --max-tokens 128000 --repeat 8 --thinking-mode deepseek-v3
|
||||
```
|
||||
|
||||
The mean accuracy over 8 runs shows 0.797, which matches the number 0.799 in official tech report.
|
||||
```bash Command
|
||||
Repeat: 8, mean: 0.797
|
||||
Scores: ['0.808', '0.798', '0.808', '0.798', '0.783', '0.788', '0.803', '0.793']
|
||||
```
|
||||
|
||||
For DeepSeek V3.2, DeepSeek recommends setting the sampling parameters to temperature = 1.0, top_p = 0.95:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.test.run_eval --port 30000 --eval-name gpqa --num-examples 198 --max-tokens 128000 --repeat 8 --top-p 0.95 --temperature 1.0 --thinking-mode deepseek-v3
|
||||
|
||||
Repeat: 8, mean: 0.840
|
||||
Scores: ['0.848', '0.808', '0.848', '0.838', '0.879', '0.813', '0.838', '0.848']
|
||||
```
|
||||
which matches the official score, 0.824, as reported in the [DeepSeek-V3.2 technical report](https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/main/assets/paper.pdf).
|
||||
|
||||
### Accuracy Test with `aime 2025`
|
||||
|
||||
Prepare the environment by installing NeMo-Skills in the docker or your own virtual environment:
|
||||
|
||||
```
|
||||
pip install git+https://github.com/NVIDIA/NeMo-Skills.git --ignore-installed blinker
|
||||
```
|
||||
|
||||
Then launch the SGLang server:
|
||||
```text Output
|
||||
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --dp 8 --enable-dp-attention
|
||||
```
|
||||
|
||||
**For `DeepSeek-V3.2` and `DeepSeek-V3.2-Speciale`**:
|
||||
|
||||
```text Output
|
||||
python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3.2 --trust-remote-code --tp-size 8 --dp-size 8 --enable-dp-attention --tool-call-parser deepseekv32 --reasoning-parser deepseek-v3
|
||||
```
|
||||
|
||||
Run the following script to evaluate AIME 2025:
|
||||
```text Output
|
||||
#! /bin/bash
|
||||
export NEMO_SKILLS_DISABLE_UNCOMMITTED_CHANGES_CHECK=1
|
||||
|
||||
ns prepare_data aime25
|
||||
|
||||
PORT=30000
|
||||
BACKEND=sglang
|
||||
MODEL="deepseek-ai/DeepSeek-V3.2-Exp" # Should be changed to the model name
|
||||
MODEL_NAME="dsv32-fp8"
|
||||
|
||||
echo "Starting AIME25 evaluation with model $MODEL on port $PORT using backend $BACKEND..."
|
||||
ns eval \
|
||||
--benchmarks=aime25:4 \
|
||||
--server_type=$BACKEND \
|
||||
--model=$MODEL \
|
||||
--server_address=http://localhost:${PORT}/v1 \
|
||||
--output_dir=nemo_skills_aime25_${MODEL_NAME}_output_${BACKEND}_$(date +%Y%m%d_%H%M%S) \
|
||||
++chat_template_kwargs.thinking=true \
|
||||
++inference.temperature=1.0 \
|
||||
++inference.top_p=0.95 \
|
||||
++inference.tokens_to_generate=64000
|
||||
# ++inference.tokens_to_generate=120000 for Speciale model
|
||||
```
|
||||
|
||||
Test results (8*B200):
|
||||
|
||||
DeepSeek-V3.2-Exp:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "16%"}} />
|
||||
<col style={{width: "16%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>evaluation_mode</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>num_entries</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>avg_tokens</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>gen_seconds</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>symbolic_correct</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>no_answer</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@1[avg-of-4]</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>30</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>15040</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1673</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>87.50% ± 1.67%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.00%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>majority@4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>30</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>15040</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1673</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>90.00%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.00%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>30</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>15040</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1673</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>90.00%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.00%</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
DeepSeek-V3.2:
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "16%"}} />
|
||||
<col style={{width: "16%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>evaluation_mode</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>num_entries</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>avg_tokens</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>gen_seconds</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>symbolic_correct</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>no_answer</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@1[avg-of-4]</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>30</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>13550</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1632</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>92.50% ± 1.67%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.00%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>majority@4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>30</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>13550</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1632</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>94.71%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.00%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>30</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>13550</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1632</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>96.67%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.00%</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
DeepSeek-V3.2-Speciale:
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "16%"}} />
|
||||
<col style={{width: "16%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>evaluation_mode</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>num_entries</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>avg_tokens</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>gen_seconds</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>symbolic_correct</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>no_answer</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@1[avg-of-4]</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>30</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>24155</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>3583</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>95.00% ± 1.92%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.00%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>majority@4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>30</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>24155</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>3583</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>95.83%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.00%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>30</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>24155</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>3583</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>100.00%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.00%</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
## DSA long sequence context parallel optimization(experimental)
|
||||
|
||||
**Note: This feature is only verified on Hopper machines**
|
||||
|
||||
For context parallel in DeepSeek V3.2 model, we provide two different modes of splitting tokens, which can be controlled with argument `--dsa-prefill-cp-mode`.
|
||||
|
||||
### In sequence splitting
|
||||
|
||||
The first mode can be enabled by `--dsa-prefill-cp-mode in-seq-split`. This mode implements context parallel for DSA by splitting the sequence uniformly between context parallel ranks. At attention stage, each cp rank computes the indexer results of sharded sequence, and collects the whole kv cache through all gather operator. Add `attn_cp_size` for communication group for context parallel.
|
||||
|
||||
Note that the in-sequence splitting mode has the following restrictions:
|
||||
- The batch size is restricted to 1 for prefill batches
|
||||
- `moe_dense_tp_size=1`, `moe_a2a_backend = "deepep"`
|
||||
- To ensure `cp_size > 1`, the passed in `tp_size` must be larger than `dp_size`
|
||||
|
||||
For more details, please refer to PR https://github.com/sgl-project/sglang/pull/12065.
|
||||
|
||||
Example:
|
||||
```bash Command
|
||||
# In-seq splitting mode launched with EP + DP
|
||||
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --ep 8 --dp 2 --enable-dp-attention --enable-dsa-prefill-context-parallel --attn-cp-size 4 --dsa-prefill-cp-mode in-seq-split --max-running-requests 32
|
||||
```
|
||||
|
||||
### Round robin splitting (default setting)
|
||||
|
||||
This mode can be enabled by specifying the parameter `--dsa-prefill-cp-mode round-robin-split`, which distributes tokens across ranks based on `token_idx % cp_size`.
|
||||
|
||||
In this scenario, compared to the in-sequence splitting method, it additionally supports the fused MoE backend (the fused MoE backend may deliver better performance than DeepEP in single-machine scenarios), FP8 KV-cache, and multi-batch prefill inference. However, it cannot be enabled with DP attention together.
|
||||
|
||||
For more details, please refer to PR https://github.com/sgl-project/sglang/pull/13959.
|
||||
|
||||
Example usage:
|
||||
```bash Command
|
||||
# Launch with FusedMoe + CP8
|
||||
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --enable-dsa-prefill-context-parallel --attn-cp-size 8 --dsa-prefill-cp-mode round-robin-split --max-running-requests 32
|
||||
```
|
||||
### Pipeline Parallel + Context Parallel (PP + CP)
|
||||
|
||||
This mode combines Pipeline Parallelism (PP) and Context Parallelism (CP) to scale across multiple nodes, which can achieve better throughput and Time To First Token (TTFT). Note that this method has only been tested on H20 96G.
|
||||
|
||||
#### Standard Usage
|
||||
|
||||
To launch with PP=2 and CP (via `round-robin-split` mode) on 2 nodes. This configuration uses the fused MoE kernel by default, which generally provides better performance.
|
||||
|
||||
For related development details, please refer to:
|
||||
- Fused MoE + CP support: [PR #13959](https://github.com/sgl-project/sglang/pull/13959)
|
||||
- PP + CP support: [Issue #15358](https://github.com/sgl-project/sglang/issues/15358) and [PR #16380](https://github.com/sgl-project/sglang/pull/16380)
|
||||
|
||||
Node 0:
|
||||
```bash Command
|
||||
export SGLANG_PP_LAYER_PARTITION=30,31
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3.2-Exp \
|
||||
--nnodes 2 --node-rank 0 \
|
||||
--dist-init-addr <HEAD_NODE_IP>:62001 \
|
||||
--tp 8 --pp-size 2 \
|
||||
--dp-size 1 --moe-dense-tp-size 1 \
|
||||
--enable-dsa-prefill-context-parallel \
|
||||
--attn-cp-size 8 \
|
||||
--dsa-prefill-cp-mode round-robin-split \
|
||||
--trust-remote-code \
|
||||
--disable-radix-cache \
|
||||
--mem-fraction-static 0.8 \
|
||||
--max-running-requests 128 \
|
||||
--chunked-prefill-size 16384 \
|
||||
--cuda-graph-max-bs 8 \
|
||||
--page-size 64 \
|
||||
--watchdog-timeout 3600 \
|
||||
--host 0.0.0.0 --port 8000 \
|
||||
--tool-call-parser deepseekv32
|
||||
```
|
||||
|
||||
Node 1:
|
||||
```bash Command
|
||||
export SGLANG_PP_LAYER_PARTITION=30,31
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3.2-Exp \
|
||||
--nnodes 2 --node-rank 1 \
|
||||
--dist-init-addr <HEAD_NODE_IP>:62001 \
|
||||
--tp 8 --pp-size 2 \
|
||||
--dp-size 1 --moe-dense-tp-size 1 \
|
||||
--enable-dsa-prefill-context-parallel \
|
||||
--attn-cp-size 8 \
|
||||
--dsa-prefill-cp-mode round-robin-split \
|
||||
--trust-remote-code \
|
||||
--disable-radix-cache \
|
||||
--mem-fraction-static 0.8 \
|
||||
--max-running-requests 128 \
|
||||
--chunked-prefill-size 16384 \
|
||||
--cuda-graph-max-bs 8 \
|
||||
--page-size 64 \
|
||||
--watchdog-timeout 3600 \
|
||||
--host 0.0.0.0 --port 8000 \
|
||||
--tool-call-parser deepseekv32
|
||||
```
|
||||
|
||||
#### PD Disaggregation with PP + CP
|
||||
|
||||
If using PD (Prefill-Decode) Disaggregation, the Prefill nodes can be configured with PP + CP as follows.
|
||||
|
||||
Prefill Node 0:
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3.2-Exp \
|
||||
--served-model-name deepseek-v32 \
|
||||
--nnodes 2 --node-rank 0 \
|
||||
--dist-init-addr <PREFILL_HEAD_IP>:20102 \
|
||||
--tp 8 --pp-size 2 \
|
||||
--dp-size 1 --moe-dense-tp-size 1 \
|
||||
--enable-dsa-prefill-context-parallel \
|
||||
--attn-cp-size 8 \
|
||||
--dsa-prefill-cp-mode round-robin-split \
|
||||
--disaggregation-ib-device mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3 \
|
||||
--trust-remote-code \
|
||||
--disable-radix-cache \
|
||||
--max-running-requests 512 \
|
||||
--chunked-prefill-size 4096 \
|
||||
--context-length 131072 \
|
||||
--mem-fraction-static 0.9 \
|
||||
--page-size 64 \
|
||||
--enable-metrics \
|
||||
--collect-tokens-histogram \
|
||||
--tokenizer-worker-num 8 \
|
||||
--host 0.0.0.0 --port 30000
|
||||
```
|
||||
|
||||
Prefill Node 1:
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3.2-Exp \
|
||||
--served-model-name deepseek-v32-prefill \
|
||||
--nnodes 2 --node-rank 1 \
|
||||
--dist-init-addr <PREFILL_HEAD_IP>:20102 \
|
||||
--tp 8 --pp-size 2 \
|
||||
--dp-size 1 --moe-dense-tp-size 1 \
|
||||
--enable-dsa-prefill-context-parallel \
|
||||
--attn-cp-size 8 \
|
||||
--dsa-prefill-cp-mode round-robin-split \
|
||||
--disaggregation-ib-device mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3 \
|
||||
--trust-remote-code \
|
||||
--disable-radix-cache \
|
||||
--max-running-requests 512 \
|
||||
--chunked-prefill-size 4096 \
|
||||
--context-length 131072 \
|
||||
--mem-fraction-static 0.9 \
|
||||
--page-size 64 \
|
||||
--enable-metrics \
|
||||
--collect-tokens-histogram \
|
||||
--tokenizer-worker-num 8 \
|
||||
--host 0.0.0.0 --port 30000
|
||||
```
|
||||
|
||||
For the Decode nodes, it is recommended to use the **EP mode**.
|
||||
|
||||
## HiSparse: Hierarchical Sparse Attention for DSA (experimental)
|
||||
|
||||
HiSparse reduces per-request GPU memory during decode by keeping only a small "hot" KV buffer on GPU while storing complete KV data in CPU pinned memory. A CUDA kernel dynamically swaps in the top-k most relevant KV entries from host memory on each decode step. This enables significantly higher decode concurrency for long-context DSA models.
|
||||
|
||||
HiSparse currently requires PD disaggregation mode and is enabled on the decode instance only. For detailed design, configuration, and deployment instructions, see the [HiSparse Guide](../advanced_features/hisparse_guide).
|
||||
@@ -1,75 +0,0 @@
|
||||
---
|
||||
title: "Launch GLM-4.5 / GLM-4.6 / GLM-4.7 with SGLang"
|
||||
metatags:
|
||||
description: "Deploy GLM-4.5/4.6/4.7 models with SGLang: FP8 inference, EAGLE speculative decoding, function calling support. Optimized for H100/H200 GPUs."
|
||||
---
|
||||
## Launch GLM-4.5 / GLM-4.6 / GLM-4.7 with SGLang
|
||||
|
||||
To serve GLM-4.5 / GLM-4.6 FP8 models on 8xH100/H200 GPUs:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server --model zai-org/GLM-4.6-FP8 --tp 8
|
||||
```
|
||||
|
||||
### EAGLE Speculative Decoding
|
||||
|
||||
**Description**: SGLang has supported GLM-4.5 / GLM-4.6 models
|
||||
with [EAGLE speculative decoding](../advanced_features/speculative_decoding#EAGLE-Decoding).
|
||||
|
||||
**Usage**:
|
||||
Add arguments `--speculative-algorithm`, `--speculative-num-steps`, `--speculative-eagle-topk` and
|
||||
`--speculative-num-draft-tokens` to enable this feature. For example:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path zai-org/GLM-4.6-FP8 \
|
||||
--tp-size 8 \
|
||||
--tool-call-parser glm45 \
|
||||
--reasoning-parser glm45 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--mem-fraction-static 0.9 \
|
||||
--served-model-name glm-4.6-fp8 \
|
||||
--enable-custom-logit-processor
|
||||
```
|
||||
|
||||
<Tip>
|
||||
To enable the experimental overlap scheduler for EAGLE speculative decoding, set the environment variable `SGLANG_ENABLE_SPEC_V2=1`. This can improve performance by enabling overlap scheduling between draft and verification stages.
|
||||
</Tip>
|
||||
|
||||
### Thinking Budget for GLM-4.5 / GLM-4.6
|
||||
**Note**: For GLM-4.7, `--tool-call-parser` should be set to `glm47`, for GLM-4.5 and GLM-4.6, it should be set to `glm45`.
|
||||
|
||||
In SGLang, we can implement thinking budget with `CustomLogitProcessor`.
|
||||
|
||||
Launch a server with `--enable-custom-logit-processor` flag on.
|
||||
|
||||
Sample Request:
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
from rich.pretty import pprint
|
||||
from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor
|
||||
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.6",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Question: Is Paris the Capital of France?",
|
||||
}
|
||||
],
|
||||
max_tokens=1024,
|
||||
extra_body={
|
||||
"custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(),
|
||||
"custom_params": {
|
||||
"thinking_budget": 512,
|
||||
},
|
||||
},
|
||||
)
|
||||
pprint(response)
|
||||
```
|
||||
@@ -1,139 +0,0 @@
|
||||
---
|
||||
title: "GLM-4.6V / GLM-4.5V Usage"
|
||||
metatags:
|
||||
description: "Deploy GLM-4.6V/4.5V vision models with SGLang: FP8 and BF16 modes, expert parallelism, video understanding. Supports H100, H200, A100 GPUs."
|
||||
---
|
||||
## Launch commands for SGLang
|
||||
|
||||
Below are suggested launch commands tailored for different hardware / precision modes
|
||||
|
||||
### FP8 (quantised) mode
|
||||
|
||||
For high memory-efficiency and latency optimized deployments (e.g., on H100, H200) where FP8 checkpoint is supported:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path zai-org/GLM-4.6V-FP8 \
|
||||
--tp 2 \
|
||||
--ep 2 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--keep-mm-feature-on-device
|
||||
```
|
||||
|
||||
### Non-FP8 (BF16 / full precision) mode
|
||||
For deployments on A100/H100 where BF16 is used (or FP8 snapshot not used):
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path zai-org/GLM-4.6V \
|
||||
--tp 4 \
|
||||
--ep 4 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
## Hardware-specific notes / recommendations
|
||||
|
||||
- On H100 with FP8: Use the FP8 checkpoint for best memory efficiency.
|
||||
- On A100 / H100 with BF16 (non-FP8): It’s recommended to use `--mm-max-concurrent-calls` to control parallel throughput and GPU memory usage during image/video inference.
|
||||
- On H200 & B200: The model can be run “out of the box”, supporting full context length plus concurrent image + video processing.
|
||||
|
||||
## Sending Image/Video Requests
|
||||
|
||||
### Image input:
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
url = f"http://localhost:30000/v1/chat/completions"
|
||||
|
||||
data = {
|
||||
"model": "zai-org/GLM-4.6V",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "What’s in this image?"},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true"
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
"max_tokens": 300,
|
||||
}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
print(response.text)
|
||||
```
|
||||
|
||||
### Video Input:
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
url = f"http://localhost:30000/v1/chat/completions"
|
||||
|
||||
data = {
|
||||
"model": "zai-org/GLM-4.6V",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "What’s happening in this video?"},
|
||||
{
|
||||
"type": "video_url",
|
||||
"video_url": {
|
||||
"url": "https://github.com/sgl-project/sgl-test-files/raw/refs/heads/main/videos/jobs_presenting_ipod.mp4"
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
"max_tokens": 300,
|
||||
}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
print(response.text)
|
||||
```
|
||||
|
||||
## Important Server Parameters and Flags
|
||||
|
||||
When launching the model server for **multimodal support**, you can use the following command-line arguments to fine-tune performance and behavior:
|
||||
|
||||
- `--mm-attention-backend`: Specify multimodal attention backend. Eg. `fa3`(Flash Attention 3)
|
||||
- `--mm-max-concurrent-calls <value>`: Specifies the **maximum number of concurrent asynchronous multimodal data processing calls** allowed on the server. Use this to control parallel throughput and GPU memory usage during image/video inference.
|
||||
- `--mm-per-request-timeout <seconds>`: Defines the **timeout duration (in seconds)** for each multimodal request. If a request exceeds this time limit (e.g., for very large video inputs), it will be automatically terminated.
|
||||
- `--keep-mm-feature-on-device`: Instructs the server to **retain multimodal feature tensors on the GPU** after processing. This avoids device-to-host (D2H) memory copies and improves performance for repeated or high-frequency inference workloads.
|
||||
- `--mm-enable-dp-encoder`: Placing the ViT in data parallel while keeping the LLM in tensor parallel consistently lowers TTFT and boosts end-to-end throughput.
|
||||
- `SGLANG_USE_CUDA_IPC_TRANSPORT=1`: Shared memory pool based CUDA IPC for multi-modal data transport. For significantly improving e2e latency.
|
||||
|
||||
### Example usage with the above optimizations:
|
||||
```bash Command
|
||||
SGLANG_USE_CUDA_IPC_TRANSPORT=1 \
|
||||
SGLANG_VLM_CACHE_SIZE_MB=0 \
|
||||
python -m sglang.launch_server \
|
||||
--model-path zai-org/GLM-4.6V \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--trust-remote-code \
|
||||
--tp-size 8 \
|
||||
--enable-cache-report \
|
||||
--log-level info \
|
||||
--max-running-requests 64 \
|
||||
--mem-fraction-static 0.65 \
|
||||
--chunked-prefill-size 8192 \
|
||||
--attention-backend fa3 \
|
||||
--mm-attention-backend fa3 \
|
||||
--mm-enable-dp-encoder \
|
||||
--enable-metrics
|
||||
```
|
||||
|
||||
### Thinking Budget for GLM-4.5V / GLM-4.6V
|
||||
|
||||
In SGLang, we can implement thinking budget with `CustomLogitProcessor`.
|
||||
|
||||
Launch a server with the `--enable-custom-logit-processor` flag. Then, use `Glm4MoeThinkingBudgetLogitProcessor` in the request, similar to the `GLM-4.6` example in [glm45.md](./glm45).
|
||||
@@ -1,181 +0,0 @@
|
||||
---
|
||||
title: "GPT OSS Usage"
|
||||
metatags:
|
||||
description: "Deploy GPT-OSS with SGLang: OpenAI Responses API compatible, built-in tools for web search and Python execution, reasoning levels, MCP tool server support."
|
||||
---
|
||||
Please refer to [#8833](https://github.com/sgl-project/sglang/issues/8833).
|
||||
|
||||
## Responses API & Built-in Tools
|
||||
|
||||
### Responses API
|
||||
|
||||
GPT‑OSS is compatible with the OpenAI Responses API. Use `client.responses.create(...)` with `model`, `instructions`, `input`, and optional `tools` to enable built‑in tool use. You can set reasoning level via `instructions`, e.g., "Reasoning: high" (also supports "medium" and "low") — levels: low (fast), medium (balanced), high (deep).
|
||||
|
||||
### Built-in Tools
|
||||
|
||||
GPT‑OSS can call built‑in tools for web search and Python execution. You can use the demo tool server or connect to external MCP tool servers.
|
||||
|
||||
#### Python Tool
|
||||
|
||||
- Executes short Python snippets for calculations, parsing, and quick scripts.
|
||||
- By default runs in a Docker-based sandbox. To run on the host, set `PYTHON_EXECUTION_BACKEND=UV` (this executes model-generated code locally; use with care).
|
||||
- Ensure Docker is available if you are not using the UV backend. It is recommended to run `docker pull python:3.11` in advance.
|
||||
|
||||
#### Web Search Tool
|
||||
|
||||
- Uses the Exa backend for web search.
|
||||
- Requires an Exa API key; set `EXA_API_KEY` in your environment. Create a key at `https://exa.ai`.
|
||||
|
||||
### Tool & Reasoning Parser
|
||||
|
||||
- We support OpenAI Reasoning and Tool Call parser, as well as our SGLang native api for tool call and reasoning. Refer to [reasoning parser](../advanced_features/separate_reasoning) and [tool call parser](../advanced_features/tool_parser) for more details.
|
||||
|
||||
|
||||
## Notes
|
||||
|
||||
- Use **Python 3.12** for the demo tools. And install the required `gpt-oss` packages.
|
||||
- The default demo integrates the web search tool (Exa backend) and a demo Python interpreter via Docker.
|
||||
- For search, set `EXA_API_KEY`. For Python execution, either have Docker available or set `PYTHON_EXECUTION_BACKEND=UV`.
|
||||
|
||||
Examples:
|
||||
```bash Command
|
||||
export EXA_API_KEY=YOUR_EXA_KEY
|
||||
# Optional: run Python tool locally instead of Docker (use with care)
|
||||
export PYTHON_EXECUTION_BACKEND=UV
|
||||
```
|
||||
|
||||
Launch the server with the demo tool server:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path openai/gpt-oss-120b \
|
||||
--tool-server demo \
|
||||
--tp 2
|
||||
```
|
||||
|
||||
For production usage, sglang can act as an MCP client for multiple services. An [example tool server](https://github.com/openai/gpt-oss/tree/main/gpt-oss-mcp-server) is provided. Start the servers and point sglang to them:
|
||||
```bash Command
|
||||
mcp run -t sse browser_server.py:mcp
|
||||
mcp run -t sse python_server.py:mcp
|
||||
|
||||
python -m sglang.launch_server ... --tool-server ip-1:port-1,ip-2:port-2
|
||||
```
|
||||
The URLs should be MCP SSE servers that expose server information and well-documented tools. These tools are added to the system prompt so the model can use them.
|
||||
|
||||
## Speculative Decoding
|
||||
|
||||
SGLang supports speculative decoding for GPT-OSS models using EAGLE3 algorithm. This can significantly improve decoding speed, especially for small batch sizes.
|
||||
|
||||
**Usage**:
|
||||
Add `--speculative-algorithm EAGLE3` along with the draft model path.
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path openai/gpt-oss-120b \
|
||||
--speculative-algorithm EAGLE3 \
|
||||
--speculative-draft-model-path lmsys/EAGLE3-gpt-oss-120b-bf16 \
|
||||
--tp 2
|
||||
```
|
||||
|
||||
<Tip>
|
||||
To enable the experimental overlap scheduler for EAGLE3 speculative decoding, set the environment variable `SGLANG_ENABLE_SPEC_V2=1`. This can improve performance by enabling overlap scheduling between draft and verification stages.
|
||||
</Tip>
|
||||
|
||||
### Quick Demo
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="sk-123456"
|
||||
)
|
||||
|
||||
tools = [
|
||||
{"type": "code_interpreter"},
|
||||
{"type": "web_search_preview"},
|
||||
]
|
||||
|
||||
# Reasoning level example
|
||||
response = client.responses.create(
|
||||
model="openai/gpt-oss-120b",
|
||||
instructions="You are a helpful assistant."
|
||||
reasoning_effort="high" # Supports high, medium, or low
|
||||
input="In one sentence, explain the transformer architecture.",
|
||||
)
|
||||
print("====== reasoning: high ======")
|
||||
print(response.output_text)
|
||||
|
||||
# Test python tool
|
||||
response = client.responses.create(
|
||||
model="openai/gpt-oss-120b",
|
||||
instructions="You are a helfpul assistant, you could use python tool to execute code.",
|
||||
input="Use python tool to calculate the sum of 29138749187 and 29138749187", # 58,277,498,374
|
||||
tools=tools
|
||||
)
|
||||
print("====== test python tool ======")
|
||||
print(response.output_text)
|
||||
|
||||
# Test browser tool
|
||||
response = client.responses.create(
|
||||
model="openai/gpt-oss-120b",
|
||||
instructions="You are a helfpul assistant, you could use browser to search the web",
|
||||
input="Search the web for the latest news about Nvidia stock price",
|
||||
tools=tools
|
||||
)
|
||||
print("====== test browser tool ======")
|
||||
print(response.output_text)
|
||||
```
|
||||
|
||||
Example output:
|
||||
```text Output
|
||||
====== test python tool ======
|
||||
The sum of 29,138,749,187 and 29,138,749,187 is **58,277,498,374**.
|
||||
====== test browser tool ======
|
||||
**Recent headlines on Nvidia (NVDA) stock**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Date (2025)</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Source</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Key news points</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Stock‑price detail</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**May 13**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Reuters</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The market data page shows Nvidia trading “higher” at **$116.61** with no change from the previous close.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>**$116.61** – latest trade (delayed ≈ 15 min)【14†L34-L38】</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Aug 18**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>CNBC</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Morgan Stanley kept an **overweight** rating and lifted its price target to **$206** (up from $200), implying a 14 % upside from the Friday close. The firm notes Nvidia shares have already **jumped 34 % this year**.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No exact price quoted, but the article signals strong upside expectations【9†L27-L31】</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Aug 20**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The Motley Fool</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Nvidia is set to release its Q2 earnings on Aug 27. The article lists the **current price of $175.36**, down 0.16 % on the day (as of 3:58 p.m. ET).</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>**$175.36** – current price on Aug 20【10†L12-L15】【10†L53-L57】</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**What the news tells us**
|
||||
|
||||
* Nvidia’s share price has risen sharply this year – up roughly a third according to Morgan Stanley – and analysts are still raising targets (now $206).
|
||||
* The most recent market quote (Reuters, May 13) was **$116.61**, but the stock has surged since then, reaching **$175.36** by mid‑August.
|
||||
* Upcoming earnings on **Aug 27** are a focal point; both the Motley Fool and Morgan Stanley expect the results could keep the rally going.
|
||||
|
||||
**Bottom line:** Nvidia’s stock is on a strong upward trajectory in 2025, with price targets climbing toward $200‑$210 and the market price already near $175 as of late August.
|
||||
|
||||
```
|
||||
@@ -1,106 +0,0 @@
|
||||
---
|
||||
title: "Kimi-K2.5 Usage"
|
||||
metatags:
|
||||
description: "Deploy Kimi-K2.5 with SGLang: 1T-parameter multimodal MoE model, 256K context, MLA attention, MoonViT vision encoder, thinking and instant modes, tool calling support."
|
||||
---
|
||||
[Kimi-K2.5](https://huggingface.co/moonshotai/Kimi-K2.5) is Moonshot AI's open-source, native multimodal, agentic MoE. It is a 1T-parameter model (32B active) with 256K context, MLA attention, and a MoonViT vision encoder, supporting both thinking and instant modes.
|
||||
|
||||
In SGLang, Kimi-K2.5 uses the `kimi_k2` reasoning and tool-call parsers for correct thinking and tool handling.
|
||||
|
||||
```{note} Example
|
||||
Kimi-K2.5 support is in SGLang main and will land in the next release. Use the latest main or a nightly image until then.
|
||||
```
|
||||
|
||||
Official deployment guide: [Kimi-K2.5 deployment guide](https://huggingface.co/moonshotai/Kimi-K2.5/blob/main/docs/deploy_guidance)
|
||||
|
||||
## Install (Latest Main)
|
||||
|
||||
```bash Command
|
||||
uv pip install "sglang @ git+https://github.com/sgl-project/sglang.git#subdirectory=python"
|
||||
# For CUDA 12:
|
||||
uv pip install "nvidia-cudnn-cu12==9.16.0.29"
|
||||
# For CUDA 13:
|
||||
uv pip install "nvidia-cudnn-cu13==9.16.0.29"
|
||||
```
|
||||
|
||||
## Launch Kimi-K2.5 with SGLang
|
||||
|
||||
Example: single node, TP8 on H200.
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path moonshotai/Kimi-K2.5 \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--tool-call-parser kimi_k2 \
|
||||
--reasoning-parser kimi_k2
|
||||
```
|
||||
|
||||
### Parser Requirements
|
||||
|
||||
- `--tool-call-parser kimi_k2`: Required for tool calling.
|
||||
- `--reasoning-parser kimi_k2`: Required to parse thinking content; thinking mode is enabled by default.
|
||||
|
||||
## Test the Deployment
|
||||
|
||||
Thinking mode is enabled by default. To disable thinking (instant mode), pass `extra_body.chat_template_kwargs.thinking=false`.
|
||||
|
||||
```bash Command
|
||||
# Thinking mode (default)
|
||||
curl http://localhost:30000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "moonshotai/Kimi-K2.5",
|
||||
"messages": [
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "Explain mixture-of-experts in one sentence."}
|
||||
],
|
||||
"max_tokens": 256
|
||||
}'
|
||||
```
|
||||
|
||||
```bash Command
|
||||
# Instant mode (thinking disabled)
|
||||
curl http://localhost:30000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "moonshotai/Kimi-K2.5",
|
||||
"messages": [
|
||||
{"role": "user", "content": "Give one sentence on MoE models."}
|
||||
],
|
||||
"max_tokens": 128,
|
||||
"extra_body": {"chat_template_kwargs": {"thinking": false}}
|
||||
}'
|
||||
```
|
||||
|
||||
## Multimodal Inputs (Image/Video)
|
||||
|
||||
Kimi-K2.5 is multimodal. Image inputs are supported via the OpenAI-compatible vision API. For more details, see `openai_api_vision.ipynb`.
|
||||
|
||||
```bash Command
|
||||
# Image input (SGLang)
|
||||
curl http://localhost:30000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "moonshotai/Kimi-K2.5",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "Describe this image."},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"max_tokens": 256
|
||||
}'
|
||||
```
|
||||
|
||||
<Note>
|
||||
Video chat is experimental and is only supported in the official Moonshot API for now.
|
||||
</Note>
|
||||
@@ -1,117 +0,0 @@
|
||||
---
|
||||
title: "Llama4 Usage"
|
||||
metatags:
|
||||
description: "Deploy Llama 4 Scout (109B) and Maverick (400B) with SGLang: up to 10M context, hybrid KV cache, vision support. Optimized for H100/H200 GPUs."
|
||||
---
|
||||
[Llama 4](https://github.com/meta-llama/llama-models/blob/main/models/llama4/MODEL_CARD) is Meta's latest generation of open-source LLM model with industry-leading performance.
|
||||
|
||||
SGLang has supported Llama 4 Scout (109B) and Llama 4 Maverick (400B) since [v0.4.5](https://github.com/sgl-project/sglang/releases/tag/v0.4.5).
|
||||
|
||||
Ongoing optimizations are tracked in the [Roadmap](https://github.com/sgl-project/sglang/issues/5118).
|
||||
|
||||
## Launch Llama 4 with SGLang
|
||||
|
||||
To serve Llama 4 models on 8xH100/H200 GPUs:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
|
||||
--tp 8 \
|
||||
--context-length 1000000
|
||||
```
|
||||
|
||||
### Configuration Tips
|
||||
|
||||
- **OOM Mitigation**: Adjust `--context-length` to avoid a GPU out-of-memory issue. For the Scout model, we recommend setting this value up to 1M on 8\*H100 and up to 2.5M on 8\*H200. For the Maverick model, we don't need to set context length on 8\*H200. When hybrid kv cache is enabled, `--context-length` can be set up to 5M on 8\*H100 and up to 10M on 8\*H200 for the Scout model.
|
||||
|
||||
- **Attention Backend Auto-Selection**: SGLang automatically selects the optimal attention backend for Llama 4 based on your hardware. You typically don't need to specify `--attention-backend` manually:
|
||||
- **Blackwell GPUs (B200/GB200)**: `trtllm_mha`
|
||||
- **Hopper GPUs (H100/H200)**: `fa3`
|
||||
- **AMD GPUs**: `aiter`
|
||||
- **Intel XPU**: `intel_xpu`
|
||||
- **Other platforms**: `triton` (fallback)
|
||||
|
||||
To override the auto-selection, explicitly specify `--attention-backend` with one of the supported backends: `fa3`, `aiter`, `triton`, `trtllm_mha`, or `intel_xpu`.
|
||||
|
||||
- **Chat Template**: Add `--chat-template llama-4` for chat completion tasks.
|
||||
- **Enable Multi-Modal**: Add `--enable-multimodal` for multi-modal capabilities.
|
||||
- **Enable Hybrid-KVCache**: Set `--swa-full-tokens-ratio` to adjust the ratio of SWA layer (for Llama4, it's local attention layer) KV tokens / full layer KV tokens. (default: 0.8, range: 0-1)
|
||||
|
||||
|
||||
### EAGLE Speculative Decoding
|
||||
**Description**: SGLang has supported Llama 4 Maverick (400B) with [EAGLE speculative decoding](../advanced_features/speculative_decoding#EAGLE-Decoding).
|
||||
|
||||
**Usage**:
|
||||
Add arguments `--speculative-draft-model-path`, `--speculative-algorithm`, `--speculative-num-steps`, `--speculative-eagle-topk` and `--speculative-num-draft-tokens` to enable this feature. For example:
|
||||
```text Output
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--speculative-algorithm EAGLE3 \
|
||||
--speculative-draft-model-path nvidia/Llama-4-Maverick-17B-128E-Eagle3 \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--trust-remote-code \
|
||||
--tp 8 \
|
||||
--context-length 1000000
|
||||
```
|
||||
|
||||
- **Note** The Llama 4 draft model *nvidia/Llama-4-Maverick-17B-128E-Eagle3* can only recognize conversations in chat mode.
|
||||
|
||||
## Benchmarking Results
|
||||
|
||||
### Accuracy Test with `lm_eval`
|
||||
|
||||
The accuracy on SGLang for both Llama4 Scout and Llama4 Maverick can match the [official benchmark numbers](https://ai.meta.com/blog/llama-4-multimodal-intelligence/).
|
||||
|
||||
Benchmark results on MMLU Pro dataset with 8*H100:
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "34%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}></th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama-4-Scout-17B-16E-Instruct</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-4-Maverick-17B-128E-Instruct</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Official Benchmark</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>74.3</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>80.5</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>SGLang</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>75.2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>80.7</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
Commands:
|
||||
|
||||
```bash Command
|
||||
# Llama-4-Scout-17B-16E-Instruct model
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
|
||||
--port 30000 \
|
||||
--tp 8 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--context-length 65536
|
||||
lm_eval --model local-chat-completions --model_args model=meta-llama/Llama-4-Scout-17B-16E-Instruct,base_url=http://localhost:30000/v1/chat/completions,num_concurrent=128,timeout=999999,max_gen_toks=2048 --tasks mmlu_pro --batch_size 128 --apply_chat_template --num_fewshot 0
|
||||
|
||||
# Llama-4-Maverick-17B-128E-Instruct
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--port 30000 \
|
||||
--tp 8 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--context-length 65536
|
||||
lm_eval --model local-chat-completions --model_args model=meta-llama/Llama-4-Maverick-17B-128E-Instruct,base_url=http://localhost:30000/v1/chat/completions,num_concurrent=128,timeout=999999,max_gen_toks=2048 --tasks mmlu_pro --batch_size 128 --apply_chat_template --num_fewshot 0
|
||||
```
|
||||
|
||||
Details can be seen in [this PR](https://github.com/sgl-project/sglang/pull/5092).
|
||||
@@ -1,88 +0,0 @@
|
||||
---
|
||||
title: "MiniMax M2.5/M2.1/M2 Usage"
|
||||
metatags:
|
||||
description: "Deploy MiniMax M2.5/M2.1/M2 with SGLang: 230B MoE model (10B active), up to 3M context, optimized for coding and agentic tasks, tool use support."
|
||||
---
|
||||
[MiniMax-M2.5](https://huggingface.co/MiniMaxAI/MiniMax-M2.5), [MiniMax-M2.1](https://huggingface.co/MiniMaxAI/MiniMax-M2.1), and [MiniMax-M2](https://huggingface.co/MiniMaxAI/MiniMax-M2) are advanced large language models created by [MiniMax](https://www.minimax.io/).
|
||||
|
||||
The MiniMax-M2 series redefines efficiency for agents. These compact, fast, and cost-effective MoE models (230 billion total parameters with 10 billion active parameters) are built for elite performance in coding and agentic tasks, all while maintaining powerful general intelligence. With just 10 billion activated parameters, the MiniMax-M2 series provides sophisticated, end-to-end tool use performance expected from today's leading models, but in a streamlined form factor that makes deployment and scaling easier than ever.
|
||||
|
||||
## Supported Models
|
||||
|
||||
This guide applies to the following models. You only need to update the model name during deployment. The following examples use **MiniMax-M2**:
|
||||
|
||||
- [MiniMaxAI/MiniMax-M2.5](https://huggingface.co/MiniMaxAI/MiniMax-M2.5)
|
||||
- [MiniMaxAI/MiniMax-M2.1](https://huggingface.co/MiniMaxAI/MiniMax-M2.1)
|
||||
- [MiniMaxAI/MiniMax-M2](https://huggingface.co/MiniMaxAI/MiniMax-M2)
|
||||
|
||||
## System Requirements
|
||||
|
||||
The following are recommended configurations; actual requirements should be adjusted based on your use case:
|
||||
|
||||
- 4x 96GB GPUs: Supported context length of up to 400K tokens.
|
||||
- 8x 144GB GPUs: Supported context length of up to 3M tokens.
|
||||
|
||||
## Deployment with Python
|
||||
|
||||
4-GPU deployment command:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path MiniMaxAI/MiniMax-M2 \
|
||||
--tp-size 4 \
|
||||
--tool-call-parser minimax-m2 \
|
||||
--reasoning-parser minimax-append-think \
|
||||
--host 0.0.0.0 \
|
||||
--trust-remote-code \
|
||||
--port 8000 \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
|
||||
8-GPU deployment command:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path MiniMaxAI/MiniMax-M2 \
|
||||
--tp-size 8 \
|
||||
--ep-size 8 \
|
||||
--tool-call-parser minimax-m2 \
|
||||
--reasoning-parser minimax-append-think \
|
||||
--host 0.0.0.0 \
|
||||
--trust-remote-code \
|
||||
--port 8000 \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
|
||||
### AMD GPUs (MI300X/MI325X/MI355X)
|
||||
|
||||
8-GPU deployment command:
|
||||
|
||||
```bash Command
|
||||
SGLANG_USE_AITER=1 python -m sglang.launch_server \
|
||||
--model-path MiniMaxAI/MiniMax-M2.5 \
|
||||
--tp-size 8 \
|
||||
--ep-size 8 \
|
||||
--attention-backend aiter \
|
||||
--tool-call-parser minimax-m2 \
|
||||
--reasoning-parser minimax-append-think \
|
||||
--host 0.0.0.0 \
|
||||
--trust-remote-code \
|
||||
--port 8000 \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
|
||||
## Testing Deployment
|
||||
|
||||
After startup, you can test the SGLang OpenAI-compatible API with the following command:
|
||||
|
||||
```bash Command
|
||||
curl http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "MiniMaxAI/MiniMax-M2",
|
||||
"messages": [
|
||||
{"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]},
|
||||
{"role": "user", "content": [{"type": "text", "text": "Who won the world series in 2020?"}]}
|
||||
]
|
||||
}'
|
||||
```
|
||||
@@ -8,4 +8,4 @@ description: Core APIs and common usage patterns for SGLang.
|
||||
- [Offline Engine API](./offline_engine_api)
|
||||
- [Native API](./native_api)
|
||||
- [Sampling Parameters](./sampling_params)
|
||||
- [Popular Model Usage](./popular_model_usage) — DeepSeek, GLM, Qwen, Llama, and more
|
||||
- [Popular Model Usage](/cookbook/autoregressive/intro) — DeepSeek, GLM, Qwen, Llama, and more
|
||||
|
||||
@@ -1,17 +0,0 @@
|
||||
---
|
||||
title: "Popular Model Usage (DeepSeek, GPT-OSS, GLM, Llama, MiniMax, Qwen, and more)"
|
||||
description: "Documentation for Popular Model Usage (DeepSeek, GPT-OSS, GLM, Llama, MiniMax, Qwen, and more)"
|
||||
---
|
||||
For more usage examples and recipes, visit the [SGLang Cookbook](https://cookbook.sglang.io/).
|
||||
|
||||
- [Deepseek V3](./deepseek_v3)
|
||||
- [Deepseek V32](./deepseek_v32)
|
||||
- [Glm45](./glm45)
|
||||
- [Glmv](./glmv)
|
||||
- [Gpt Oss](./gpt_oss)
|
||||
- [Minimax M2](./minimax_m2)
|
||||
- [Qwen3](./qwen3)
|
||||
- [Qwen3 5](./qwen3_5)
|
||||
- [Qwen3 Vl](./qwen3_vl)
|
||||
- [Deepseek Ocr](./deepseek_ocr)
|
||||
- [Llama4](./llama4)
|
||||
@@ -1,16 +0,0 @@
|
||||
Popular Model Usage (DeepSeek, GPT-OSS, GLM, Llama, MiniMax, Qwen, and more)
|
||||
===============================================================
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
deepseek_v3.md
|
||||
deepseek_v32.md
|
||||
glm45.md
|
||||
glmv.md
|
||||
gpt_oss.md
|
||||
kimi_k2_5.md
|
||||
minimax_m2.md
|
||||
qwen3.md
|
||||
qwen3_vl.md
|
||||
llama4.md
|
||||
@@ -1,42 +0,0 @@
|
||||
---
|
||||
title: "Qwen3-Next Usage"
|
||||
metatags:
|
||||
description: "Deploy Qwen3-Next with SGLang: 80B hybrid Mamba model, MambaRadixCache prefix caching, EAGLE speculative decoding. Supports H100/H200 GPUs."
|
||||
---
|
||||
SGLang has supported Qwen3-Next-80B-A3B-Instruct and Qwen3-Next-80B-A3B-Thinking since [this PR](https://github.com/sgl-project/sglang/pull/10233).
|
||||
|
||||
## Launch Qwen3-Next with SGLang
|
||||
|
||||
To serve Qwen3-Next models on 4xH100/H200 GPUs:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server --model Qwen/Qwen3-Next-80B-A3B-Instruct --tp 4
|
||||
```
|
||||
|
||||
### Configuration Tips
|
||||
- `--max-mamba-cache-size`: Adjust `--max-mamba-cache-size` to increase mamba cache space and max running requests capability. It will decrease KV cache space as a trade-off. You can adjust it according to workload.
|
||||
- `--mamba-ssm-dtype`: `bfloat16` or `float32`, use `bfloat16` to save mamba cache size and `float32` to get more accurate results. The default setting is `float32`.
|
||||
- `--mamba-full-memory-ratio`: The ratio of mamba state memory to full kv cache memory. The default is 0.9.
|
||||
|
||||
### Mamba Radix Cache
|
||||
SGLang supports prefix caching for Qwen3-Next models named `MambaRadixCache`, which improves inference speed by reusing computation results. There are two versions of `MambaRadixCache`:
|
||||
- `no_buffer`: The default version, which is also other hybrid linear models' choice. When it is enabled, SGLang will automatically close overlap schedule for compatibility reasons.
|
||||
- `extra_buffer`: An optimized version that is compatible with features like page size > 1, overlap schedule, and speculative decoding. It also supports storing mamba state in branching positions. However, it requires two extra mamba spaces for a ping-pong buffer for each request. To enable it, add the argument `--mamba-scheduler-strategy extra_buffer` when launching the server.
|
||||
|
||||
### EAGLE Speculative Decoding
|
||||
**Description**: SGLang has supported Qwen3-Next models with [EAGLE speculative decoding](../advanced_features/speculative_decoding#EAGLE-Decoding).
|
||||
|
||||
**Usage**:
|
||||
Add arguments `--speculative-algorithm`, `--speculative-num-steps`, `--speculative-eagle-topk` and `--speculative-num-draft-tokens` to enable this feature. For example:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-Next-80B-A3B-Instruct \
|
||||
--tp 4 \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--speculative-algo NEXTN
|
||||
```
|
||||
|
||||
Details can be seen in [this PR](https://github.com/sgl-project/sglang/pull/10233).
|
||||
@@ -1,80 +0,0 @@
|
||||
---
|
||||
title: "Qwen 3.5 Usage"
|
||||
metatags:
|
||||
description: "Qwen 3.5 is Alibaba's latest generation LLM featuring a hybrid attention architecture, advanced MoE with shared experts, and native multimodal capabilities."
|
||||
---
|
||||
|
||||
Qwen 3.5 is Alibaba's latest generation LLM featuring a hybrid attention architecture, advanced MoE with shared experts, and native multimodal capabilities.
|
||||
|
||||
Key architecture features:
|
||||
- **Hybrid Attention**: Gated Delta Networks (linear, O(n) complexity) combined with full attention every 4th layer for high associative recall
|
||||
- **MoE with Shared Experts**: Top-8 active out of 64 routed experts plus a dedicated shared expert for universal features
|
||||
- **Multimodal**: DeepStack Vision Transformer with Conv3d for native image and video understanding
|
||||
|
||||
## Launch Qwen 3.5 with SGLang
|
||||
|
||||
### Dense Model
|
||||
|
||||
To serve `Qwen/Qwen3.5-397B-A17B` on 8 GPUs:
|
||||
|
||||
```bash
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3.5-397B-A17B \
|
||||
--tp 8 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
### AMD GPU (MI300X / MI325X / MI35X)
|
||||
|
||||
On AMD Instinct GPUs, use the `triton` attention backend. Both the full attention layers and the Gated Delta Net (linear attention) layers use Triton-based kernels on ROCm:
|
||||
|
||||
```bash
|
||||
SGLANG_USE_AITER=1 python3 -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3.5-397B-A17B \
|
||||
--tp 8 \
|
||||
--attention-backend triton \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
<Tip>
|
||||
Set `SGLANG_USE_AITER=1` to enable AMD's optimized aiter kernels for MoE and GEMM operations.
|
||||
</Tip>
|
||||
|
||||
### Configuration Tips
|
||||
|
||||
- `--attention-backend`: Use `triton` on AMD GPUs for Qwen 3.5. The hybrid attention architecture (Gated Delta Networks + full attention) works best with the Triton backend on ROCm. The linear attention (GDN) layers always use Triton kernels internally via the `GDNAttnBackend`.
|
||||
- `--watchdog-timeout`: Increase to `1200` or higher for this large model, as weight loading takes significant time.
|
||||
- `--model-loader-extra-config '{"enable_multithread_load": true}'`: Enables parallel weight loading for faster startup.
|
||||
|
||||
### Reasoning and Tool Calling
|
||||
|
||||
Qwen 3.5 supports reasoning and tool calling via the Qwen3 parsers:
|
||||
|
||||
```bash
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3.5-397B-A17B \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--reasoning-parser qwen3 \
|
||||
--tool-call-parser qwen3_coder
|
||||
```
|
||||
|
||||
## Accuracy Evaluation
|
||||
|
||||
You can evaluate the model accuracy using `lm-eval`:
|
||||
|
||||
```bash
|
||||
pip install lm-eval[api]
|
||||
|
||||
lm_eval --model local-completions \
|
||||
--model_args '{"base_url": "http://localhost:8000/v1/completions", "model": "Qwen/Qwen3.5-397B-A17B", "num_concurrent": 256, "max_retries": 10, "max_gen_toks": 2048}' \
|
||||
--tasks gsm8k \
|
||||
--batch_size auto \
|
||||
--num_fewshot 5 \
|
||||
--trust_remote_code
|
||||
```
|
||||
|
||||
## Additional Resources
|
||||
|
||||
- [AMD Day 0 Support for Qwen 3.5 on AMD Instinct GPUs](https://www.amd.com/en/developer/resources/technical-articles/2026/day-0-support-for-qwen-3-5-on-amd-instinct-gpus.html)
|
||||
- [HuggingFace Model Card](https://huggingface.co/Qwen/Qwen3.5-397B-A17B)
|
||||
@@ -1,133 +0,0 @@
|
||||
---
|
||||
title: "Qwen3-VL Usage"
|
||||
metatags:
|
||||
description: "Deploy Qwen3-VL vision models with SGLang: FP8 and BF16 modes, image and video input, expert parallelism. Supports H100, H200, A100 GPUs."
|
||||
---
|
||||
[Qwen3-VL](https://huggingface.co/collections/Qwen/qwen3-vl)
|
||||
is Alibaba’s latest multimodal large language model with strong text, vision, and reasoning capabilities.
|
||||
SGLang supports Qwen3-VL Family of models with Image and Video input support.
|
||||
|
||||
## Launch commands for SGLang
|
||||
|
||||
Below are suggested launch commands tailored for different hardware / precision modes
|
||||
|
||||
### FP8 (quantised) mode
|
||||
For high memory-efficiency and latency optimized deployments (e.g., on H100, H200) where FP8 checkpoint is supported:
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-VL-235B-A22B-Instruct-FP8 \
|
||||
--tp 8 \
|
||||
--ep 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--keep-mm-feature-on-device
|
||||
```
|
||||
|
||||
### Non-FP8 (BF16 / full precision) mode
|
||||
For deployments on A100/H100 where BF16 is used (or FP8 snapshot not used):
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-VL-235B-A22B-Instruct \
|
||||
--tp 8 \
|
||||
--ep 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
```
|
||||
|
||||
## Hardware-specific notes / recommendations
|
||||
|
||||
- On H100 with FP8: Use the FP8 checkpoint for best memory efficiency.
|
||||
- On A100 / H100 with BF16 (non-FP8): It’s recommended to use `--mm-max-concurrent-calls` to control parallel throughput and GPU memory usage during image/video inference.
|
||||
- On H200 & B200: The model can be run “out of the box”, supporting full context length plus concurrent image + video processing.
|
||||
|
||||
## Sending Image/Video Requests
|
||||
|
||||
### Image input:
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
url = f"http://localhost:30000/v1/chat/completions"
|
||||
|
||||
data = {
|
||||
"model": "Qwen/Qwen3-VL-30B-A3B-Instruct",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "What’s in this image?"},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true"
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
"max_tokens": 300,
|
||||
}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
print(response.text)
|
||||
```
|
||||
|
||||
### Video Input:
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
url = f"http://localhost:30000/v1/chat/completions"
|
||||
|
||||
data = {
|
||||
"model": "Qwen/Qwen3-VL-30B-A3B-Instruct",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "What’s happening in this video?"},
|
||||
{
|
||||
"type": "video_url",
|
||||
"video_url": {
|
||||
"url": "https://github.com/sgl-project/sgl-test-files/raw/refs/heads/main/videos/jobs_presenting_ipod.mp4"
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
"max_tokens": 300,
|
||||
}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
print(response.text)
|
||||
```
|
||||
|
||||
## Important Server Parameters and Flags
|
||||
|
||||
When launching the model server for **multimodal support**, you can use the following command-line arguments to fine-tune performance and behavior:
|
||||
|
||||
- `--mm-attention-backend`: Specify multimodal attention backend. Eg. `fa3`(Flash Attention 3)
|
||||
- `--mm-max-concurrent-calls <value>`: Specifies the **maximum number of concurrent asynchronous multimodal data processing calls** allowed on the server. Use this to control parallel throughput and GPU memory usage during image/video inference.
|
||||
- `--mm-per-request-timeout <seconds>`: Defines the **timeout duration (in seconds)** for each multimodal request. If a request exceeds this time limit (e.g., for very large video inputs), it will be automatically terminated.
|
||||
- `--keep-mm-feature-on-device`: Instructs the server to **retain multimodal feature tensors on the GPU** after processing. This avoids device-to-host (D2H) memory copies and improves performance for repeated or high-frequency inference workloads.
|
||||
- `SGLANG_USE_CUDA_IPC_TRANSPORT=1`: Shared memory pool based CUDA IPC for multi-modal data transport. For significantly improving e2e latency.
|
||||
|
||||
### Example usage with the above optimizations:
|
||||
```bash Command
|
||||
SGLANG_USE_CUDA_IPC_TRANSPORT=1 \
|
||||
SGLANG_VLM_CACHE_SIZE_MB=0 \
|
||||
python -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-VL-235B-A22B-Instruct \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--trust-remote-code \
|
||||
--tp-size 8 \
|
||||
--enable-cache-report \
|
||||
--log-level info \
|
||||
--max-running-requests 64 \
|
||||
--mem-fraction-static 0.65 \
|
||||
--chunked-prefill-size 8192 \
|
||||
--attention-backend fa3 \
|
||||
--mm-attention-backend fa3 \
|
||||
--enable-metrics
|
||||
```
|
||||
Reference in New Issue
Block a user