Add new Mintlify documentation site (docs_new/) (#23001)
Co-authored-by: AdityaVKochar <adityavardhankochar@gmail.com> Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com> Co-authored-by: adhyan-jain <adhyanjain2006@gmail.com> Co-authored-by: Adhyan Jain <71976554+adhyan-jain@users.noreply.github.com> Co-authored-by: Maitri-shah29 <maitrirajivshah@gmail.com> Co-authored-by: Adarsh Shirawalmath <114558126+adarshxs@users.noreply.github.com> Co-authored-by: Maitri Shah <shah29maitri@gmail.com> Co-authored-by: Aditya Vardhan Kochar <80113212+AdityaVKochar@users.noreply.github.com> Co-authored-by: Rishit Shivam <164783543+pokymono@users.noreply.github.com> Co-authored-by: Rishitshivam <164783543+Rishitshivam@users.noreply.github.com> Co-authored-by: IshhanKheria <ishhankheria06@gmail.com> Co-authored-by: Ishita Joshi <ishitata.joshi@gmail.com> Co-authored-by: Richard Chen <104477092+Richardczl98@users.noreply.github.com> Co-authored-by: longGGGGGG <553746008@qq.com> Co-authored-by: Richard <richardchen@radixark.ai> Co-authored-by: Nakul Sinha <nakul.new4socials@gmail.com> Co-authored-by: Divyam Agrawal <ludicrouslytrue@gmail.com> Co-authored-by: Richardczl98 <Zhenlinc@stanford.edu> Co-authored-by: Krishang Zinzuwadia <krishangzinzuwadia@gmail.com> Co-authored-by: nimeshas <nimesha.s106@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jignas Paturu <86356085+JignasP@users.noreply.github.com> Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
This commit is contained in:
Mingyi
co-authored by
AdityaVKochar
mintlify[bot]
adhyan-jain
Adhyan Jain
Maitri-shah29
Adarsh Shirawalmath
Maitri Shah
Aditya Vardhan Kochar
Rishit Shivam
Rishitshivam
IshhanKheria
Ishita Joshi
Richard Chen
longGGGGGG
Richard
Nakul Sinha
Divyam Agrawal
Richardczl98
Krishang Zinzuwadia
nimeshas
Claude Opus 4.6
github-actions[bot]
Jignas Paturu
zijiexia
AdityaVKochar
mintlify[bot]
adhyan-jain
Adhyan Jain
Maitri-shah29
Adarsh Shirawalmath
Maitri Shah
Aditya Vardhan Kochar
Rishit Shivam
Rishitshivam
IshhanKheria
Ishita Joshi
Richard Chen
longGGGGGG
Richard
Nakul Sinha
Divyam Agrawal
Richardczl98
Krishang Zinzuwadia
nimeshas
Claude Opus 4.6
github-actions[bot]
Jignas Paturu
zijiexia
parent
575fdc2c4c
commit
a3291b5654
@@ -0,0 +1,520 @@
|
||||
---
|
||||
title: "Attention Backend"
|
||||
metatags:
|
||||
description: "SGLang attention backend guide: FlashInfer, FA3, FA4, Triton, FlashMLA, TRTLLM MLA, hybrid attention. Support matrix for MHA and MLA models."
|
||||
---
|
||||
SGLang supports a large variety of attention backends. Each of them has different pros and cons.
|
||||
You can test them according to your needs.
|
||||
|
||||
<Warning>
|
||||
Selecting an optimal attention backend is crucial for maximizing your performance. Different backends excel in various scenarios, so choose based on your model, hardware, and use case. Not all backends are supported on all platforms and model architectures.
|
||||
|
||||
If you don't specify `--attention-backend`, SGLang makes a best effort to automatically select the most performant backend based on your hardware and model architecture.
|
||||
</Warning>
|
||||
|
||||
## Support Matrix
|
||||
|
||||
The support matrix is split into two parts: MHA (standard attention) and MLA (multi-head latent attention). For an explanation of the key differences between MHA and MLA, please see the [SGLang documentation on DeepSeek MLA](../basic_usage/deepseek_v3.md#multi-head-latent-attention-mla-throughput-optimizations) and the original [DeepSeek MLA paper](https://arxiv.org/pdf/2405.04434).
|
||||
|
||||
### MHA Backends
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "13%"}} />
|
||||
<col style={{width: "13%"}} />
|
||||
<col style={{width: "13%"}} />
|
||||
<col style={{width: "13%"}} />
|
||||
<col style={{width: "12%"}} />
|
||||
<col style={{width: "12%"}} />
|
||||
<col style={{width: "12%"}} />
|
||||
<col style={{width: "12%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Backend**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>**Page Size > 1 (native)**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**FP8 KV Cache**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>**FP4 KV Cache**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Spec topk=1**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>**Spec topk>1**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Sliding Window**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>**MultiModal**</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**FlashInfer**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**FA3 (FlashAttention 3)**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**FA4 (FlashAttention 4)**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>128</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Triton**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Torch Native (SDPA)**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**FlexAttention (PyTorch)**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**TRTLLM MHA**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16, 32 or 64</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Dual Chunk FlashAttention**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**AITER (ROCm)**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Wave (ROCm)**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Ascend (NPU)**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Intel XPU**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Intel AMX (CPU)**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
### MLA Backends
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "15%"}} />
|
||||
<col style={{width: "15%"}} />
|
||||
<col style={{width: "14%"}} />
|
||||
<col style={{width: "14%"}} />
|
||||
<col style={{width: "14%"}} />
|
||||
<col style={{width: "14%"}} />
|
||||
<col style={{width: "14%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Backend**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>**Native Page Sizes**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**FP8 KV Cache**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>**FP4 KV Cache**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Chunked Prefix Cache**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>**Spec topk=1**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Spec topk>1**</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**FlashInfer MLA**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**FlashMLA**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>64</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Cutlass MLA**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>128</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**TRTLLM MLA (Blackwell)**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>32 or 64</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**FA3 (FlashAttention 3)**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>n/a</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>⚠️ (page_size=1 only)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Triton**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>n/a</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>⚠️ (page_size=1 only)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**FA4**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Ascend MLA (NPU)**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>128</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<Note>
|
||||
Multimodal attention is selected by `--mm-attention-backend`. The "MultiModal" column indicates whether a corresponding multimodal implementation exists for that backend family.
|
||||
</Note>
|
||||
|
||||
<Note>
|
||||
- FlashAttention 4 is prefill-only for now.
|
||||
- NSA is specifically designed for [DeepSeek V3.2 DSA](https://lmsys.org/blog/2025-09-29-deepseek-V32/).
|
||||
</Note>
|
||||
|
||||
<Note>
|
||||
For the KV4 FA4 scenario, FA4 requires using a different --decode-attention-backend to run. Except for trtllm_mha being incompatible with FA4, all other decode backends behave as shown in the table.
|
||||
</Note>
|
||||
|
||||
<Tip>
|
||||
Speculative decoding topk: `topk` is the number of draft tokens sampled per step from the draft model. `topk = 1` follows classic EAGLE; `topk > 1` explores multiple branches and requires backend support in both draft and verification paths.
|
||||
</Tip>
|
||||
|
||||
<Tip>
|
||||
Page size controls how many tokens are grouped into a KV cache block. For the prefix cache to take effect, the number of tokens must fill at least one complete page. For example, if your prompt is only 32 tokens and `page_size = 64`, it won't fill a complete page and cannot be matched in the prefix cache (pages cannot be padded). With 65 tokens and `page_size = 64`, only the first page of 64 tokens will be cached and matched; the remaining 1 token is discarded. Use `page_size = 1` for maximum prefix reuse (token-level matching).
|
||||
</Tip>
|
||||
|
||||
Many backends that do not natively operate on pages can emulate `page_size > 1` at the wrapper layer by expanding page tables to per-token indices. The "Page Size > 1 (native)" column indicates true in-kernel paging. Some backends require fixed native page sizes and cannot be reduced/emulated differently: TRTLLM MHA (16/32/64), TRTLLM MLA (32/64), FlashMLA (64), Cutlass MLA (128), Ascend (128).
|
||||
|
||||
MLA page-size constraints:
|
||||
- FlashInfer MLA: page_size = 1.
|
||||
- FlashMLA: page_size = 64.
|
||||
- Cutlass MLA: page_size = 128.
|
||||
- TRTLLM MLA: page_size ∈ {32, 64}.
|
||||
|
||||
### Hybrid attention (different backends for prefill vs decode) (Experimental)
|
||||
|
||||
<Warning>
|
||||
Hybrid attention is an experimental feature.
|
||||
</Warning>
|
||||
|
||||
You can mix-and-match attention backends for prefill and decode. This is useful when one backend excels at prefill and another excels at decode. For the implementation details, please see `python/sglang/srt/layers/attention/hybrid_attn_backend.py`.
|
||||
|
||||
```bash Command
|
||||
# Example: Prefill with FA4, Decode with TRTLLM MLA (Blackwell)
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path nvidia/DeepSeek-R1-FP4 \
|
||||
--tp 8 \
|
||||
--attention-backend trtllm_mla \
|
||||
--moe-runner-backend flashinfer_trtllm \
|
||||
--quantization modelopt_fp4 \
|
||||
--prefill-attention-backend fa4
|
||||
```
|
||||
|
||||
#### Speculative decoding with hybrid attention
|
||||
|
||||
Hybrid attention also works with speculative decoding. The backend used for draft decoding and target verification depends on `--speculative-attention-mode`:
|
||||
|
||||
- `--speculative-attention-mode decode` (recommended): draft/verify use the decode backend.
|
||||
- `--speculative-attention-mode prefill` (default): draft/verify use the prefill backend.
|
||||
|
||||
Constraints when combining hybrid attention with speculative decoding:
|
||||
|
||||
- If any attention backend is `trtllm_mha`, speculative decoding supports only `--speculative-eagle-topk 1`.
|
||||
- For paged MHA backends with `--page-size > 1` and `--speculative-eagle-topk > 1`, only `flashinfer` is supported.
|
||||
- CUDA Graph: the decode backend is always captured; the prefill backend is captured only when `--speculative-attention-mode prefill`.
|
||||
|
||||
|
||||
<Tip>
|
||||
If you set only one of `--prefill-attention-backend` or `--decode-attention-backend`, the unspecified phase inherits `--attention-backend`.
|
||||
If both are specified and differ, SGLang automatically enables a hybrid wrapper to dispatch to the chosen backend per phase.
|
||||
</Tip>
|
||||
|
||||
## Attention Backend Selection Guide (CUDA)
|
||||
|
||||
If the `--attention-backend` argument is not specified, SGLang automatically selects the best backend based on the hardware (CUDA) and model architecture.
|
||||
|
||||
### Automatic Selection Logic
|
||||
|
||||
**1. MHA Models (e.g., Llama, Qwen)**
|
||||
- **Hopper (e.g., H100, H200)**: Defaults to `fa3` if using CUDA 12.3+ and the model configuration is supported.
|
||||
- **Blackwell (e.g., B200)**: Defaults to `trtllm_mha`, unless using speculative decoding with `topk > 1`.
|
||||
- **Other Architectures (Ampere, Ada, etc.)**: Defaults to `flashinfer` if available; otherwise falls back to `triton`.
|
||||
|
||||
**2. MLA Models (e.g., DeepSeek V3)**
|
||||
- **Hopper**: Defaults to `fa3` (requires CUDA 12.3+).
|
||||
- **Blackwell**: Defaults to `trtllm_mla`.
|
||||
- **Other Architectures**: Defaults to `triton`.
|
||||
|
||||
|
||||
## User Guide
|
||||
|
||||
### Launch Command for Different Attention Backends
|
||||
|
||||
- FlashInfer (Default for Non-Hopper Machines, e.g., A100, A40)
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--attention-backend flashinfer
|
||||
python3 -m sglang.launch_server \
|
||||
--tp 8 \
|
||||
--model deepseek-ai/DeepSeek-V3 \
|
||||
--attention-backend flashinfer \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
- FlashAttention 3 (Default for Hopper Machines, e.g., H100, H200, H20)
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--attention-backend fa3
|
||||
python3 -m sglang.launch_server \
|
||||
--tp 8 \
|
||||
--model deepseek-ai/DeepSeek-V3 \
|
||||
--trust-remote-code \
|
||||
--attention-backend fa3
|
||||
```
|
||||
|
||||
- Triton
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--attention-backend triton
|
||||
python3 -m sglang.launch_server \
|
||||
--tp 8 \
|
||||
--model deepseek-ai/DeepSeek-V3 \
|
||||
--attention-backend triton \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
- FlashMLA
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--tp 8 \
|
||||
--model deepseek-ai/DeepSeek-R1 \
|
||||
--attention-backend flashmla \
|
||||
--trust-remote-code
|
||||
python3 -m sglang.launch_server \
|
||||
--tp 8 \
|
||||
--model deepseek-ai/DeepSeek-R1 \
|
||||
--attention-backend flashmla \
|
||||
--kv-cache-dtype fp8_e4m3 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
- TRTLLM MLA (Optimized for Blackwell Architecture, e.g., B200)
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--tp 8 \
|
||||
--model deepseek-ai/DeepSeek-R1 \
|
||||
--attention-backend trtllm_mla \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
- TRTLLM MLA with FP8 KV Cache (Higher concurrency, lower memory footprint)
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--tp 8 \
|
||||
--model deepseek-ai/DeepSeek-R1 \
|
||||
--attention-backend trtllm_mla \
|
||||
--kv-cache-dtype fp8_e4m3 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
- FlashAttention 4 (MHA & MLA)
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--tp 8 \
|
||||
--model deepseek-ai/DeepSeek-R1 \
|
||||
--prefill-attention-backend fa4 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
- Cutlass MLA
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--tp 8 \
|
||||
--model deepseek-ai/DeepSeek-R1 \
|
||||
--attention-backend cutlass_mla \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
- Ascend
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--attention-backend ascend
|
||||
```
|
||||
|
||||
- Intel XPU
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--attention-backend intel_xpu
|
||||
```
|
||||
|
||||
- Wave
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--attention-backend wave
|
||||
```
|
||||
|
||||
- FlexAttention
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--attention-backend flex_attention
|
||||
```
|
||||
|
||||
- Dual Chunk FlashAttention
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model Qwen/Qwen2.5-14B-Instruct-1M \
|
||||
--attention-backend dual_chunk_flash_attn
|
||||
```
|
||||
|
||||
- Torch Native
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--attention-backend torch_native
|
||||
```
|
||||
|
||||
## Steps to add a new attention backend
|
||||
To add a new attention backend, you can learn from the existing backends
|
||||
(`python/sglang/srt/layers/attention/triton_backend.py`, `python/sglang/srt/layers/attention/flashattention_backend.py`)
|
||||
and follow the steps below.
|
||||
|
||||
1. Run without cuda graph. Support the two forward functions
|
||||
- forward_extend
|
||||
- Will be used for prefill, prefill with KV cache, and target verification
|
||||
- It will be called once per layer
|
||||
- forward_decode
|
||||
- Will be used for normal decode, and draft decode
|
||||
- It will be called once per layer
|
||||
- init_forward_metadata
|
||||
- Initialize the class and common metadata shared by all layers
|
||||
- Call the plan function for optimizations like split_kv
|
||||
- It will be called once per forward
|
||||
2. Run with cuda graph. It has two phases (capture and replay) and you need to implement three functions
|
||||
- init_cuda_graph_state
|
||||
- It will be called once during life time
|
||||
- Create all common shared buffers
|
||||
- init_forward_metadata_capture_cuda_graph
|
||||
- It will be called before capturing a cuda graph
|
||||
- It is similar to init_forward_metadata but write the medatada to some pre-defined buffers
|
||||
- init_forward_metadata_replay_cuda_graph
|
||||
- It will be called before replaying a cuda graph
|
||||
- This function is in the critical path and needs to be fast
|
||||
@@ -0,0 +1,192 @@
|
||||
---
|
||||
title: "Breakable CUDA Graph"
|
||||
metatags:
|
||||
description: "Use Breakable CUDA Graph to insert targeted eager graph breaks for debugging and CUDA graph compatibility."
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
||||
Standard CUDA graphs capture an entire forward pass as a single, opaque graph. This is great for performance, but creates two problems:
|
||||
|
||||
1. **Debugging is hard.** When something goes wrong inside a captured graph (wrong outputs, numerical mismatches, crashes), there is no way to step through the operations or insert print statements because the graph replays as a monolithic unit.
|
||||
|
||||
2. **Some ops are incompatible.** Certain operations — dynamic control flow, host-device synchronization, JIT compilation, or ops that change behavior across iterations — cannot be captured into a CUDA graph at all. Today, the only workaround is to disable CUDA graphs entirely, which sacrifices the kernel launch overhead savings for the rest of the model.
|
||||
|
||||
**Breakable CUDA Graph** solves both problems by allowing graph breaks to be inserted at specific points. The computation is split into multiple captured graph segments with eager (non-graph) execution in between. This preserves most of the CUDA graph performance benefit while allowing targeted operations to run outside the graph.
|
||||
|
||||
## Usage
|
||||
|
||||
### Debug Mode: Run Everything Eagerly
|
||||
|
||||
The simplest use case is debugging. The `--debug-cuda-graph` flag wraps the entire decode forward pass in a graph break, so every operation runs eagerly while still going through the full CUDA graph capture/replay code path. This lets you debug CUDA graph issues without changing model code.
|
||||
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--debug-cuda-graph
|
||||
```
|
||||
|
||||
This mode is intended for debugging only — it eliminates the performance benefit of CUDA graphs since every op runs eagerly.
|
||||
|
||||
### Selective Graph Breaks in Model Code
|
||||
|
||||
For production use, you can mark specific functions as "non-graphable" using the `@eager_on_graph` decorator. During CUDA graph capture, these functions run eagerly between captured graph segments. Outside of capture, they behave normally.
|
||||
|
||||
```python
|
||||
from sglang.srt.model_executor.breakable_cuda_graph.breakable_cuda_graph import eager_on_graph
|
||||
|
||||
@eager_on_graph(enable=True)
|
||||
def my_dynamic_op(x):
|
||||
# This op is incompatible with CUDA graph capture
|
||||
return some_dynamic_operation(x)
|
||||
```
|
||||
|
||||
You can also insert a bare graph break (no computation) using the `break_graph()` helper:
|
||||
|
||||
```python
|
||||
from sglang.srt.model_executor.breakable_cuda_graph.breakable_cuda_graph import break_graph
|
||||
|
||||
def forward(self, x):
|
||||
x = self.layer1(x)
|
||||
break_graph() # force a segment split here
|
||||
x = self.layer2(x)
|
||||
return x
|
||||
```
|
||||
|
||||
To enable breakable CUDA graph at the environment level (without debug mode), set the environment variable:
|
||||
|
||||
```bash
|
||||
export SGLANG_USE_BREAKABLE_CUDA_GRAPH=1
|
||||
python -m sglang.launch_server \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct
|
||||
```
|
||||
|
||||
### Server Args
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "34%"}} />
|
||||
<col style={{width: "18%"}} />
|
||||
<col style={{width: "48%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--debug-cuda-graph</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>False</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Enable debug/eager mode. Wraps the entire forward pass in a graph break so every op runs eagerly through the capture/replay path.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>SGLANG_USE_BREAKABLE_CUDA_GRAPH</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>0</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Environment variable. Enables breakable CUDA graph without debug mode. Required for <code>@eager_on_graph</code> decorators to take effect.</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## How It Works
|
||||
|
||||
### Capture
|
||||
|
||||
Breakable CUDA graph extends PyTorch's `torch.cuda.CUDAGraph` by splitting a single capture into multiple segments separated by graph breaks.
|
||||
|
||||
During capture, the flow is:
|
||||
|
||||
```
|
||||
Begin capture (segment 1)
|
||||
... graphable ops ...
|
||||
@eager_on_graph function encountered:
|
||||
1. End current capture segment
|
||||
2. Run the function eagerly (allocates output tensors)
|
||||
3. Record the function for later replay
|
||||
4. Begin new capture segment
|
||||
... more graphable ops ...
|
||||
End capture (segment N)
|
||||
```
|
||||
|
||||
Each segment is independently instantiated as a CUDA graph executable. The non-graph functions and their argument references are stored for replay.
|
||||
|
||||
### Replay
|
||||
|
||||
During replay:
|
||||
|
||||
```
|
||||
For each segment i:
|
||||
1. Launch CUDA graph segment i
|
||||
2. Run the recorded non-graph function i eagerly
|
||||
Launch final CUDA graph segment
|
||||
```
|
||||
|
||||
The non-graph functions are re-invoked with the same tensor references as capture time. Since these references point to the CUDA graph's static input/output buffers, they see updated values on each replay.
|
||||
|
||||
### Output Writeback
|
||||
|
||||
When a non-graph function produces output during replay, the result must be written back into the same tensor buffers that downstream graph segments reference. The mechanism handles:
|
||||
|
||||
- **Plain tensors**: In-place `copy_()` into the original buffer.
|
||||
- **Structured outputs** (dataclasses, objects with tensor attributes): Tensor fields are copied in-place; non-tensor fields are replaced.
|
||||
- **Dicts of tensors**: Tensor values are copied in-place; non-tensor values are replaced.
|
||||
|
||||
### Stream Fork/Join Tracking
|
||||
|
||||
Some models fork work onto secondary CUDA streams (e.g., for overlapped computation). Breakable CUDA graph hooks `torch.cuda.Stream.wait_stream` to track which streams are forked from the capture stream. When a graph break occurs, all forked streams are automatically joined back before ending the capture segment, and re-forked after beginning the next segment.
|
||||
|
||||
## Compatibility
|
||||
|
||||
- **NVIDIA CUDA only.** Breakable CUDA graph is not supported on ROCm/HIP or other non-CUDA platforms. On unsupported platforms, `--debug-cuda-graph` is automatically disabled with a warning.
|
||||
- **Requires `cuda-python`.** The `cuda.bindings` package must be installed (`pip install cuda-python`).
|
||||
- **Not compatible with memory saver mode.** Cannot be used together with `SGLANG_MEMORY_SAVER_CUDA_GRAPH`.
|
||||
|
||||
## Performance
|
||||
|
||||
When no graph breaks are inserted, breakable CUDA graph has minimal overhead compared to standard CUDA graph — the capture/replay path is nearly identical.
|
||||
|
||||
Each graph break adds:
|
||||
- One `cudaGraphLaunch` call (to replay the segment before the break)
|
||||
- One eager Python function call
|
||||
- One `cudaStreamBeginCapture` / `cudaStreamEndCapture` pair during capture
|
||||
|
||||
For typical use cases with a small number of graph breaks, the overhead is negligible compared to the saved kernel launch overhead from the captured segments.
|
||||
|
||||
## Code Reference
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "52%"}} />
|
||||
<col style={{width: "48%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>File</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/model_executor/breakable_cuda_graph/breakable_cuda_graph.py</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Core implementation: <code>eager_on_graph</code>, <code>BreakableCUDAGraph</code>, <code>BreakableCUDAGraphCapture</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/model_executor/breakable_cuda_graph/cuda_utils.py</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>CUDA runtime binding utilities</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/model_executor/cuda_graph_runner.py</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Integration with the main CUDA graph runner</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/server_args.py</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>--debug-cuda-graph</code> flag and environment variable handling</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/environ.py</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>SGLANG_USE_BREAKABLE_CUDA_GRAPH</code> environment variable definition</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
@@ -0,0 +1,257 @@
|
||||
---
|
||||
title: "Checkpoint Engine Integration"
|
||||
metatags:
|
||||
description: "SGLang checkpoint engine: distributed model weight loading, parallel multi-node setup, broadcast and P2P modes. Reduces loading time for large models."
|
||||
---
|
||||
The SGLang checkpoint engine integration provides an efficient way to load model weights using a distributed checkpoint loading system. This feature significantly reduces model loading time, especially for large models and multi-node setups, by parallelizing the weight loading process across multiple processes and nodes.
|
||||
|
||||
## Overview
|
||||
|
||||
The checkpoint engine integration allows SGLang to:
|
||||
- Load model weights in parallel using multiple processes
|
||||
- Distribute weight loading across multiple nodes to increase effective disk bandwidth
|
||||
- Overlap weight loading with other initialization tasks like CUDA graph capture
|
||||
- Support both single-node and multi-node deployments
|
||||
|
||||
## Installation
|
||||
|
||||
First, install the checkpoint engine package:
|
||||
|
||||
```bash Command
|
||||
pip install 'checkpoint-engine[p2p]'
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
The system consists of two main components:
|
||||
|
||||
1. **SGLang Server**: Runs with `--wait-for-initial-weights` flag to wait for weights before becoming ready
|
||||
2. **Checkpoint Engine Workers**: Separate processes (managed by torchrun) that load and distribute model weights
|
||||
|
||||
The checkpoint engine uses a parameter server architecture with support for:
|
||||
- **Broadcast mode**: Weights are broadcast from loading processes to inference processes
|
||||
- **P2P mode**: Direct peer-to-peer weight transfer between processes
|
||||
- **All mode**: Combination of both broadcast and P2P methods
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Single Node Setup
|
||||
|
||||
**Terminal 1 - Launch SGLang Server:**
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-8B \
|
||||
--tp 8 \
|
||||
--load-format dummy \
|
||||
--wait-for-initial-weights
|
||||
```
|
||||
|
||||
**Terminal 2 - Run Checkpoint Engine:**
|
||||
|
||||
Using sglang entrypoint:
|
||||
```bash Command
|
||||
python -m sglang.srt.checkpoint_engine.update \
|
||||
--update-method broadcast \
|
||||
--checkpoint-path /path/to/Qwen/Qwen3-8B/ \
|
||||
--inference-parallel-size 8
|
||||
```
|
||||
|
||||
Using torchrun directly:
|
||||
```bash Command
|
||||
torchrun --nproc-per-node 8 \
|
||||
examples/checkpoint_engine/update.py \
|
||||
--update-method broadcast \
|
||||
--checkpoint-path /path/to/Qwen/Qwen3-8B/ \
|
||||
--inference-parallel-size 8
|
||||
```
|
||||
|
||||
### Multi-Node Setup (2 Nodes)
|
||||
|
||||
**Node 0:**
|
||||
|
||||
Launch SGLang server:
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-8B \
|
||||
--tp 8 \
|
||||
--load-format dummy \
|
||||
--wait-for-initial-weights \
|
||||
--host [IP]
|
||||
```
|
||||
|
||||
Run checkpoint engine:
|
||||
|
||||
Using sglang entrypoint (recommended):
|
||||
```bash Command
|
||||
python -m sglang.srt.checkpoint_engine.update \
|
||||
--update-method broadcast \
|
||||
--checkpoint-path /path/to/Qwen/Qwen3-8B/ \
|
||||
--inference-parallel-size 8
|
||||
```
|
||||
|
||||
Using torchrun directly:
|
||||
```bash Command
|
||||
torchrun --nproc-per-node 8 \
|
||||
--nnodes 2 \
|
||||
--node-rank 0 \
|
||||
--master-addr [IP] \
|
||||
--master-port 29500 \
|
||||
examples/checkpoint_engine/update.py \
|
||||
--update-method broadcast \
|
||||
--checkpoint-path /path/to/Qwen/Qwen3-8B/ \
|
||||
--inference-parallel-size 8
|
||||
```
|
||||
|
||||
**Node 1:**
|
||||
|
||||
Launch SGLang server:
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-8B \
|
||||
--tp 8 \
|
||||
--load-format dummy \
|
||||
--wait-for-initial-weights \
|
||||
--host [IP]
|
||||
```
|
||||
|
||||
Run checkpoint engine:
|
||||
|
||||
Using sglang entrypoint (recommended):
|
||||
```bash Command
|
||||
python -m sglang.srt.checkpoint_engine.update \
|
||||
--update-method broadcast \
|
||||
--checkpoint-path /path/to/Qwen/Qwen3-8B/ \
|
||||
--inference-parallel-size 8
|
||||
```
|
||||
|
||||
Using torchrun directly:
|
||||
```bash Command
|
||||
torchrun --nproc-per-node 8 \
|
||||
--nnodes 2 \
|
||||
--node-rank 1 \
|
||||
--master-addr [IP] \
|
||||
--master-port 29500 \
|
||||
examples/checkpoint_engine/update.py \
|
||||
--update-method broadcast \
|
||||
--checkpoint-path /path/to/Qwen/Qwen3-8B/ \
|
||||
--inference-parallel-size 8
|
||||
```
|
||||
|
||||
### Multi-Node Setup with Tensor Parallelism (TP=16)
|
||||
|
||||
**Node 0:**
|
||||
|
||||
Launch SGLang server:
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-8B \
|
||||
--tp 8 \
|
||||
--load-format dummy \
|
||||
--wait-for-initial-weights \
|
||||
--host [IP] \
|
||||
--dist-init-addr [IP]:9120 \
|
||||
--nnodes 2 \
|
||||
--node-rank 0
|
||||
```
|
||||
|
||||
Run checkpoint engine:
|
||||
|
||||
Using sglang entrypoint (recommended):
|
||||
```bash Command
|
||||
python -m sglang.srt.checkpoint_engine.update \
|
||||
--update-method broadcast \
|
||||
--checkpoint-path /path/to/Qwen/Qwen3-8B/ \
|
||||
--inference-parallel-size 16
|
||||
```
|
||||
|
||||
Using torchrun directly:
|
||||
```bash Command
|
||||
torchrun --nproc-per-node 8 \
|
||||
--nnodes 2 \
|
||||
--node-rank 0 \
|
||||
--master-addr [IP] \
|
||||
--master-port 29500 \
|
||||
examples/checkpoint_engine/update.py \
|
||||
--update-method broadcast \
|
||||
--checkpoint-path /path/to/Qwen/Qwen3-8B/ \
|
||||
--inference-parallel-size 16
|
||||
```
|
||||
|
||||
**Node 1:**
|
||||
|
||||
Launch SGLang server:
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-8B \
|
||||
--tp 8 \
|
||||
--load-format dummy \
|
||||
--wait-for-initial-weights \
|
||||
--host [IP] \
|
||||
--dist-init-addr [IP]:9120 \
|
||||
--nnodes 2 \
|
||||
--node-rank 1
|
||||
```
|
||||
|
||||
Run checkpoint engine:
|
||||
|
||||
Using sglang entrypoint (recommended):
|
||||
```bash Command
|
||||
python -m sglang.srt.checkpoint_engine.update \
|
||||
--update-method broadcast \
|
||||
--checkpoint-path /path/to/Qwen/Qwen3-8B/ \
|
||||
--inference-parallel-size 16
|
||||
```
|
||||
|
||||
Using torchrun directly:
|
||||
```bash Command
|
||||
torchrun --nproc-per-node 8 \
|
||||
--nnodes 2 \
|
||||
--node-rank 1 \
|
||||
--master-addr [IP] \
|
||||
--master-port 29500 \
|
||||
examples/checkpoint_engine/update.py \
|
||||
--update-method broadcast \
|
||||
--checkpoint-path /path/to/Qwen/Qwen3-8B/ \
|
||||
--inference-parallel-size 16
|
||||
```
|
||||
|
||||
## Configuration Options
|
||||
|
||||
### SGLang Server Options
|
||||
|
||||
- `--load-format dummy`: Use dummy format for initial loading (allows overlapping with other tasks)
|
||||
- `--wait-for-initial-weights`: Wait for checkpoint engine to provide weights before becoming ready
|
||||
- `--host`: Host address for multi-node setups
|
||||
- `--dist-init-addr`: Distributed initialization address for tensor parallelism
|
||||
|
||||
### Checkpoint Engine Options
|
||||
|
||||
- `--update-method`: Weight update method (`broadcast`, `p2p`, or `all`)
|
||||
- `--checkpoint-path`: Path to model checkpoint directory
|
||||
- `--inference-parallel-size`: Number of inference parallel processes
|
||||
- `--endpoint`: SGLang server endpoint (default: `http://localhost:19730`)
|
||||
- `--checkpoint-name`: Name for the checkpoint (default: `my-checkpoint-iter-0`)
|
||||
- `--save-metas-file`: File to save checkpoint metadata
|
||||
- `--load-metas-file`: File to load checkpoint metadata from
|
||||
- `--uds`: Unix domain socket path for communication
|
||||
- `--weight-version`: Version identifier for weights
|
||||
|
||||
## Performance Benefits
|
||||
|
||||
The checkpoint engine provides significant time savings in two main aspects:
|
||||
|
||||
1. **Multi-node Loading**: Each node only loads a portion of weights from disk, effectively increasing disk bandwidth. More participating nodes provide greater acceleration. Preliminary tests show 20-second acceleration when loading DeepSeek-R1 on H20-3e with two nodes.
|
||||
|
||||
2. **Single Process Optimization**: Using dummy format allows overlapping disk-to-CPU transfer with CUDA graph capture and other initialization tasks, providing additional time savings.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- Ensure checkpoint engine package is installed: `pip install 'checkpoint-engine[p2p]'`
|
||||
- Verify network connectivity between nodes in multi-node setups
|
||||
- Check that the checkpoint path contains valid model files
|
||||
- Monitor logs for connection errors between SGLang server and checkpoint engine
|
||||
- Use `--sleep-time` parameter to add delays if needed for debugging
|
||||
|
||||
## References
|
||||
|
||||
- [Checkpoint Engine Repository](https://github.com/MoonshotAI/checkpoint-engine)
|
||||
@@ -0,0 +1,76 @@
|
||||
---
|
||||
title: "Cuda Graph for Multi-Modal Encoder in SGLang"
|
||||
metatags:
|
||||
description: "CUDA Graph optimization for ViT in SGLang: reduce kernel launch overhead, dynamic input handling, support for Qwen2.5-VL and Qwen3-VL models."
|
||||
---
|
||||
## Motivation
|
||||
|
||||
In multimodal reasoning services, the visual encoder (ViT / Vision Transformer) typically has a few characteristic traits:
|
||||
|
||||
Many layers, fragmented operators: Each layer includes LN, QKV projections, attention, MLP, residual connections, etc., resulting in extremely frequent kernel launches.
|
||||
|
||||
Server-side “small batch / low latency” is common: The batch size is very small (sometimes it looks like 1 after “flattening” the batch), so kernel launch overhead accounts for a large portion of end-to-end latency.
|
||||
|
||||
Input token count (number of patches) varies frequently: Different image/video resolutions and different batch composition lead to different sequence lengths
|
||||
S — and this is precisely the biggest obstacle for CUDA Graph (unstable shapes).
|
||||
|
||||
The value of CUDA Graph: It captures a long sequence of GPU kernels with fixed shapes and fixed memory addresses into a graph; later, for the same shapes, it can replay the graph directly, dramatically reducing launch overhead and making GPU scheduling more compact.
|
||||
|
||||
This led us to seek a CUDA Graph enabled feature for ViT in order to improve ViT performance.
|
||||
|
||||
## Design and Restrictions
|
||||
|
||||
The new CUDA Graph enabled ViT logic is built on ViTCudaGraphRunner. This runner captures the "blocks + merger + deepstack merger (optional)" part of a vision transformer into a CUDA graph and replays it for identical shapes. See the following design consideration and restrictions for more details.
|
||||
|
||||
### Dynamic inputs to fit static constraints of CUDA Graph
|
||||
|
||||
Variable sequence length S is very common in ViT. While CUDA Graph requires fixed shapes. The solution is to build a graph cache by S(e.g., graph_key = S). The first time create a new S, and then capture a graph; afterwards, replay it.
|
||||
|
||||
If there are many distinct S values, we need to increase VRAM usage which is graph-private memory pools for many graphs.
|
||||
|
||||
### Stable addresses
|
||||
|
||||
Everything "parameter-like" becomes a static buffer:
|
||||
|
||||
- block_input / block_ws / block_output
|
||||
- cu_full_len / cu_window_len and their kk variants
|
||||
- sin_cos_ws
|
||||
|
||||
In this way to solve the underlying requirement: during replay, not allowed to swap tensors, can only modify tensor contents.
|
||||
|
||||
### Attention backend arguments
|
||||
Attention backend arguments are fixed inside the graph:
|
||||
|
||||
TritonAttn expects [cu_seqlens, cu_seqlens_kk, max_len]
|
||||
FA3 expects [cu_seqlens, max_len]
|
||||
|
||||
max_len is frozen as an int constant.
|
||||
cu_seqlens is cached into a dict during create_graph(), and its contents are not updated during subsequent replays.
|
||||
|
||||
For the same graph_key = S, you not only require the input shape to match, but also require the segmentation pattern in cu_seqlens (and window seqlens) to be identical. Otherwise, attention will segment the sequence incorrectly.
|
||||
|
||||
### Rotary buffer management
|
||||
The feature reallocates a larger sin_cos_ws when seq_len increases.
|
||||
The max_content_len is used to make sure the maximum size of the allocated rotary buffer.
|
||||
|
||||
|
||||
## Command Example
|
||||
You can enable CUDA Graph for ViT by setting env variable `SGLANG_VIT_ENABLE_CUDA_GRAPH=1`, for example:
|
||||
```shell Command
|
||||
SGLANG_VIT_ENABLE_CUDA_GRAPH=1 \
|
||||
python3 -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-VL-8B-Instruct
|
||||
```
|
||||
Or you can run CUDA Graph for ViT together with Piecewise CUDA Graph feature by both setting env variable `SGLANG_VIT_ENABLE_CUDA_GRAPH=1` and setting `--enable-piecewise-cuda-graph`, for example:
|
||||
```shell Command
|
||||
SGLANG_VIT_ENABLE_CUDA_GRAPH=1 \
|
||||
python3 -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-VL-8B-Instruct \
|
||||
--piecewise-cuda-graph-max-tokens 4096 \
|
||||
--enable-piecewise-cuda-graph \
|
||||
--piecewise-cuda-graph-compiler eager
|
||||
```
|
||||
|
||||
## Known supported models
|
||||
- Qwen2.5-VL (https://github.com/sgl-project/sglang/pull/14422)
|
||||
- Qwen3-VL (https://github.com/sgl-project/sglang/pull/15320)
|
||||
@@ -0,0 +1,215 @@
|
||||
---
|
||||
title: "Deterministic Inference"
|
||||
metatags:
|
||||
description: "SGLang deterministic inference: consistent outputs for RL training, testing, and production. Supports FlashInfer, FA3, Triton backends with CUDA Graph."
|
||||
---
|
||||
## Why Deterministic Inference Matters
|
||||
|
||||
Deterministic inference ensures consistent LLM outputs across runs, which is critical for:
|
||||
- **Reinforcement Learning**: Ensures consistent logprobs across runs, reducing stochastic noise and making RL training more stable, reproducible, and debuggable.
|
||||
- **Testing & Debugging**: Enables reproducible validation
|
||||
- **Production**: Improves reliability and user experience
|
||||
|
||||
Even with `temperature=0`, standard LLM inference can produce different outputs due to dynamic batching and varying reduction orders in GPU kernels.
|
||||
|
||||
## The Root Cause of Non-Determinism
|
||||
|
||||
The main source is **varying batch sizes**. Different batch sizes cause GPU kernels to split reduction operations differently, leading to different addition orders. Due to floating-point non-associativity (`(a + b) + c ≠ a + (b + c)`), this produces different results even for identical inputs.
|
||||
|
||||
|
||||
## SGLang's Solution
|
||||
|
||||
Building on [Thinking Machines Lab's batch-invariant operators](https://github.com/thinking-machines-lab/batch_invariant_ops), SGLang achieves fully deterministic inference while maintaining compatibility with chunked prefill, CUDA graphs, radix cache, and non-greedy sampling. The development roadmap for deterministic inference features can be found in this [issue](https://github.com/sgl-project/sglang/issues/10278).
|
||||
|
||||
### Supported Backends
|
||||
|
||||
Deterministic inference is only supported with the following three attention backends: **FlashInfer**, **FlashAttention 3 (FA3)**, and **Triton**.
|
||||
|
||||
The following table shows feature compatibility for deterministic inference across different attention backends:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "20%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Attention Backend</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>CUDA Graph</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Chunked Prefill</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Radix Cache</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Non-greedy Sampling (Temp > 0)</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**FlashInfer**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅ Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌ No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ Yes</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**FlashAttention 3 (FA3)**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅ Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅ Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ Yes</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Triton**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅ Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅ Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ Yes</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Enable deterministic inference by adding the `--enable-deterministic-inference` flag:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-8B \
|
||||
--attention-backend fa3 \
|
||||
--enable-deterministic-inference
|
||||
```
|
||||
|
||||
### Server Arguments
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "34%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type/Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enable-deterministic-inference`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>flag; default: disabled</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Enable deterministic inference with batch-invariant operations</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--attention-backend`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>string; default: fa3</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Choose attention backend (flashinfer, fa3, or triton)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
### Example Configurations
|
||||
|
||||
#### Qwen3-8B
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-8B \
|
||||
--attention-backend flashinfer \
|
||||
--enable-deterministic-inference
|
||||
```
|
||||
|
||||
#### Llama Models
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
||||
--attention-backend fa3 \
|
||||
--enable-deterministic-inference
|
||||
```
|
||||
|
||||
#### Qwen3-30B-A3B (MoE Model)
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-30B-A3B \
|
||||
--attention-backend fa3 \
|
||||
--enable-deterministic-inference
|
||||
```
|
||||
|
||||
### Deterministic Inference with Non-Greedy Sampling (Temperature > 0)
|
||||
|
||||
SGLang supports deterministic inference even with non-greedy sampling by using sampling seeds. This is particularly useful for reinforcement learning scenarios like GRPO (Group Relative Policy Optimization) where you need multiple diverse but reproducible responses.
|
||||
|
||||
#### Default Behavior
|
||||
|
||||
By default, SGLang uses a sampling seed of `42` for reproducible sampling:
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
response = requests.post(
|
||||
"http://localhost:30000/generate",
|
||||
json={
|
||||
"text": "Tell me a joke",
|
||||
"sampling_params": {
|
||||
"temperature": 0.8, # Non-greedy sampling
|
||||
"max_new_tokens": 128,
|
||||
},
|
||||
},
|
||||
)
|
||||
print(response.json())
|
||||
# This will always produce the same response across runs
|
||||
```
|
||||
|
||||
#### Generating Multiple Reproducible Responses
|
||||
|
||||
To sample different responses from the same prompt while maintaining reproducibility (e.g., for GRPO training), provide different sampling seeds in your requests:
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
# Prepare a list of sampling seeds for different responses
|
||||
sampling_seeds = [42, 43, 44, 45, 46]
|
||||
|
||||
responses = []
|
||||
for seed in sampling_seeds:
|
||||
response = requests.post(
|
||||
"http://localhost:30000/generate",
|
||||
json={
|
||||
"text": "Tell me a joke",
|
||||
"sampling_params": {
|
||||
"temperature": 0.8,
|
||||
"max_new_tokens": 128,
|
||||
"sampling_seed": seed, # Specify sampling seed
|
||||
},
|
||||
},
|
||||
)
|
||||
responses.append(response.json())
|
||||
|
||||
# Each seed will produce a different but reproducible response
|
||||
# Using the same seed will always produce the same response
|
||||
```
|
||||
|
||||
This approach ensures that:
|
||||
- Different seeds produce diverse responses
|
||||
- The same seed always produces the same response across different runs
|
||||
- Results are reproducible for debugging and evaluation
|
||||
|
||||
|
||||
## Verification
|
||||
|
||||
Run deterministic tests to verify consistent outputs:
|
||||
|
||||
```bash Command
|
||||
# Single test: same prompt, varying batch sizes
|
||||
python3 -m sglang.test.test_deterministic --test-mode single --n-trials 50
|
||||
|
||||
# Prefix test: prompts with different prefix lengths
|
||||
python3 -m sglang.test.test_deterministic --test-mode prefix --n-trials 50
|
||||
|
||||
# Radix Cache Consistency mode: test radix cache determinism (cached vs uncached prefill)
|
||||
python3 -m sglang.test.test_deterministic --test-mode radix_cache
|
||||
```
|
||||
|
||||
Expected result: All tests should show `Unique samples: 1` (perfectly deterministic).
|
||||
@@ -0,0 +1,509 @@
|
||||
---
|
||||
title: "DP, DPA and SGLang DP Router"
|
||||
metatags:
|
||||
description: "Learn the differences between Data Parallelism, Data Parallelism Attention, and SGLang Model Gateway routing for production DP deployments."
|
||||
---
|
||||
|
||||
This guide explains the difference between Data Parallelism (DP) and Data Parallelism Attention (DPA), how to enable each mode correctly, and how to use the SGLang Model Gateway (SMG) for production-grade DP deployments.
|
||||
|
||||
## Data Parallelism (DP)
|
||||
|
||||
**Data Parallelism (DP)** is the most common parallelism strategy that replicates the entire model across multiple GPU sets and processes different batches of requests in parallel. Each GPU set handles independent requests. With dedicated routing strategies, as we will introduce later, with those proper routing algorithms in SGLang Model Gateway, the throughput of your serving system could be multiplied nearly linearly.
|
||||
|
||||
### Key characteristics
|
||||
|
||||
- Each replica has a full copy of the model
|
||||
- Requests are distributed/scattered across replicas
|
||||
- No inter-replica communication during one request's inference (for simple DP)
|
||||
|
||||
## Data Parallelism Attention (DPA)
|
||||
|
||||
**Data Parallelism Attention (DPA)**, also known as DP Attention, is an advanced parallelism strategy. While DPA provides the most significant benefits for **Multi-Head Latent Attention (MLA)** models (such as DeepSeek, MiniMax, Kimi-K2), it also supports **standard attention models** like Qwen.
|
||||
|
||||
### The Problem with Tensor Parallelism for MLA Models
|
||||
|
||||
The most common parallelism strategy for inference is **Tensor Parallelism (TP)**. However, TP might not be the most efficient strategy for certain models. For example, DeepSeek models use MLA and only have **one KV head**. If we use tensor parallelism on 8 GPUs, it will lead to:
|
||||
|
||||
- **Duplicated KV cache** across all GPUs
|
||||
- **Unwanted memory usage** that limits batch size
|
||||
- **Reduced throughput** due to memory constraints
|
||||
|
||||
### How DPA Works
|
||||
|
||||
DPA addresses these limitations by applying **data parallelism specifically to the attention component**.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "50%"}} />
|
||||
<col style={{width: "50%"}} />
|
||||
</colgroup>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", verticalAlign: "top", backgroundColor: "rgba(255,255,255,0.02)"}}>
|
||||
<img src="/images/dpa.png" alt="DPA + EP Architecture" style={{width: "100%", height: "auto"}} />
|
||||
</td>
|
||||
<td style={{padding: "9px 12px", verticalAlign: "top", backgroundColor: "rgba(255,255,255,0.05)"}}>
|
||||
<p><strong>Each DP replica:</strong></p>
|
||||
<ul>
|
||||
<li>Processes different batches independently (can be in different forward modes: prefill, decode, or idle)</li>
|
||||
<li>Maintains its own KV cache (no duplication)</li>
|
||||
<li>Enables significantly larger batch sizes due to memory savings</li>
|
||||
</ul>
|
||||
<p><strong>Communication patterns in DPA + EP:</strong></p>
|
||||
<ul>
|
||||
<li><strong>All2All (Dispatch)</strong>: Routes tokens to expert sub-groups based on gating decisions</li>
|
||||
<li><strong>All2All (Combine)</strong>: Gathers computed results from experts back to original token positions</li>
|
||||
</ul>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
### Key benefits of DPA
|
||||
|
||||
1. **Significantly reduced KV cache memory**: Each DP replica only stores KV cache for its own batches
|
||||
2. **Larger batch sizes**: Memory savings enable larger batch sizes
|
||||
3. **Improved decoding throughput**: Significant throughput gains for MLA-based models
|
||||
4. **Independent forward modes**: Each DP replica can be in different forward modes (prefill, decode, or idle) and handles its assigned batches independently during attention computation
|
||||
|
||||
### DPA with Expert Parallelism for MoE
|
||||
|
||||
For MoE models like DeepSeek, DPA is **often** paired with Expert Parallelism (EP) for best throughput at scale. However, **DPA does not require EP**: you can enable DPA without EP if your deployment does not need expert sharding.
|
||||
|
||||
- Distribute 256+ expert weights across GPUs (cannot fit on a single GPU)
|
||||
- Enable efficient all-to-all token routing via DeepEP
|
||||
- Scale to large clusters (up to 5x throughput improvement over vanilla TP)
|
||||
|
||||
### Recommended setup for DeepSeek
|
||||
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3 \
|
||||
--tp 8 \
|
||||
--dp-size 8 \
|
||||
--ep 8 \
|
||||
--enable-dp-attention \
|
||||
--moe-a2a-backend deepep \
|
||||
--moe-runner-backend deep_gemm
|
||||
```
|
||||
|
||||
> **Note**: `--dp-size` must be explicitly set when using `--enable-dp-attention`. If `dp_size` is 1 (default), DPA will be disabled.
|
||||
|
||||
For detailed EP configuration (DeepEP, Two-Batch Overlap, EPLB), see [Expert Parallelism](/docs/advanced_features/expert_parallelism).
|
||||
|
||||
### Target Models
|
||||
|
||||
DPA supports the following model architectures:
|
||||
|
||||
- **MLA (Multi-Head Latent Attention) models** - where DPA provides the most significant benefits:
|
||||
- DeepSeek family (DeepSeek-V2, DeepSeek-V3, DeepSeek-R1)
|
||||
- MiniMax models
|
||||
- Kimi-K2
|
||||
- Other models using MLA architecture
|
||||
|
||||
- **Standard attention models** - also supported:
|
||||
- Qwen models (see [PR #6121](https://github.com/sgl-project/sglang/pull/6121))
|
||||
|
||||
For models like Llama, with standard GQA, standard DP, or TP is typically recommended.
|
||||
|
||||
To enable DPA, add `--enable-dp-attention` to your server launch command.
|
||||
|
||||
### Activation Logic
|
||||
|
||||
DPA is enabled explicitly via server arguments (CLI or config). You must set both `--dp-size` and `--enable-dp-attention`:
|
||||
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3 \
|
||||
--tp 8 \
|
||||
--dp-size 8 \
|
||||
--enable-dp-attention
|
||||
```
|
||||
|
||||
**Important**: `--dp-size` must be greater than 1 for DPA to work. When `dp_size == 1` (default), `--enable-dp-attention` is automatically disabled. The constraint `tp_size % dp_size == 0` must also be satisfied.
|
||||
|
||||
### Standard DP for MLA models
|
||||
|
||||
Note that MLA models, of course, also support DP. Suppose you want to enable standard DP for MLA models. First, launch each MLA model's replica independently. You may launch these replicas one by one with DPA enabled. After launching each MLA model's replica, launch an SMG and connect all the replicas to the SMG. A detailed explanation of SMG is as follows.
|
||||
|
||||
## Modern Data Parallelism SGLang Model Gateway (SMG)
|
||||
|
||||
### Native DP Mode
|
||||
|
||||
Native DP (built-in Data Parallelism) in SGLang creates multiple worker processes within a single SGLang instance, under the control of `DataParallelController` with the launching parameter of `dp-size`.
|
||||
|
||||
```bash
|
||||
# Native DP mode
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--dp-size 4
|
||||
```
|
||||
|
||||
**Limitations:**
|
||||
|
||||
- Built-in in-process load balancing only (e.g., `round_robin`, `total_requests`, `total_tokens`)
|
||||
- No cache-aware routing
|
||||
- Limited observability and metrics
|
||||
- No fault tolerance or circuit breakers
|
||||
- Not suitable for production workloads
|
||||
|
||||
⚠️ Native DP is **highly not recommended for use right now**. It is only used in some ancient/outdated RL frameworks. You can use SGLang Model Gateway (SMG) to power up your data parallelism in any use case.
|
||||
|
||||
### SMG-Based DP (Recommended)
|
||||
|
||||
Starting from September 2024, SGLang Model Gateway, i.e., SMG, formerly named as SGLang DP Router, was built especially as a production-ready DP routing system with Rust. It starts from DP routing, but later we further expanded its scope to coordinate RL, PD Disaggregation, and other scenarios. This doc only discusses SMG's usage in DP routing. For other usage, please refer to [SGLang Model Gateway Documentation](/docs/advanced_features/sgl_model_gateway).
|
||||
|
||||
> To achieve the best production-level routing performance and reduce the overhead to an extreme extent, we use Rust to build SMG, but not Python, since Python is never FAST enough.
|
||||
|
||||
**We strongly recommend using the SGLang Model Gateway (SMG) for production-grade Data Parallelism.** SMG provides significant advantages over native DP mode.
|
||||
|
||||
```bash
|
||||
# SMG-based DP mode (Recommended)
|
||||
python -m sglang_router.launch_server \
|
||||
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--dp-size 4
|
||||
```
|
||||
|
||||
⚠️ Note that **SMG and Naive DP share the same launching parameter, `--dp-size`**. But the entrypoint of Naive DP is `python -m sglang.launch_server`, and SMG's entrypoint is `python -m sglang_router.launch_server`.
|
||||
|
||||
**Advantages of SMG-Based DP:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "28%"}} />
|
||||
<col style={{width: "34%"}} />
|
||||
<col style={{width: "38%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Feature</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Native DP</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>SMG-Based DP</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Load Balancing</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Built-in in-process methods</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Advanced policies (cache-aware, power-of-two, etc.)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Cache Awareness</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌ No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ Yes - significantly higher cache hit rate</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Throughput</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Baseline</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Significant improvement</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Multi-Node Support</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Limited</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ Full support</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Worker Health Monitoring</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Basic</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ Circuit breakers, health checks</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Reliability</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Basic</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ Retries, rate limiting, queuing</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Observability</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Basic metrics</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ 40+ Prometheus metrics, OpenTelemetry</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Hot Worker Add/Remove</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌ No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ Yes</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
### SMG's Performance
|
||||
|
||||
The cache-aware routing policy in SMG significantly improves performance for workloads with shared prefixes:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "34%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Metric</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Without Cache-Aware</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>With Cache-Aware SMG</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Throughput (token/s)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>82,665</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>158,596 (+92%)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Cache Hit Rate</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>20%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>75% (+275%)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
*Benchmark from [SGLang v0.4 blog](https://lmsys.org/blog/2024-12-04-sglang-v0-4/), workload with multiple long prefix groups, 8x A100 80GB GPUs, dp-size=8*
|
||||
|
||||
### When to Use Each
|
||||
|
||||
**Use Native DP when:**
|
||||
|
||||
- ~Never use Native/Naive DP~
|
||||
- Learning material of DP routing
|
||||
|
||||
**Use SMG-Based DP when:**
|
||||
|
||||
- In any case, when you think DP is needed
|
||||
- Production deployments
|
||||
- Multi-node distributed setups
|
||||
- Workloads with shared prefixes (high cache reuse potential)
|
||||
- You need high availability and reliability features
|
||||
- You require detailed observability and metrics
|
||||
- You want to have highly efficient RL rollout systems
|
||||
|
||||
Note that for RL rollout systems, **there are four crucial reasons that SMG-Based DP is far better than naive DP routing**. Details can be found at [Load Balancing Router in RL](/docs/advanced_features/sglang_for_rl#load-balancing-router).
|
||||
|
||||
### Quick Start For SMG
|
||||
|
||||
**Installation**
|
||||
|
||||
```bash
|
||||
pip install sglang-router
|
||||
# or
|
||||
pip install "sglang[all]"
|
||||
```
|
||||
|
||||
**Option A: Co-launch Workers and SMG (Simplest)**
|
||||
|
||||
This is the easiest way to get started - SMG and workers are launched together:
|
||||
|
||||
```bash
|
||||
python -m sglang_router.launch_server \
|
||||
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--dp-size 4 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Option B: Separate Launch (Multi-Node)**
|
||||
|
||||
For distributed deployments across multiple machines:
|
||||
|
||||
1. Launch workers on each node
|
||||
|
||||
```bash
|
||||
# Node 1
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--port 8000
|
||||
|
||||
# Node 2
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
2. Launch SMG pointing to workers
|
||||
|
||||
```bash
|
||||
python -m sglang_router.launch_router \
|
||||
--worker-urls http://node1:8000 http://node2:8000 \
|
||||
--policy cache_aware \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Option C: Dynamic Worker Registration**
|
||||
|
||||
For elastic deployments where workers can be added/removed dynamically:
|
||||
|
||||
```bash
|
||||
# Launch SMG first
|
||||
python -m sglang_router.launch_router \
|
||||
--policy cache_aware \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
|
||||
# Register workers dynamically
|
||||
curl -X POST http://localhost:30000/workers \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"url": "http://worker1:8000"}'
|
||||
|
||||
curl -X POST http://localhost:30000/workers \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"url": "http://worker2:8000"}'
|
||||
```
|
||||
|
||||
### Load Balancing Policies
|
||||
|
||||
SMG supports multiple load balancing policies:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "24%"}} />
|
||||
<col style={{width: "42%"}} />
|
||||
<col style={{width: "34%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Policy</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Best For</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>cache_aware</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Combines cache locality with load balancing</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Recommended for most workloads</strong></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>round_robin</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Cycles through workers in order</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Simple, predictable distribution</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>random</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Random worker selection</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Baseline, testing</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>power_of_two</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Samples two workers, picks lighter one</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Low latency requirements</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Cache-Aware Policy (Default, Recommended)**
|
||||
|
||||
The cache-aware policy provides the best performance for most workloads:
|
||||
|
||||
```bash
|
||||
python -m sglang_router.launch_router \
|
||||
--worker-urls http://worker1:8000 http://worker2:8000 \
|
||||
--policy cache_aware \
|
||||
--cache-threshold 0.5 \
|
||||
--balance-abs-threshold 32 \
|
||||
--balance-rel-threshold 1.5 \
|
||||
--eviction-interval-secs 120 \
|
||||
--max-tree-size 67108864
|
||||
```
|
||||
|
||||
**How it works:**
|
||||
|
||||
1. Maintains an approximate radix tree for each worker based on request history
|
||||
2. Routes requests to workers with the highest prefix match (cache hit)
|
||||
3. Falls back to shortest-queue routing when load is imbalanced
|
||||
4. Automatically evicts old entries to prevent memory overflow
|
||||
|
||||
### Best Practices
|
||||
|
||||
1. **Start with `cache_aware` policy** - It provides the best balance between cache locality and load distribution for most workloads
|
||||
2. **Use SMG for production** - Prefer `sglang_router.launch_server` over `sglang.launch_server` for better reliability and observability
|
||||
3. **Enable health checks** - Configure `--router-health-check-interval-secs` to detect and remove unhealthy workers automatically
|
||||
|
||||
**Recommended command with best practices applied:**
|
||||
|
||||
```bash
|
||||
python -m sglang_router.launch_server \
|
||||
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--dp-size 4 \
|
||||
--router-policy cache_aware \
|
||||
--router-health-check-interval-secs 30 \
|
||||
--router-prometheus-port 10001 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
For advanced configuration (circuit breakers, retries, Prometheus metrics, K8s integration), see [SGLang Model Gateway Documentation](/docs/advanced_features/sgl_model_gateway).
|
||||
|
||||
### Verifying Traffic Distribution
|
||||
|
||||
After launching SMG, verify that traffic is being distributed correctly:
|
||||
|
||||
**1. Check worker status:**
|
||||
|
||||
```bash
|
||||
curl http://localhost:30000/workers
|
||||
```
|
||||
|
||||
**2. Check load distribution:**
|
||||
|
||||
```bash
|
||||
curl http://localhost:30000/get_loads
|
||||
```
|
||||
|
||||
**3. Monitor metrics (if Prometheus enabled):**
|
||||
|
||||
```bash
|
||||
# Key metrics to check
|
||||
smg_router_requests_total{model="..."}
|
||||
smg_worker_requests_active{worker="..."}
|
||||
sglang_cache_hit_rate{source="..."}
|
||||
```
|
||||
|
||||
For detailed metrics and monitoring setup, see [SGLang Model Gateway Documentation](/docs/advanced_features/sgl_model_gateway).
|
||||
|
||||
## Reference
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "34%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Strategy</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Key Benefit</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Native DP</strong> (<code>--dp-size</code>)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Never</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Easy to understand, not rust based</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>SMG-Based DP</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Production (recommended)</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Cache-aware routing, high availability</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>DPA</strong> (<code>--dp-size N --enable-dp-attention</code>)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek/MLA models</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Eliminates KV cache duplication, improved throughput</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>DPA + EP</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek MoE models</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Significant throughput improvement vs vanilla TP</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Recommended production setup for DeepSeek:**
|
||||
1. Enable **DPA** for attention layers (`--dp-size 8 --enable-dp-attention`)
|
||||
2. Enable **EP** for MoE layers (`--ep 8 --moe-a2a-backend deepep`)
|
||||
3. Use **SMG** with **cache_aware** policy
|
||||
|
||||
**Related documentation:**
|
||||
- [Expert Parallelism](./expert_parallelism) - DeepEP, Two-Batch Overlap, EPLB
|
||||
- [SGLang Model Gateway Documentation](./sgl_model_gateway) - SMG configuration & troubleshooting
|
||||
- [Large-Scale EP Blog](https://lmsys.org/blog/2025-05-05-large-scale-ep/) - 96 GPU deployment guide
|
||||
@@ -0,0 +1,33 @@
|
||||
---
|
||||
title: "DP for Multi-Modal Encoder in SGLang"
|
||||
metatags:
|
||||
description: "Data parallelism for VLM vision encoder in SGLang: reduce TTFT, boost throughput. Supports Qwen2.5-VL, Qwen3-VL, InternVL, GLM-4.5V/4.6V."
|
||||
---
|
||||
A typical VLM architecture involves two main components: an multi-modal encoder and a text decoder.
|
||||
|
||||
Most VLMs utilize a Vision Transformer (ViT) as their multi-modal encoder, it is responsible for processing visual data, extracting features (objects, colors, textures, etc.), and transforming them into a format that can be understood by the model.
|
||||
|
||||
The text deocoder is based on LLM. It processes textual data and generates output based on the encoded visual features.
|
||||
|
||||
However, since the size of ViT is very small compared to language decoders,
|
||||
there is relatively little gain from TP. On the other hand, TP incurs significant communication
|
||||
overhead because of all-reduce being performed after every layer.
|
||||
|
||||
Placing the ViT in data parallel while keeping the LLM in tensor parallel consistently lowers TTFT and boosts end-to-end throughput. In this hybrid layout, the vision front-end becomes parallel and lightweight, while scarce interconnect bandwidth and collective ops are reserved for the LLM.
|
||||
|
||||
Data parallelism replicates the entire model across multiple GPU sets and processes different batches of requests in parallel.
|
||||
|
||||
## Command Example
|
||||
You can enable batch-level DP by setting `mm-enable-dp-encoder`, for example:
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen2.5-VL-7B-Instruct \
|
||||
--tp 2 \
|
||||
--mm-enable-dp-encoder
|
||||
```
|
||||
|
||||
## Known supported models
|
||||
- Qwen2.5-VL (<https://github.com/sgl-project/sglang/pull/13126>)
|
||||
- Qwen3-VL (<https://github.com/sgl-project/sglang/pull/13724>)
|
||||
- InternVL (<https://github.com/sgl-project/sglang/pull/13925>)
|
||||
- GLM-4.5V & GLM-4.6V (<https://github.com/sgl-project/sglang/pull/14097>)
|
||||
@@ -0,0 +1,83 @@
|
||||
---
|
||||
title: "EPD Disaggregation"
|
||||
metatags:
|
||||
description: "SGLang EPD disaggregation: separate encoder, prefill, decode stages for VLM inference. Independent scaling, load balancing, three-tier architecture."
|
||||
---
|
||||
## Why and What is EPD Disaggregation?
|
||||
|
||||
In modern Vision-Language Model (VLM) inference, request execution naturally decomposes into three distinct stages: Encoder, Prefill, and Decode.
|
||||
The Encoder stage performs vision preprocessing and ViT-based image encoding, which is highly compute-intensive but only required during request initialization. The Prefill stage processes the full multimodal input sequence to initialize the language model’s Key-Value (KV) cache, while the Decode stage is dominated by memory bandwidth and KV cache access for autoregressive token generation.
|
||||
|
||||
Existing deployments typically colocate these stages within a unified execution engine, or at best apply Prefill–Decode (PD) disaggregation. However, such designs still tightly couple vision encoding with language prefill, leading to inefficient resource utilization, limited scalability for image-heavy workloads, and suboptimal scheduling under load.
|
||||
|
||||
To address these challenges, we introduce Encoder–Prefill–Decode (EPD) Disaggregation in SGLang. EPD further separates vision encoding from language processing, enabling independent horizontal scaling of encoder servers, improved load balancing for multimodal requests, and seamless integration with existing PD disaggregation to form a fully decoupled three-tier inference architecture.
|
||||
|
||||
### Usage
|
||||
|
||||
You can launch a language-only model using `--language-only`, or an encoder-only model using `--encoder-only`.
|
||||
When launching a language-only model, you must additionally specify the encoder service endpoints via `--encoder-urls`.
|
||||
|
||||
We support multiple encoder transfer backends, including zmq_to_scheduler, zmq_to_tokenizer, and mooncake (the default is zmq_to_scheduler). The backend can be selected using `--encoder-transfer-backend`.
|
||||
|
||||
#### Qwen VL
|
||||
|
||||
- EP Disaggregation
|
||||
|
||||
```bash Command
|
||||
# encoder 0
|
||||
python -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-VL-8B-Instruct \
|
||||
--encoder-only \
|
||||
--encoder-transfer-backend zmq_to_scheduler \
|
||||
--port 30000
|
||||
# encoder 1
|
||||
python -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-VL-8B-Instruct \
|
||||
--encoder-only \
|
||||
--encoder-transfer-backend zmq_to_scheduler \
|
||||
--port 30001
|
||||
# language-only server
|
||||
python -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-VL-8B-Instruct \
|
||||
--language-only \
|
||||
--encoder-urls http://127.0.0.1:30000 http://127.0.0.1:30001 \
|
||||
--encoder-transfer-backend zmq_to_scheduler \
|
||||
--port 30002
|
||||
```
|
||||
|
||||
- EPD Disaggregation
|
||||
|
||||
```bash Command
|
||||
# encoder 0
|
||||
python -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-VL-8B-Instruct \
|
||||
--encoder-only \
|
||||
--encoder-transfer-backend zmq_to_scheduler \
|
||||
--port 30000
|
||||
# encoder 1
|
||||
python -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-VL-8B-Instruct \
|
||||
--encoder-only \
|
||||
--encoder-transfer-backend zmq_to_scheduler \
|
||||
--port 30001
|
||||
# prefill 0
|
||||
python -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-VL-8B-Instruct \
|
||||
--disaggregation-mode prefill \
|
||||
--language-only \
|
||||
--encoder-urls http://127.0.0.1:30000 http://127.0.0.1:30001 \
|
||||
--encoder-transfer-backend zmq_to_scheduler \
|
||||
--port 30002
|
||||
# decode 0
|
||||
python -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-VL-8B-Instruct \
|
||||
--disaggregation-mode decode \
|
||||
--port 30003
|
||||
# router
|
||||
python -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--prefill http://$PREFILL_HOST:30002 \
|
||||
--decode http://$DECODE_HOST:30003 \
|
||||
--port 8000
|
||||
|
||||
```
|
||||
@@ -0,0 +1,288 @@
|
||||
---
|
||||
title: "Expert Parallelism"
|
||||
metatags:
|
||||
description: "SGLang Expert Parallelism: distribute MoE experts across GPUs, DeepEP all-to-all, grouped GEMMs, TBO/SBO overlap, EPLB load balancing."
|
||||
---
|
||||
Expert Parallelism (EP) in SGLang distributes expert weights across multiple devices in Mixture-of-Experts (MoE) models, addressing memory bottlenecks and enabling efficient scaling for high-performance inference. It is particularly vital for serving large-scale MoE models where tokens are dynamically routed to specialized experts across GPUs. By leveraging optimized all-to-all communication and grouped matrix multiplications (GEMMs), EP reduces latency, boosts throughput, and minimizes idle GPU time. SGLang's EP offers strong extensibility through its modular framework, allowing seamless integration of custom kernels, backends, and optimizations without refactoring core logic, supporting diverse hardware and quantization schemes.
|
||||
|
||||
## Supported Backends and Selection Guidance
|
||||
|
||||
SGLang's EP integrates diverse, highly efficient backends for different use cases, allowing fine-grained control over performance trade-offs. Users specify backends via command-line flags:
|
||||
- `--moe-a2a-backend`: Selects the backend for all-to-all communication.
|
||||
- `--moe-runner-backend`: Selects the backend for MoE computation.
|
||||
|
||||
### Backends for All-to-All Communication
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "34%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Backend</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Use Cases</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**`none` (default)**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Disables all-to-all for EP. Uses All-Reduce or All-Gather for token dispatch.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Hybrid EP and TP setups.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`deepep`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepEP, a communication library for efficient token shuffling in MoE models.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Large-scale EP deployments.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`mooncake`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>An extension of DeepEP for elastic inference, leveraging RDMA for high-performance data transfers.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Elastic EP serving.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`flashinfer`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Flashinfer implementation of all-to-all.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Large-scale EP deployments.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`ascend_fuseep`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Ascend NPU native fused all-to-all communication.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Ascend NPU deployments.</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
DeepEP and Mooncake backends support two modes for token dispatch: `normal` mode (optimized for prefill workloads with high throughput) and `low_latency` mode (optimized for decode workloads with low latency and CUDA Graph compatibility). Users are recommended to set `--deepep-mode auto` to enable automatic dispatch mode switching during runtime. Setting `--deepep-mode normal` or `--deepep-mode low_latency` is useful for debugging or development purposes.
|
||||
|
||||
Currently, DeepEP and Mooncake only support cases where `ep_size = tp_size`. For hybrid EP and TP (i.e., `ep_size < tp_size`), only the `none` backend (All-Reduce or All-Gather-based dispatching) is supported.
|
||||
|
||||
### Backends for MoE Computation
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "34%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Backend</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Use Cases</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**`auto` (default)**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Automatically selects the optimal backend based on model architecture, hardware (e.g., NVIDIA architecture like Ampere, Hopper, Blackwell), quantization scheme (e.g., FP8, FP4), and runtime conditions.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>General-purpose deployments; ensures compatibility and performance without user intervention.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`triton`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Triton-based implementation for grouped GEMMs. To achieve higher performance, it's highly recommended to create [tuned configurations](https://github.com/sgl-project/sglang/blob/main/benchmark/kernels/fused_moe_triton/README).</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Custom kernel development or scenarios requiring high extensibility with Torch compilation support.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`deep_gemm`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepGEMM backend optimized for MoE matrix multiplications, supporting contiguous layouts for prefill and masked layouts for decode; often JIT-compiled for performance.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Large-scale EP deployments with FP8 block-wise quantization.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`cutlass`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>CUTLASS-based backend for efficient GEMMs.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>NVIDIA architectures with CUTLASS support.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`flashinfer_trtllm`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>FlashInfer integrated with TensorRT-LLM for accelerated MoE computations, supporting FP4 communication operators and high-performance GEMMs.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Blackwell with TRT-LLM.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`flashinfer_cutlass`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>FlashInfer combined with CUTLASS for high-performance grouped GEMMs in MoE layers, handling FP4/FP8 quantization efficiently.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Blackwell with FP4/FP8 models.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`flashinfer_mxfp4`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>FlashInfer variant optimized for MXFP4 (mixed FP4) quantization in MoE runners, focusing on memory-efficient low-precision inference.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Low-precision models with MXFP4.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`flashinfer_cutedsl`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>FlashInfer with a custom DSL for flexible and efficient MoE kernel generation, integrated with ModelOpt FP4 quantization.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Low-precision models with NVFP4.</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
### Examples
|
||||
|
||||
Launch with DeepEP and DeepGEMM for DeepSeek-V3:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3 --moe-a2a-backend deepep --moe-runner-backend deep_gemm --tp 8 --ep 8
|
||||
```
|
||||
|
||||
## Extensible EP Framework
|
||||
|
||||
SGLang's EP framework provides modular abstractions for easy integration of custom kernels, backends, and optimizations. It decouples the MoE forward pass into stages (dispatch → pre-permute → core runner → post-permute → combine), enabling seamless extensions without refactoring core logic.
|
||||
|
||||
### Framework Overview
|
||||
|
||||
The framework centers on `FusedMoE` as the unified entry point for a single, extensible structure. Key components include:
|
||||
- **Dispatcher**: Manages dispatch/combine for backends like DeepEP (implements `BaseDispatcher` subclasses).
|
||||
- **MoeRunner**: Orchestrates grouped-GEMM execution via `MoeRunnerCore` implementations (e.g., `TritonRunnerCore`).
|
||||
- **PermuteMethodPool**: Auto-registers layout conversions (e.g., pre/post-permute via `register_pre_permute` and `register_post_permute` for dynamic modes, or `register_fused_func` for static, torch.compile-compatible fused operations).
|
||||
- **TopK Router**: Backend-agnostic expert selection.
|
||||
|
||||
This design supports multiple backends via `--moe-a2a-backend` and `--moe-runner-backend`, with quantization integrated through a standardized `apply()` method. The computation flow ensures modularity:
|
||||
|
||||
```text Output
|
||||
[input_hidden_states]
|
||||
|
|
||||
v
|
||||
TopK.forward -> select_experts / triton_kernels.routing / bypass
|
||||
|
|
||||
v
|
||||
[TopKOutput]
|
||||
|
|
||||
v
|
||||
FusedMoE.forward -> Dispatcher.dispatch -> DeepEP / bypass
|
||||
| |
|
||||
| v
|
||||
| [DispatchOutput]
|
||||
| |
|
||||
| v
|
||||
| quant_method.apply -> MoeRunner.forward
|
||||
| | |
|
||||
| | v
|
||||
| | pre-permute + grouped_gemm + post-permute
|
||||
| | |
|
||||
| |--------------
|
||||
| v
|
||||
| [CombineInput]
|
||||
| |
|
||||
| v
|
||||
| Dispatcher.combine -> DeepEP / bypass
|
||||
| |
|
||||
|---------------------
|
||||
v
|
||||
[final_hidden_states]
|
||||
```
|
||||
|
||||
For details, see the [MoE Refactor Roadmap](https://github.com/sgl-project/sglang/issues/8715).
|
||||
|
||||
### Implementing New Backends
|
||||
|
||||
To add a new backend:
|
||||
1. For a new all-to-all dispatcher, implement a `BaseDispatcher` subclass with `dispatch` and `combine` methods.
|
||||
2. For a new MoE runner backend, define a `MoeRunnerCore` subclass for core operations (e.g., grouped GEMMs).
|
||||
3. Define new input/output formats for the dispatcher or model runner (e.g., `RunnerInput`, `RunnerOutput`).
|
||||
4. Register permute/unpermute methods to ensure compatibility:
|
||||
- **Fused Mode** (static, torch.compile-compatible): Use `register_fused_func` for end-to-end operations.
|
||||
- **Permute Mode** (dynamic): Register `register_pre_permute` and `register_post_permute` for flexible layouts.
|
||||
|
||||
See the [MoE Refactor Implementation PR](https://github.com/sgl-project/sglang/pull/9269) for full changes, including type hints and config expansions.
|
||||
|
||||
### Examples
|
||||
|
||||
For an example implementation, see [moe_runner/triton.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/layers/moe/moe_runner/triton.py), which demonstrates Triton-based grouped GEMMs with registered fused and permutation functions.
|
||||
|
||||
## Computation and Communication Overlap
|
||||
|
||||
SGLang's EP employs advanced overlap techniques to hide communication latency behind computation, maximizing GPU utilization in MoE layers.
|
||||
|
||||
### Two-Batch Overlap (TBO)
|
||||
|
||||
TBO splits requests into micro-batches, interleaving attention computation with dispatch/combine operations. Yield points in the execution graph allow pausing for overlaps, increasing overall throughput without peak memory spikes:
|
||||
|
||||
```python Example
|
||||
operations = [
|
||||
self._forward_attn,
|
||||
YieldOperation(), # Overlap with dispatch of prior micro-batch
|
||||
self._forward_dispatch,
|
||||
self._forward_mlp,
|
||||
YieldOperation(), # Overlap with combine
|
||||
self._forward_combine,
|
||||
]
|
||||
```
|
||||
|
||||
Users need to specify `--enable-two-batch-overlap` to unlock up to 2x throughput. For details, see the [Large-Scale EP Blog](https://lmsys.org/blog/2025-05-05-large-scale-ep/#two-batch-overlap).
|
||||
|
||||
### Single-Batch Overlap (SBO)
|
||||
|
||||
SGLang introduces a dispatcher-hook system for Single-Batch Overlap (SBO), enabling the overlap of operations within a single batch—such as shared experts computation with communication—while decentralizing logic to enhance modularity. These hooks execute before and after the `dispatch` and `combine` operations without modifying core MoE modules. This design simplifies interfaces, reduces coupling, and improves extensibility. For implementation details and an example of overlapping shared experts with DeepEP's combine operation, refer to [PR #13327](https://github.com/sgl-project/sglang/pull/13327). Users can set `--enable-single-batch-overlap` to enable this feature.
|
||||
|
||||
|
||||
## Workload Balancer
|
||||
|
||||
SGLang integrates the [Expert Parallelism Load Balancer (EPLB)](https://github.com/deepseek-ai/EPLB) from DeepSeek to address routing imbalances in MoE models. By analyzing expert activation statistics, EPLB computes an optimal expert arrangement, strategically placing or replicating experts to minimize GPU utilization variance, reduce idle cycles, and enhance scalability.
|
||||
|
||||
To enable EPLB, use the flags `--enable-eplb`. For optimal performance, increase batch sizes to stabilize activation statistics and configure periodic rebalancing (e.g., every 1000 requests) to adapt to evolving workloads. Simulations demonstrate significant improvements in load balancedness (ratio of mean to max computation time), correlating strongly with throughput gains.
|
||||
|
||||
For more details, refer to the [EPLB Section in the Large-Scale EP Blog](https://lmsys.org/blog/2025-05-05-large-scale-ep/#expert-parallelism-load-balancer) and the [EPLB Repository](https://github.com/deepseek-ai/eplb).
|
||||
|
||||
|
||||
## EP with Spectulative Decoding
|
||||
|
||||
|
||||
When utilizing speculative decoding with MTP on MoE architectures, use the `--speculative-moe-runner-backend` and `--speculative-moe-a2a-backend` arguments to customize the MoE layer behavior for the draft model. While they default to the target model’s settings, users can differentiate them for varying precisions between target and draft models.
|
||||
|
||||
For model like `nvidia/DeepSeek-R1-0528-NVFP4-v2`, the target model uses NVFP4 precision while the draft model uses BF16. To apply `flashinfer_trtllm` kernel for target MoE layer while falling back to triton fused MoE kernel for draft MoE layer, users can set the arguments as follows:
|
||||
```text Output
|
||||
...
|
||||
--moe-runner-backend flashinfer_trtllm \
|
||||
--speculative-moe-runner-backend triton \
|
||||
...
|
||||
```
|
||||
|
||||
|
||||
## Ascend NPU Guidance
|
||||
### Guidance on SGLang configuration in Ascend NPU
|
||||
- `--moe-a2a-backend` only supports deepep and ascend_fuseep backends,
|
||||
|
||||
- `deepep`: The mechanism is consistent with the above description.
|
||||
|
||||
- `ascend_fuseep`: Offer a large fused operator which integrates all operations between dispatch and combine to boost MoE computation. Only used for decode stage in PD Disaggregation Mode.
|
||||
|
||||
- `--moe-runner-backend` parameter does not need to be configured.
|
||||
|
||||
- `--deepep-mode`:
|
||||
|
||||
- In PD mixed mode, please set `--deepep-mode` auto.
|
||||
|
||||
- In PD Disaggregation Mode, prefill instance sets `--deepep-mode` normal, and decode instance sets `--deepep-mode` low_latency.
|
||||
|
||||
### DeepEP Ascend Introduction
|
||||
DeepEP Ascend is the adapted version of the DeepEP communication library for Huawei Ascend NPUs, specifically designed for Mixture-of-Experts (MoE) model Expert Parallelism (EP). It supports the Ant-moving Function (Split the sequence length into rounds for streaming batch transmission) to optimize the buffer size occupied during collective communication in prefill stage, especially for long sequences.
|
||||
|
||||
Ant-moving Function can be enabled for both the dispatch and combine phases via the following environment variables:
|
||||
|
||||
- `DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS`: Enable ant-moving function in dispatch stage. Indicates the number of tokens transmitted per round on each rank, default 8192.
|
||||
|
||||
- `DEEPEP_NORMAL_LONG_SEQ_ROUND`: Enable ant-moving function in dispatch stage. Indicates the number of rounds transmitted on each rank, default 1.
|
||||
|
||||
- `DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ`: Enable ant-moving function in combine stage, default 0 (means disabled).
|
||||
|
||||
`DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS * DEEPEP_NORMAL_LONG_SEQ_ROUND` means input sequence length. When the input sequence length exceeds 8192, it is recommended to enable the ant-moving function in both dispatch and combine phase.
|
||||
|
||||
The environment variable `HCCL_BUFFSIZE` is used to configure the buffer size (MB) actually allocated. Its calculation formula is as follows:
|
||||
```text Output
|
||||
# Enable Ant-moving Function
|
||||
HCCL_BUFFSIZE >= 2 * (102MB + 4MB + DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS * (hidden_size + hidden_size + hidden_size) * topk) + PADDING_BUFFSIZE
|
||||
|
||||
# Disable Ant-moving Function
|
||||
HCCL_BUFFSIZE >= 2 * (102MB + 4MB + TOTAL_SEQ_LEN * (hidden_size + hidden_size) * topk) + PADDING_BUFFSIZE
|
||||
```
|
||||
Wherein the parameters are described as follows:
|
||||
|
||||
- `hidden_size`: hidden size in model config.
|
||||
|
||||
- `topk`: The number of selected routing experts.
|
||||
|
||||
- `TOTAL_SEQ_LEN`: input sequence length.
|
||||
|
||||
- `PADDING_BUFFSIZE`: A value of 20 or greater is recommended.
|
||||
@@ -0,0 +1,298 @@
|
||||
---
|
||||
title: "Model Forward Hooks"
|
||||
metatags:
|
||||
description: "SGLang forward hooks: attach PyTorch hooks to model submodules via JSON config. Log activations, debug internals, export hidden states."
|
||||
---
|
||||
|
||||
## Model Hooks
|
||||
|
||||
SGLang supports attaching PyTorch forward hooks to specific submodules in the loaded model, configured entirely via `server_args` JSON.
|
||||
|
||||
This is useful for:
|
||||
|
||||
* Logging intermediate activations
|
||||
* Debugging model internals
|
||||
* Exporting hidden states to external tooling
|
||||
|
||||
Hooks are attached once during `ModelRunner.initialize` and run on every forward pass.
|
||||
|
||||
***
|
||||
### Configuration overview
|
||||
|
||||
Hooks are configured via a `ServerArgs` field:
|
||||
|
||||
```python Example
|
||||
class ServerArgs:
|
||||
...
|
||||
# For forward hooks
|
||||
forward_hooks: Optional[List[dict[str, Any]]] = None
|
||||
````
|
||||
|
||||
In JSON form, a minimal configuration looks like:
|
||||
|
||||
```jsonc Example
|
||||
{
|
||||
"forward_hooks": [
|
||||
{
|
||||
"name": "outer_linear_hooks",
|
||||
"target_modules": ["outer.0", "outer.1"],
|
||||
"hook_factory": "my_project.hooks:dummy_hook_factory",
|
||||
"config": {
|
||||
"tag": "outer-layer"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### Top-level fields
|
||||
|
||||
* `forward_hooks` (optional list of objects)
|
||||
Each element is a hook spec describing:
|
||||
|
||||
* Which modules to target
|
||||
* Which Python factory to call
|
||||
* What configuration to pass into that factory
|
||||
|
||||
***
|
||||
### Hook spec schema
|
||||
|
||||
Each entry in `forward_hooks` is a JSON object with the following shape:
|
||||
|
||||
```jsonc Example
|
||||
{
|
||||
"name": "optional-descriptive-name",
|
||||
"target_modules": ["pattern1", "pattern2", "..."],
|
||||
"hook_factory": "module.submodule:factory_name",
|
||||
"config": {
|
||||
"...": "arbitrary JSON"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### `name` (optional)
|
||||
|
||||
* Human-readable name for logging.
|
||||
* Used only in log messages such as:
|
||||
|
||||
```text Output
|
||||
Registered forward hook 'outer_linear_hooks' on outer.0
|
||||
```
|
||||
|
||||
#### `target_modules` (required)
|
||||
|
||||
* List of **module name patterns** used to match entries in `model.named_modules()`.
|
||||
* Patterns are matched using `fnmatch.fnmatch`, so:
|
||||
|
||||
* `"outer.0"` matches exactly `"outer.0"`.
|
||||
* `"outer.*"` matches `"outer.0"`, `"outer.1"`, `"outer.inner"`, etc.
|
||||
* `"outer.inner.*"` matches children under `outer.inner`.
|
||||
|
||||
> If no modules match the given patterns, hook registration does **not** fail.
|
||||
> Instead, SGLang logs a warning and continues:
|
||||
>
|
||||
> ```text
|
||||
> No modules matched hook spec 'name' patterns=['...']
|
||||
> ```
|
||||
|
||||
#### `hook_factory` (required)
|
||||
|
||||
* String path to the Python factory function that creates the hook.
|
||||
* Supported formats:
|
||||
|
||||
* `"package.module:factory_name"`
|
||||
* `"package.module.submodule.factory_name"`
|
||||
|
||||
The path is resolved via:
|
||||
|
||||
```python Example
|
||||
def resolve_callable(path: Optional[str]) -> Optional[Callable]:
|
||||
if path is None:
|
||||
return None
|
||||
|
||||
if ":" in path:
|
||||
module_name, fn_name = path.split(":", 1)
|
||||
else:
|
||||
parts = path.split(".")
|
||||
if len(parts) < 2:
|
||||
raise ValueError(
|
||||
f"Invalid hook callable path '{path}'. "
|
||||
"Expected 'module.submodule:factory' or 'module.submodule.factory'."
|
||||
)
|
||||
*mod_parts, fn_name = parts
|
||||
module_name = ".".join(mod_parts)
|
||||
|
||||
module = importlib.import_module(module_name)
|
||||
try:
|
||||
return getattr(module, fn_name)
|
||||
except AttributeError as e:
|
||||
raise AttributeError(
|
||||
f"Module '{module_name}' has no attribute '{fn_name}' "
|
||||
f"(from hook path '{path}')"
|
||||
) from e
|
||||
```
|
||||
|
||||
**Failure modes**:
|
||||
|
||||
* If the path is malformed (not enough dots and no `:`), a `ValueError` is raised at startup.
|
||||
* If the module imports but the attribute is missing, an `AttributeError` is raised with a clear error message.
|
||||
* If the hook factory returns `None`, a warning is logged and no hook is registered for that spec (initialization continues).
|
||||
|
||||
The first two cause initialization to fail fast with a descriptive error; the last one is non-fatal.
|
||||
|
||||
#### `config` (optional)
|
||||
|
||||
* Arbitrary JSON object.
|
||||
* Passed directly to the hook factory as a Python `dict`.
|
||||
* This lets you parameterize hook behavior from config (e.g. tags, log levels, sampling rates, etc.).
|
||||
|
||||
***
|
||||
### Hook lifecycle and behavior
|
||||
|
||||
Hooks are registered in `ModelRunner.initialize()`:
|
||||
|
||||
```python Example
|
||||
if server_args.forward_hooks:
|
||||
register_forward_hooks(self.model, server_args.forward_hooks)
|
||||
```
|
||||
|
||||
The actual registration logic is implemented by `register_forward_hooks`:
|
||||
|
||||
```python Example
|
||||
def register_forward_hooks(model: nn.Module, hook_specs: List[dict[str, Any]]) -> None:
|
||||
"""
|
||||
hook_specs is a list of dicts from server_args.forward_hooks.
|
||||
Attaches forward hooks to the matching modules.
|
||||
"""
|
||||
name_to_module = dict(model.named_modules())
|
||||
|
||||
for spec in hook_specs:
|
||||
spec_name = spec.get("name", "")
|
||||
target_patterns = spec.get("target_modules", [])
|
||||
if not target_patterns:
|
||||
logger.warning(
|
||||
f"Hook spec '{spec_name}' has no 'target_modules', skipping"
|
||||
)
|
||||
continue
|
||||
|
||||
hook_factory_path = spec.get("hook_factory")
|
||||
if not hook_factory_path:
|
||||
logger.warning(
|
||||
f"Hook spec '{spec_name}' has no 'hook_factory', skipping"
|
||||
)
|
||||
continue
|
||||
|
||||
config = spec.get("config") or {}
|
||||
hook_factory = resolve_callable(hook_factory_path)
|
||||
|
||||
hook = hook_factory(config) if hook_factory else None
|
||||
if hook is None:
|
||||
logger.warning(
|
||||
f"Hook factory '{hook_factory_path}' for spec '{spec_name}' "
|
||||
"returned None, not registering any hook"
|
||||
)
|
||||
continue
|
||||
|
||||
# Resolve patterns like "model.layers.*.mlp"
|
||||
matched = []
|
||||
for name, module in name_to_module.items():
|
||||
if any(fnmatch.fnmatch(name, pattern) for pattern in target_patterns):
|
||||
matched.append((name, module))
|
||||
|
||||
if not matched:
|
||||
logger.warning(
|
||||
f"No modules matched hook spec '{spec_name}' "
|
||||
f"patterns={target_patterns}"
|
||||
)
|
||||
continue
|
||||
|
||||
for module_name, module in matched:
|
||||
if hook:
|
||||
_ = module.register_forward_hook(hook)
|
||||
logger.info(
|
||||
f"Registered forward hook '{spec_name}' "
|
||||
f"on {module_name}"
|
||||
)
|
||||
```
|
||||
|
||||
Key points:
|
||||
|
||||
* Hooks are **forward hooks only** (via `module.register_forward_hook`).
|
||||
* They are attached once at initialization.
|
||||
* Hook handles are currently not stored on `ModelRunner` (they cannot be removed later via this API).
|
||||
* Failure to match any modules is non-fatal; a warning is logged instead.
|
||||
* If a hook factory returns `None`, a warning is logged and that spec is skipped.
|
||||
|
||||
***
|
||||
### Writing a hook factory
|
||||
|
||||
A hook factory is a regular Python function:
|
||||
|
||||
* Takes a `config: dict` (from JSON)
|
||||
* Returns a forward hook function with signature `(module, inputs, output)`
|
||||
|
||||
Example:
|
||||
|
||||
```python Example
|
||||
HOOK_CALLS = []
|
||||
|
||||
def dummy_hook_factory(config):
|
||||
"""Factory that returns a forward hook capturing a tag from config."""
|
||||
tag = config.get("tag", "default")
|
||||
|
||||
def hook(module, inputs, output):
|
||||
HOOK_CALLS.append(
|
||||
{
|
||||
"module_type": type(module).__name__,
|
||||
"tag": tag,
|
||||
"shape": tuple(output.shape),
|
||||
}
|
||||
)
|
||||
return output # must return output if you don’t want to modify the tensor
|
||||
|
||||
return hook
|
||||
```
|
||||
|
||||
In JSON:
|
||||
|
||||
```jsonc Example
|
||||
{
|
||||
"forward_hooks": [
|
||||
{
|
||||
"name": "capture_outer",
|
||||
"target_modules": ["outer.0", "outer.1"],
|
||||
"hook_factory": "my_project.hooks:dummy_hook_factory",
|
||||
"config": {
|
||||
"tag": "outer"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
This will:
|
||||
|
||||
* Resolve `my_project.hooks:dummy_hook_factory` to a Python callable.
|
||||
* Call it with `config = {"tag": "outer"}`.
|
||||
* Use the returned hook for all modules matching `outer.0` and `outer.1`.
|
||||
* Append metadata about each call to `HOOK_CALLS`.
|
||||
|
||||
***
|
||||
### Summary
|
||||
|
||||
* Define `forward_hooks` as a list of specs in `ServerArgs` to turn on the feature.
|
||||
|
||||
* Each spec:
|
||||
|
||||
* selects modules via `target_modules` (glob patterns over `model.named_modules()`),
|
||||
* points to a hook factory via `hook_factory`,
|
||||
* passes arbitrary `config` into that factory.
|
||||
|
||||
* Hook factories are resolved via `resolve_callable`, which supports `module:factory` and `module.submodule.factory`.
|
||||
|
||||
* Hooks are standard PyTorch forward hooks, attached once at startup and invoked on every forward pass.
|
||||
|
||||
* Misconfiguration is either:
|
||||
|
||||
* **fatal and explicit** (bad path / missing attribute), or
|
||||
* **non-fatal with clear warnings** (no targets matched, or factory returned `None`).
|
||||
@@ -0,0 +1,8 @@
|
||||
---
|
||||
title: "Hierarchical KV Caching (HiCache)"
|
||||
metatags:
|
||||
description: "SGLang HiCache: three-tier KV caching (GPU, CPU, storage) for long-context and multi-turn inference. Supports Mooncake, 3FS, NIXL backends."
|
||||
---
|
||||
- [Hicache Best Practices](./hicache_best_practices)
|
||||
- [Hicache Design](./hicache_design)
|
||||
- [Hicache Storage Runtime Attach Detach](./hicache_storage_runtime_attach_detach)
|
||||
@@ -0,0 +1,9 @@
|
||||
Hierarchical KV Caching (HiCache)
|
||||
=================================
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
hicache_best_practices.md
|
||||
hicache_design.md
|
||||
hicache_storage_runtime_attach_detach.md
|
||||
@@ -0,0 +1,202 @@
|
||||
---
|
||||
title: "SGLang HiCache Best Practices"
|
||||
metatags:
|
||||
description: "HiCache configuration guide: memory layout, prefetch policies, PD disaggregation, HF3FS and Mooncake deployment, custom storage backends."
|
||||
---
|
||||
## Why HiCache Matters
|
||||
|
||||
SGLang HiCache extends the traditional RadixAttention with a three-tier hierarchical KV caching system that dramatically improves performance for long-context and multi-turn conversation scenarios. By intelligently managing KV caches across GPU memory, host memory, and external storage backends, HiCache addresses the fundamental capacity bottleneck that limits cache hit rates in conventional systems.
|
||||
|
||||
## Configuration Guidelines
|
||||
|
||||
## Core HiCache Parameters
|
||||
|
||||
```bash Command
|
||||
# Essential HiCache flags
|
||||
--page-size 64 # Page size for cache management
|
||||
--enable-hierarchical-cache # Enable HiCache
|
||||
--hicache-ratio 2 # Host memory ratio (2x GPU memory)
|
||||
--hicache-size 100 # Host memory size in GBs, will override the above ratio
|
||||
--hicache-io-backend kernel # The I/O backend of moving data between CPU and GPU
|
||||
--hicache-write-policy write_through # Cache write policy from GPU to CPU
|
||||
--hicache-storage-backend # Optional storage backend (e.g., hf3fs, mooncake, etc.)
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- Besides configuring `--hicache-storage-backend` at startup, SGLang also supports **runtime attach/detach** of the HiCache storage backend (no restart required) via HTTP admin endpoints. See [Runtime Attach/Detach HiCache Storage Backend](./hicache_storage_runtime_attach_detach).
|
||||
|
||||
## Key Configurations with Storage Backends Enabled
|
||||
|
||||
### Memory Layout Optimization
|
||||
|
||||
```bash Command
|
||||
# Page-first: Optimized for I/O efficiency with zero-copy (recommended with kernel backend)
|
||||
--hicache-mem-layout page_first
|
||||
# Page-first-direct: Optimized for direct I/O operations (Compatible with fa3 and same zero-copy performance as page_first)
|
||||
--hicache-mem-layout page_first_direct
|
||||
# Layer-first
|
||||
--hicache-mem-layout layer_first
|
||||
```
|
||||
**Layout Compatibility:**
|
||||
- `page_first`: Only compatible with `kernel` I/O backend, automatically switches to `layer_first` with `direct` backend
|
||||
- `page_first_direct`: Specifically designed for `direct` I/O backend with optimized memory organization
|
||||
|
||||
### Prefetch Policies
|
||||
|
||||
```bash Command
|
||||
# Best-effort: Terminate prefetch when needed
|
||||
--hicache-storage-prefetch-policy best_effort
|
||||
# Wait-complete: Ensure complete prefetch, higher cache reuse
|
||||
--hicache-storage-prefetch-policy wait_complete
|
||||
# Timeout: Balance between completion and best-effort
|
||||
--hicache-storage-prefetch-policy timeout
|
||||
```
|
||||
|
||||
### Integration with PD Disaggregation
|
||||
|
||||
HiCache works seamlessly with PD Disaggregation. You can choose between two configurations:
|
||||
|
||||
1. **Prefill-only HiCache**: Enable HiCache only on Prefill nodes, allowing KV cache sharing among Prefill instances
|
||||
2. **Full HiCache with async offloading**: Enable HiCache on Prefill nodes and async KV cache offloading on Decode nodes, allowing Prefill nodes to reuse KV caches from Decode nodes in multi-turn dialogue scenarios
|
||||
|
||||
```bash Command
|
||||
# Prefill node with HiCache enabled for cross-prefill sharing (ideal for SystemPrompt scenarios)
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path /xxx/DeepSeek-R1/ \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 10000 \
|
||||
--enable-metrics \
|
||||
--enable-cache-report \
|
||||
--mem-fraction-static 0.85 \
|
||||
--page-size 64 \
|
||||
--enable-hierarchical-cache \
|
||||
--hicache-ratio 2 \
|
||||
--hicache-size 0 \
|
||||
--hicache-mem-layout page_first_direct \
|
||||
--hicache-io-backend direct \
|
||||
--hicache-write-policy write_through \
|
||||
--hicache-storage-backend hf3fs \
|
||||
--hicache-storage-prefetch-policy wait_complete \
|
||||
--disaggregation-ib-device mlx5_0 \
|
||||
--disaggregation-mode prefill \
|
||||
--disaggregation-transfer-backend mooncake
|
||||
|
||||
# Decode node with async offloading enabled for KV cache reuse by Prefill (ideal for multi-turn conversations)
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path /xxx/DeepSeek-R1/ \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 10000 \
|
||||
--enable-metrics \
|
||||
--enable-cache-report \
|
||||
--page-size 64 \
|
||||
--hicache-ratio 2 \
|
||||
--hicache-size 0 \
|
||||
--hicache-mem-layout page_first_direct \
|
||||
--hicache-io-backend direct \
|
||||
--hicache-write-policy write_through \
|
||||
--hicache-storage-backend hf3fs \
|
||||
--hicache-storage-prefetch-policy wait_complete \
|
||||
--disaggregation-decode-enable-offload-kvcache \ # Enable async KV cache offloading in decode node
|
||||
--disaggregation-ib-device mlx5_0 \
|
||||
--disaggregation-mode decode \
|
||||
--disaggregation-transfer-backend mooncake
|
||||
```
|
||||
|
||||
|
||||
### Deployment with HF3FS
|
||||
|
||||
Here is an example of deploying DeepSeek-R1 with HiCache-HF3FS. For more details, see the [HF3FS Documentation](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/hf3fs/docs).
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path /xxx/DeepSeek-R1/ \
|
||||
--log-level info \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 10000 \
|
||||
--enable-metrics \
|
||||
--enable-cache-report \
|
||||
--page-size 64 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--enable-hierarchical-cache \
|
||||
--hicache-ratio 2 \
|
||||
--hicache-size 0 \
|
||||
--hicache-mem-layout page_first_direct \
|
||||
--hicache-io-backend direct \
|
||||
--hicache-write-policy write_through \
|
||||
--hicache-storage-backend hf3fs \
|
||||
--hicache-storage-prefetch-policy wait_complete \
|
||||
```
|
||||
|
||||
### Deployment with Mooncake
|
||||
|
||||
Here is an example of deploying Qwen3-235B-A22B-Instruct-2507 with Mooncake. For more details, see the [Mooncake Documentation](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/mooncake_store).
|
||||
|
||||
```bash Command
|
||||
# Set Mooncake environment variables
|
||||
export MOONCAKE_TE_META_DATA_SERVER="http://127.0.0.1:8080/metadata"
|
||||
export MOONCAKE_GLOBAL_SEGMENT_SIZE=816043786240
|
||||
export MOONCAKE_PROTOCOL="rdma"
|
||||
export MOONCAKE_DEVICE="$DEVICE_LIST"
|
||||
export MOONCAKE_MASTER=127.0.0.1:50051
|
||||
|
||||
# Launch SGLang server with Mooncake backend
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path $MODEL_PATH \
|
||||
--tp 8 \
|
||||
--page-size 64 \
|
||||
--enable-hierarchical-cache \
|
||||
--hicache-ratio 2 \
|
||||
--hicache-mem-layout page_first_direct \
|
||||
--hicache-io-backend direct \
|
||||
--hicache-storage-backend mooncake \
|
||||
--hicache-write-policy write_through \
|
||||
--hicache-storage-prefetch-policy timeout
|
||||
```
|
||||
|
||||
|
||||
## Custom Storage Backend Integration
|
||||
|
||||
To integrate a new storage backend:
|
||||
|
||||
1. **Implement three core methods:**
|
||||
- `get(key)`: Retrieve value by key
|
||||
- `exists(key)`: Check key existence
|
||||
- `set(key, value)`: Store key-value pair
|
||||
|
||||
2. **Register your backend:** Add your storage backend to the HiCache [BackendFactory](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/storage/backend_factory.py#L188)
|
||||
|
||||
The HiCache controller handles all scheduling and synchronization automatically.
|
||||
|
||||
### Dynamic Backend Loading
|
||||
|
||||
Alternatively, you can use dynamic loading to avoid hard-coding your backend in the repository:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path your-model \
|
||||
--enable-hierarchical-cache \
|
||||
--hicache-storage-backend dynamic \
|
||||
--hicache-storage-backend-extra-config '{"backend_name":"custom_backend_name", "module_path": "your_module_path", "class_name": "YourHiCacheClassName"}'
|
||||
```
|
||||
|
||||
**Configuration Parameters:**
|
||||
- `--hicache-storage-backend`: Set to `dynamic`
|
||||
- `--hicache-storage-backend-extra-config`: JSON configuration with:
|
||||
- `backend_name`: Custom backend identifier
|
||||
- `module_path`: Python module path to your implementation
|
||||
- `class_name`: Your HiCache implementation class name
|
||||
- `interface_v1`: 0 (disable) or 1 (enable) to control usage of batch_get_v1 and batch_set_v1 methods
|
||||
|
||||
|
||||
## Community and Support
|
||||
|
||||
- **GitHub Issues**: Report bugs and feature requests
|
||||
- **Slack Channel**: Join community discussions in #sgl-kv-cache-store
|
||||
- **Documentation**: Refer to storage backend-specific guides
|
||||
|
||||
***
|
||||
*This document will be continuously updated based on community feedback and new features. Contributions and suggestions are welcome!*
|
||||
@@ -0,0 +1,164 @@
|
||||
---
|
||||
title: "HiCache System Design and Optimization"
|
||||
metatags:
|
||||
description: "HiCache architecture: HiRadixTree metadata, L1/L2/L3 workflow, prefetch strategies, write-back policies, zero-copy transfers, multi-rank sync."
|
||||
---
|
||||
This document provides a comprehensive overview of SGLang HiCache, covering its system architecture, workflow and key components. It also details configuration parameters, optimization techniques, and integration with various L3 storage backends, serving as a complete reference for users and developers to understand and tune HiCache for efficient LLM inference.
|
||||
|
||||
## Why and What is HiCache?
|
||||
|
||||
In large language model inference, the prefill phase is often time-consuming: input sequences need to be first converted into Key-Value cache (KV cache) for subsequent decoding. When multiple requests share the same prefix, the KV cache for that prefix is identical. By caching and reusing these shared KV caches, redundant computation can be avoided. To address this, SGLang introduced RadixAttention, which leverages idle GPU memory to cache and reuse prefix KV caches, and **HiCache**, which extends this idea to host memory and distributed storage.
|
||||
|
||||
Inspired by the classic three-level cache design of modern CPUs, HiCache organizes GPU memory as L1, host memory as L2, and distributed storage as L3. This hierarchy enables HiCache to fully exploit the "idle" storage space of GPUs and CPUs, while integrating distributed cache systems such as Mooncake, 3FS, NIXL, and AIBrix KVCache for global KV cache storage and scheduling. As a result, HiCache significantly expands KV cache capacity while maintaining strong read performance—especially in workloads such as multi-QA and long-context inference, where KV cache reuse is frequent. For detailed benchmark results, see [this blog](https://lmsys.org/blog/2025-09-10-sglang-hicache/).
|
||||
|
||||
|
||||
## System Design
|
||||
|
||||
### Overall Architecture
|
||||
|
||||
In many modern CPU architectures, the small but fast L1 and L2 caches are private to each core, enabling rapid access to the hottest data, while the larger L3 cache is shared across all cores to significantly reduce redundancy within the cache. Similarly, in HiCache, the L1 and L2 KV caches are private to each inference instance, whereas the L3 KV cache is shared among all inference instances within the cluster.
|
||||
|
||||
### HiRadixTree: Metadata Organization in HiCache
|
||||
|
||||
For KV cache data organization, HiCache builds upon the RadixTree structure introduced in RadixAttention and proposes HiRadixTree. In RadixAttention, each node of the RadixTree corresponds to the KV cache of a consecutive span of tokens in GPU memory. A path from the root to a leaf node represents the prefix of a request, and shared prefixes across multiple requests can reuse the same nodes, thereby avoiding redundant storage.
|
||||
|
||||
HiRadixTree extends this idea: each node corresponds to the KV cache of a span of consecutive tokens and records where that KV cache is stored—whether in local GPU memory, CPU memory, L3 storage, or multiple of these tiers. If stored locally, HiRadixTree maintains precise metadata, including the exact storage address. However, to reduce overhead, HiRadixTree does not store or continuously synchronize metadata for L3 KV cache. Instead, when accessing L3 data, it queries the backend in real time to retrieve the necessary metadata, such as whether the data exists and on which server and location it resides.
|
||||
|
||||
### Overall Workflow
|
||||
|
||||
The workflow of HiCache mainly involves three key operations: **local match**, **prefetch** and **write-back**. When the system receives a new request, it first searches the local L1 and L2 caches for matching KV caches. For parts not found locally, it attempts to prefetch from L3. After prefetching, all required KV caches are loaded into the GPU for computation. Once the prefill computation is complete, the system considers storing the newly generated data into L2 or L3.
|
||||
|
||||
<Frame>
|
||||
<img src="https://lmsys.org/images/blog/hicache/hicache_overview.png" alt="HiCache Workflow"/>
|
||||
</Frame>
|
||||
|
||||
### Local Match
|
||||
|
||||
Local matching is the first step in HiCache's workflow, where incoming request tokens are matched against the HiRadixTree to locate cached KV data in local memory tiers (L1 GPU memory and L2 host memory).
|
||||
|
||||
The matching algorithm traverses the HiRadixTree from the root node, following child nodes that match the token sequence prefix. At each node, the incoming token sequence is compared with the node’s stored token sequence. When `page_size > 1`, matching is performed at the page granularity to optimize memory access patterns. If a match terminates within a node’s stored sequence, the node is automatically split to create an exact boundary, improving the efficiency of future matches.
|
||||
|
||||
The algorithm returns a continuous prefix of the request, with the first part residing in L1 and the latter part in L2.
|
||||
|
||||
Since the process only requires traversing the local HiRadixTree and does not involve any actual data copying, local matching is extremely fast.
|
||||
|
||||
### Prefetch from L3
|
||||
|
||||
Data prefetching is one of HiCache’s core optimization techniques, designed to proactively load KV caches from L3 storage into local L2 memory, thereby reducing access latency during subsequent operations.
|
||||
|
||||
**Prefetch Trigger Conditions**:
|
||||
After local matching, for the parts not found in L1 or L2, the system queries L3 to retrieve metadata for the next continuous matching KV caches. If the length of hit cache in L3 exceeds a threshold (default: 256 tokens, configurable), a prefetch operation is triggered.
|
||||
|
||||
**Prefetch Strategies**: HiCache provides three different prefetch termination strategies to address different scenario needs:
|
||||
- **best_effort**: Terminates immediately when GPU can execute prefill computation, with no waiting time, suitable for scenarios extremely sensitive to latency.
|
||||
- **wait_complete**: Must wait for all prefetch operations to complete, suitable for scenarios requiring high cache hit rates.
|
||||
- **timeout**: Terminates after specified time or when complete, balancing latency and cache hit rate needs.
|
||||
|
||||
After prefetching stops, the data already fetched is used together with the local data for the prefill computation.
|
||||
|
||||
For **timeout** strategy, HiCache introduces two configuration parameters to support fine-grained control over prefetch timeout conditions:
|
||||
|
||||
* `prefetch_timeout_base`: the base timeout, representing overhead unrelated to the number of tokens (e.g., scheduling and synchronization).
|
||||
* `prefetch_timeout_per_ki_token`: the incremental timeout per thousand tokens.
|
||||
|
||||
The timeout is computed as:
|
||||
|
||||
```python Example
|
||||
timeout = prefetch_timeout_base + prefetch_timeout_per_ki_token * num_token_to_fetch / 1024
|
||||
```
|
||||
|
||||
### Data Write-back
|
||||
|
||||
The write-back mechanism is responsible for moving frequently accessed KV caches from L1 to L2 and L3, enabling larger and longer-term storage as well as cache sharing across instances.
|
||||
|
||||
**Configurable Write-back Policies**: HiCache supports three write-back strategies:
|
||||
|
||||
* **write_through**: Every access is immediately written back to the next level. When bandwidth is sufficient, this strategy provides the strongest caching benefit.
|
||||
* **write_through_selective**: Data is written back only after the access frequency exceeds a threshold. This strategy backs up only hot data, reducing I/O overhead.
|
||||
* **write_back**: Data is written back to the next level only when it is evicted from the upper level. This strategy alleviates storage pressure and is suitable for scenarios where storage capacity is limited but memory utilization must be maximized.
|
||||
|
||||
**Cross-instance Sharing**: When data is written back from L2 to L3, only data not already present in L3 is transferred. KV caches stored in L3 can then be shared across all SGLang instances in the cluster (depending on the L3 backend implementation), significantly improving cache hit rates within the same memory budget.
|
||||
|
||||
### Multi-Rank Synchronization
|
||||
|
||||
During multi-GPU parallel computation, such as tensor parallelism (TP), HiCache must ensure consistent states across different ranks. Therefore, critical computation steps require the use of `all_reduce` for state synchronization.
|
||||
|
||||
For example, during prefetching, `all_reduce(op=min)` is used to ensure that all ranks obtain the same number of L3 hits, preventing inconsistent judgments about whether the prefetch threshold has been reached. Similarly, after prefetching completes or terminates, `all_reduce(op=min)` is again required to guarantee consensus among ranks on the prefix length of the successfully retrieved KV cache.
|
||||
|
||||
### Data Transfer Optimization
|
||||
|
||||
**Zero-Copy Data Transfers**: Both prefetching and write-back involve substantial data movement. Minimizing the number of data copies can significantly improve system performance. HiCache supports passing memory addresses and sizes directly when transferring data from L2 memory to an L3 backend.
|
||||
|
||||
**“Batch-Oriented” Data Organization**: The granularity of data reads and writes has a major impact on performance. To address this, HiCache L3 stores and transfers KV cache data at the granularity of **pages** and supports different data layouts beyond the existing `layer first` scheme, including `page first` and `page first direct`. Under the `page first` and `page first direct` layouts, all KV cache data belonging to the same page is placed in contiguous memory, allowing it to be passed as a single object to L3 using zero-copy transfers.
|
||||
|
||||
<Frame>
|
||||
<img src="https://lmsys.org/images/blog/hicache/hicache_layout.png" alt="HiCache L2 MEM layout"/>
|
||||
</Frame>
|
||||
|
||||
However, because GPU KV computation is naturally performed layer by layer, the GPU inherently operates in a `layer first` layout. When transferring `page first` data from L2 to the GPU, data must be transferred at the granularity of one token per layer. The `page first direct` layout mitigates this issue by grouping together all tokens of a given layer within a page, allowing transfers from L2 to GPU to be aggregated at the page-layer level.
|
||||
|
||||
**CPU-to-GPU Transfer Optimizations**: In HiCache, moving data from CPU memory to GPU is as performance-critical as prefetching data from L3 to L2. HiCache employs several optimizations for this process:
|
||||
|
||||
* **Compute-Transfer Overlap**: During the prefill phase, when transferring data from CPU to GPU, HiCache overlaps layers by concurrently loading the KV cache of layer N+1 while computing layer N. This effectively hides data transfer latency.
|
||||
* **GPU-assisted I/O Kernels**: On top of `cudaMemcpyAsync`, HiCache implements a set of GPU-assisted I/O kernels specifically optimized for KV cache transfers between CPU and GPU. Compared to the baseline approach, these kernels achieve up to 3x higher transfer speed.
|
||||
|
||||
**Write-back Optimization for MLA**: For MHA (Multi-Head Attention) models under multi-TP, each rank holds `1/tp_size` of a token’s KV data. In contrast, for MLA (Multi-Layer Attention) models, all ranks hold the complete and identical KV data for each token. HiCache includes a dedicated optimization for MLA: only one rank initiates the write-back operation, ensuring that data is not redundantly stored across ranks.
|
||||
|
||||
### Integration with PD-Disaggregation Deployment Mode
|
||||
|
||||
SGLang supports a PD (Prefill-Decode) disaggregation deployment mode through the Mooncake TransferEngine (for details, see [this doc](./pd_disaggregation)). In the PD-disaggregation deployment mode, HiCache can be enabled on both the prefill nodes and decode nodes to optimize prefill performance. If enabled on decode nodes, the decode output will also be written back to L3.
|
||||
|
||||
### Unified Interfaces and Rich L3 Storage Backends
|
||||
|
||||
HiCache encapsulates all read, write, and query operations on L3 backends within the `class HiCacheStorage(ABC)`, exposing a set of simple and consistent interfaces. This design supports a wide range of L3 storage backends and allows users to select the one that best fits their specific use cases.
|
||||
|
||||
- **Mooncake**: Mooncake is a high-performance caching system for LLM inference that leverages RDMA and multi-NIC resources to enable zero-copy, ultra-fast data transfers. Try Mooncake [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/mooncake_store).
|
||||
|
||||
- **DeepSeek 3FS (HF3FS)**: HF3FS is a Kubernetes-native distributed storage solution with operator-based deployment. Try HF3FS [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/hf3fs).
|
||||
|
||||
- **NIXL**: NIXL provides a unified API for accessing various storage plugins, including but not limited to DeepSeek's 3FS, GPU Direct Storage (GDS) and Amazon S3-compatible object storage. Try NIXL [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/nixl).
|
||||
|
||||
- **AIBrix KVCache**: AIBrix KVCache is a production-ready KVCache Offloading Framework, which enables efficient memory tiering and low-overhead cross-engine reuse. Try AIBrix KVCache [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/aibrix_kvcache).
|
||||
|
||||
- **HiCacheFile**: A simple file-based storage backend for demonstration purposes.
|
||||
|
||||
Specifically, **LMCache**, an efficient KV cache layer for enterprise-scale LLM inference, provides an alternative solution to HiCache. Try LMCache [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/lmcache).
|
||||
|
||||
## Related Parameters
|
||||
|
||||
- **`--enable-hierarchical-cache`**: Enable hierarchical cache functionality. This is required to use HiCache.
|
||||
|
||||
- **`--hicache-ratio HICACHE_RATIO`**: The ratio of the size of host KV cache memory pool to the size of device pool. For example, a value of 2 means the host memory pool is twice as large as the device memory pool. The value of this parameter must be greater than 1, as the current implementation requires the host memory allocated for the KV cache to be larger than the device memory allocated for the KV cache.
|
||||
|
||||
- **`--hicache-size HICACHE_SIZE`**: The size of host KV cache memory pool in gigabytes. This parameter overrides `hicache-ratio` if set. For example, `--hicache-size 30` allocates 30GB (1GB = 1e9 bytes) for the host memory pool **for each rank**. If there are 8 ranks, then the total memory size is 240GB. Just like `hicache-ratio`, the value of this parameter must be larger than the size of device memory allocated for KV cache.
|
||||
|
||||
**Note**: `--hicache-ratio` and `--hicache-size` are two critical parameters. In general, a larger HiCache size leads to a higher cache hit rate, which improves prefill performance. However, the relationship between cache size and hit rate is not linear. Once most reusable KV data—especially hot tokens—are already cached, further increasing the size may yield only marginal performance gains. Users can set these parameters based on their workload characteristics and performance requirements.
|
||||
|
||||
- **`--page-size PAGE_SIZE`**: The number of tokens per page. This parameter determines the granularity of KV cache storage and retrieval. Larger page sizes reduce metadata overhead and improve I/O efficiency for storage backends, but may lower the cache hit rate when only part of a page matches the stored KV cache. For workloads with long common prefixes, larger pages can improve performance, while workloads with more diverse prefixes may benefit from smaller pages. See [Data Transfer Optimization](#data-transfer-optimization) for how page granularity affects I/O performance.
|
||||
|
||||
- **`--hicache-storage-prefetch-policy {best_effort,wait_complete,timeout}`**: Controls when prefetching from storage should stop. See [Prefetch from L3](#prefetch-from-l3) for details.
|
||||
- `best_effort`: Prefetch as much as possible without blocking
|
||||
- `wait_complete`: Wait for prefetch to complete before proceeding
|
||||
- `timeout`: Terminates after specified time or when complete (Recommended for production environments, as setting an appropriate timeout helps the system meet required SLOs)
|
||||
|
||||
- **`--hicache-write-policy {write_back,write_through,write_through_selective}`**: Controls how data is written from faster to slower memory tiers. See [Data Write-back](#data-write-back) for details.
|
||||
- `write_through`: Immediately writes data to all tiers (strongest caching benefits)
|
||||
- `write_through_selective`: Uses hit-count tracking to back up only frequently accessed data
|
||||
- `write_back`: Writes data back to slower tiers only when eviction is needed (reduces I/O load)
|
||||
|
||||
- **`--hicache-io-backend {direct,kernel}`**: Choose the I/O backend for KV cache transfer between CPU and GPU. See [Data Transfer Optimization](#data-transfer-optimization) for details.
|
||||
- `direct`: Standard CUDA memory copy operations
|
||||
- `kernel`: GPU-assisted I/O kernels (recommended for better performance)
|
||||
|
||||
- **`--hicache-mem-layout {layer_first,page_first,page_first_direct}`**: Memory layout for the host memory pool. See [Data Transfer Optimization](#data-transfer-optimization) for details.
|
||||
- `layer_first`: Compatible with GPU computation kernels (default for GPU memory)
|
||||
- `page_first`: Optimized for I/O efficiency
|
||||
- `page_first_direct`: Groups all tokens of a given layer within a page, allowing transfers from L2 to GPU to be aggregated at the page-layer level
|
||||
|
||||
- **`--hicache-storage-backend {file,mooncake,hf3fs,nixl,aibrix,dynamic}`**: Choose the storage backend for the L3 tier. Built-in backends: file, mooncake, hf3fs, nixl, aibrix. For dynamic backend, use --hicache-storage-backend-extra-config to specify: `backend_name` (custom name), `module_path` (Python module path), `class_name` (backend class name). See [Unified Interfaces and Rich L3 Storage Backends](#unified-interfaces-and-rich-l3-storage-backends) for available backends.
|
||||
|
||||
- **`--enable-lmcache`**: Using LMCache as an alternative hierarchical cache solution.
|
||||
|
||||
- **`--hicache-storage-backend-extra-config HICACHE_STORAGE_BACKEND_EXTRA_CONFIG`**: the extra config can be either
|
||||
- a JSON string containing extra configuration for the storage backend, e.g., `--hicache-storage-backend-extra-config '{"prefetch_threshold":512, "prefetch_timeout_base": 0.5, "prefetch_timeout_per_ki_token": 0.25}' `, or
|
||||
- a TOML or JSON or YAML file specifying the extra configuration for the storage backend (to differentiate from the JSON string input, prepend a `@` in front of the file name), e.g., `--hicache-storage-backend-extra-config "@config.toml"` where `config.toml` is the config file containing the complex configurations. This can be useful when the configuration consists of many or complex key-value pairs (for instance, it is preferred to use a config file for NIXL backend as its configurations can be complex).
|
||||
@@ -0,0 +1,133 @@
|
||||
---
|
||||
title: "Runtime Attach/Detach HiCache Storage Backend (No Restart)"
|
||||
metatags:
|
||||
description: "Dynamically attach/detach HiCache L3 storage backends at runtime via HTTP API. No restart required, idle-state safety checks."
|
||||
---
|
||||
This document explains how to **dynamically attach/detach the HiCache L3 storage backend at runtime** (e.g., `mooncake` / `hf3fs` / `nixl` / `file` / `aibrix` / `eic`) while **SGLang is already running and serving traffic**, without restarting the process.
|
||||
|
||||
For safety and consistency, the current implementation **strictly requires** these operations to happen only when the service is **idle**:
|
||||
|
||||
- **No running requests**
|
||||
- **No waiting/queued requests**
|
||||
|
||||
If the idle condition is not met, the API will fail fast (HTTP 400) and **will not modify** the current service state.
|
||||
|
||||
***
|
||||
## 1. Background and implementation overview
|
||||
|
||||
### 1.1 Architecture / control path
|
||||
|
||||
The control path is:
|
||||
|
||||
1. **HTTP Server** (`python/sglang/srt/entrypoints/http_server.py`)
|
||||
- Exposes `PUT /hicache/storage-backend`, `DELETE /hicache/storage-backend`, `GET /hicache/storage-backend`
|
||||
2. **TokenizerManager** (`python/sglang/srt/managers/tokenizer_communicator_mixin.py`)
|
||||
- Sends the request to the Scheduler via `_Communicator`
|
||||
3. **Scheduler** (`python/sglang/srt/managers/scheduler.py`)
|
||||
- Performs a **strict idle check**
|
||||
- Calls `tree_cache.attach_storage_backend(...)` / `detach_storage_backend(...)`
|
||||
4. **HiRadixCache** (`python/sglang/srt/mem_cache/hiradix_cache.py`)
|
||||
- Parses `hicache_storage_backend_extra_config_json` (supports both backend config and prefetch knobs)
|
||||
- Calls `cache_controller.attach_storage_backend(...)` / `detach_storage_backend(...)`
|
||||
5. **HiCacheController** (`python/sglang/srt/managers/cache_controller.py`)
|
||||
- Creates/destroys the storage backend instance (via `StorageBackendFactory`)
|
||||
- Starts/stops backend background threads at runtime (prefetch/backup)
|
||||
|
||||
***
|
||||
## 2. Idle-state requirement (strict)
|
||||
|
||||
The Scheduler uses a stricter `_is_idle_for_hicache_storage_op()`:
|
||||
|
||||
- `_is_no_request()` is true (covers running/overlap/pp/disagg and other active states)
|
||||
- `waiting_queue` is empty
|
||||
- `grammar_queue` is empty (if the grammar backend is enabled)
|
||||
|
||||
If the condition is not met, attach/detach returns an error like:
|
||||
|
||||
- `Reject attach: scheduler is not idle. #queue-req=... #running-req=...`
|
||||
|
||||
<Tip>
|
||||
before switching, drain upstream traffic and wait for the server to become idle, then call attach/detach.
|
||||
</Tip>
|
||||
|
||||
### 2.1 DP (data parallel) semantics
|
||||
|
||||
When `dp_size > 1`, the tokenizer dispatches the request to **all DP scheduler instances** and aggregates their responses:
|
||||
|
||||
- The final `success` is **true only if all DP ranks return success**
|
||||
- The final `message` concatenates messages from all DP ranks
|
||||
|
||||
This is intended to prevent “silent partial success”, but it also means you may see:
|
||||
|
||||
- Overall **failure** even though **some ranks already succeeded**
|
||||
|
||||
Currently there is **no automatic partial rollback** across DP ranks (see TODO in code). Operationally:
|
||||
|
||||
- Prefer to keep backend config identical across ranks
|
||||
- If attach fails, immediately call detach (best-effort/idempotent), fix config, then retry attach
|
||||
|
||||
***
|
||||
## 3. How to use (HTTP Admin API)
|
||||
|
||||
The examples below assume your SGLang HTTP server is at `http://127.0.0.1:30000`.
|
||||
|
||||
### 3.1 Query current storage backend status
|
||||
|
||||
```bash Command
|
||||
curl -s http://127.0.0.1:30000/hicache/storage-backend
|
||||
```
|
||||
|
||||
Example response:
|
||||
|
||||
```json Config
|
||||
{
|
||||
"hicache_storage_backend": "mooncake",
|
||||
"hicache_storage_backend_extra_config": "{\"master_server_address\":\"127.0.0.1:50051\", ...}"
|
||||
}
|
||||
```
|
||||
|
||||
### 3.2 Attach (enable) a storage backend
|
||||
```bash Command
|
||||
curl -s -X PUT http://127.0.0.1:30000/hicache/storage-backend \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"hicache_storage_backend": "mooncake"
|
||||
}'
|
||||
```
|
||||
|
||||
```bash Command
|
||||
curl -s -X PUT http://127.0.0.1:30000/hicache/storage-backend \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"hicache_storage_backend": "mooncake",
|
||||
"hicache_storage_backend_extra_config_json": "{\"master_server_address\":\"127.0.0.1:50051\",\"protocol\":\"tcp\",\"global_segment_size\":\"4gb\",\"prefetch_threshold\":256}",
|
||||
"hicache_storage_prefetch_policy": "timeout"
|
||||
}'
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `hicache_storage_backend_extra_config_json` can include both:
|
||||
- **Backend configuration** (e.g., Mooncake master/metadata/protocol, etc.)
|
||||
- **Prefetch configuration** (`prefetch_threshold`, `prefetch_timeout_base`, `prefetch_timeout_per_ki_token`, `hicache_storage_pass_prefix_keys`)
|
||||
|
||||
### 3.3 Detach (disable) the storage backend
|
||||
|
||||
```bash Command
|
||||
curl -s -X DELETE http://127.0.0.1:30000/hicache/storage-backend
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- Detach only makes SGLang **stop using** the L3 storage backend and stops prefetch/backup threads
|
||||
- It **does not automatically delete** data stored in Mooncake/HF3FS (or other remote backends)
|
||||
|
||||
***
|
||||
## 4. Behavior and caveats
|
||||
|
||||
- **No restart required**: attach/detach switches in-process at runtime
|
||||
- **Must be idle**: otherwise the request is rejected to avoid consistency issues
|
||||
- **Host KV layout constraints still apply**: for example, Mooncake still requires layouts like `page_first/page_first_direct/page_head`; if the server's HiCache host-memory layout does not satisfy the backend requirements, attach will fail with an error
|
||||
- **Observability**:
|
||||
- After attach, `server_args.hicache_storage_backend*` is updated on both the tokenizer and scheduler sides
|
||||
- If metrics are enabled, attach will create a storage metrics collector in `HiRadixCache` on demand
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
title: "Hyperparameter Tuning"
|
||||
metatags:
|
||||
description: "SGLang performance tuning: batch size, token usage, mem-fraction-static, chunked-prefill-size, CUDA graph, DP/TP optimization."
|
||||
---
|
||||
## Achieving high throughput for offline batch inference
|
||||
|
||||
Achieving a large batch size is the most important thing for attaining high throughput in offline batch inference.
|
||||
When the server is running at full load in a steady state, look for the following in the log:
|
||||
|
||||
```text Output
|
||||
Decode batch. #running-req: 233, #token: 370959, token usage: 0.82, cuda graph: True, gen throughput (token/s): 4594.01, #queue-req: 317
|
||||
```
|
||||
|
||||
### Adjust the request submission speed to control `#queue-req`
|
||||
|
||||
`#queue-req` indicates the number of requests in the queue.
|
||||
If you frequently see `#queue-req: 0`, it suggests that your client code is submitting requests too slowly.
|
||||
A healthy range for `#queue-req` is `100 - 2000`.
|
||||
However, avoid making `#queue-req` too large, as this will increase the scheduling overhead on the server.
|
||||
|
||||
### Achieve a high `token usage`
|
||||
|
||||
`token usage` indicates the KV cache memory utilization of the server. `token usage > 0.9` means good utilization.
|
||||
|
||||
If you frequently see `token usage < 0.9` and `#queue-req > 0`, it means the server is too conservative about taking in new requests. You can decrease `--schedule-conservativeness` to a value like 0.3.
|
||||
The case of a server being too conservative can happen when users send many requests with a large `max_new_tokens` but the requests stop very early due to EOS or stop strings.
|
||||
|
||||
On the other hand, if you see `token usage` very high and you frequently see warnings like
|
||||
`KV cache pool is full. Retract requests. #retracted_reqs: 1, #new_token_ratio: 0.9998 -> 1.0000`, you can increase `--schedule-conservativeness` to a value like 1.3.
|
||||
If you see `KV cache pool is full. Retract requests.` occasionally but not frequently (~1 time per minute), it is okay.
|
||||
|
||||
### Tune `--mem-fraction-static` to increase KV cache pool capacity
|
||||
SGLang allocates memory as follows:
|
||||
|
||||
Total memory usage = model weights + KV cache pool + CUDA graph buffers + activations
|
||||
|
||||
The `--mem-fraction-static` parameter determines how much memory is allocated to the first two components:
|
||||
|
||||
mem_fraction_static = (model weights + KV cache pool) / GPU memory capacity
|
||||
|
||||
To support higher concurrency, you should maximize the KV cache pool capacity by setting `--mem-fraction-static` as high as possible while still reserving enough memory for activations and CUDA graph buffers.
|
||||
|
||||
SGLang uses simple heuristics to set the default value of `--mem-fraction-static`, but you can optimize it for your use cases.
|
||||
As a rule of thumb, reserving 5–8 GB of memory for activations is typically sufficient. You can check this by inspecting the logs just before the server is ready.
|
||||
Look for log entries like this:
|
||||
|
||||
```text Output
|
||||
[2025-08-11 17:17:03] max_total_num_tokens=665690, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=4096, context_len=65536, available_gpu_mem=13.50 GB
|
||||
```
|
||||
|
||||
Check the `available_gpu_mem` value.
|
||||
- If it is between 5–8 GB, the setting is good.
|
||||
- If it is too high (e.g., 10 - 20 GB), increase `--mem-fraction-static` to allocate more memory to the KV cache.
|
||||
- If it is too low, you risk out-of-memory (OOM) errors later, so decrease `--mem-fraction-static`.
|
||||
|
||||
Another straightforward approach is to increase `--mem-fraction-static` in increments of 0.01 until you encounter OOM errors for your workloads.
|
||||
|
||||
### Avoid out-of-memory errors by tuning `--chunked-prefill-size`, `--mem-fraction-static`, and `--max-running-requests`
|
||||
|
||||
If you encounter out-of-memory (OOM) errors, you can adjust the following parameters:
|
||||
|
||||
- If OOM occurs during prefill, try reducing `--chunked-prefill-size` to `4096` or `2048`. This saves memory but slows down the prefill speed for long prompts.
|
||||
- If OOM occurs during decoding, try lowering `--max-running-requests`.
|
||||
- You can also reduce `--mem-fraction-static` to a smaller value, such as 0.8 or 0.7. This decreases the memory usage of the KV cache memory pool and helps prevent OOM errors during both prefill and decoding. However, it limits maximum concurrency and reduces peak throughput.
|
||||
|
||||
### Tune `--cuda-graph-max-bs`
|
||||
By default, CUDA graph is enabled only for small batch sizes (e.g., less than 160 or 256).
|
||||
However, for some models, especially at large tensor parallelism sizes, CUDA graph can be useful for batch sizes up to 512 or 768.
|
||||
Therefore, it may be beneficial to increase `--cuda-graph-max-bs` to a larger value.
|
||||
Note that CUDA graph consumes more memory, so you may need to reduce `--mem-fraction-static` at the same time.
|
||||
|
||||
### Tune `--dp-size` and `--tp-size`
|
||||
|
||||
Data parallelism is better for throughput. When there is enough GPU memory, always favor data parallelism for throughput. Refer to [SGLang Model Gateway (former Router)](../advanced_features/sgl_model_gateway) for a better data parallelism rather than using `dp_size` parameter.
|
||||
|
||||
### Try other options
|
||||
|
||||
- `torch.compile` accelerates small models on small batch sizes. You can enable it with `--enable-torch-compile`.
|
||||
- Try other quantization (e.g. FP8 quantization with `--quantization fp8`)
|
||||
- Try other parallelism strategies (e.g. [expert parallelism](https://lmsys.org/blog/2025-05-05-large-scale-ep/)) or DP attention for deepseek models (with `--enable-dp-attention --dp-size 8`).
|
||||
- If the workload has many shared prefixes, try `--schedule-policy lpm`. Here, `lpm` stands for longest prefix match. It reorders requests to encourage more cache hits but introduces more scheduling overhead.
|
||||
@@ -0,0 +1,714 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# LoRA Serving"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"SGLang enables the use of [LoRA adapters](https://arxiv.org/abs/2106.09685) with a base model. By incorporating techniques from [S-LoRA](https://arxiv.org/pdf/2311.03285) and [Punica](https://arxiv.org/pdf/2310.18547), SGLang can efficiently support multiple LoRA adapters for different sequences within a single batch of inputs."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Arguments for LoRA Serving"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The following server arguments are relevant for multi-LoRA serving:\n",
|
||||
"\n",
|
||||
"* `enable_lora`: Enable LoRA support for the model. This argument is automatically set to True if `--lora-paths` is provided for backward compatibility.\n",
|
||||
"\n",
|
||||
"* `enable_lora_overlap_loading`: Enable asynchronous LoRA weight loading in order to overlap H2D transfers with GPU compute. This should be enabled if you find that your LoRA workloads are bottlenecked by adapter weight loading, for example when frequently loading large LoRA adapters.\n",
|
||||
"\n",
|
||||
"* `lora_paths`: The list of LoRA adapters to load. Each adapter must be specified in one of the following formats: <PATH> | <NAME>=<PATH> | JSON with schema {\"lora_name\":str,\"lora_path\":str,\"pinned\":bool}.\n",
|
||||
"\n",
|
||||
"* `max_loras_per_batch`: Maximum number of adaptors used by each batch. This argument can affect the amount of GPU memory reserved for multi-LoRA serving, so it should be set to a smaller value when memory is scarce. Defaults to be 8.\n",
|
||||
"\n",
|
||||
"* `max_loaded_loras`: If specified, it limits the maximum number of LoRA adapters loaded in CPU memory at a time. The value must be greater than or equal to `max-loras-per-batch`.\n",
|
||||
"\n",
|
||||
"* `lora_eviction_policy`: LoRA adapter eviction policy when GPU memory pool is full. `lru`: Least Recently Used (default, better cache efficiency). `fifo`: First-In-First-Out.\n",
|
||||
"\n",
|
||||
"* `lora_backend`: The backend of running GEMM kernels for Lora modules. Currently we support Triton LoRA backend (`triton`) and Chunked SGMV backend (`csgmv`). In the future, faster backend built upon Cutlass or Cuda kernels will be added.\n",
|
||||
"\n",
|
||||
"* `max_lora_rank`: The maximum LoRA rank that should be supported. If not specified, it will be automatically inferred from the adapters provided in `--lora-paths`. This argument is needed when you expect to dynamically load adapters of larger LoRA rank after server startup.\n",
|
||||
"\n",
|
||||
"* `lora_target_modules`: The union set of all target modules where LoRA should be applied (e.g., `q_proj`, `k_proj`, `gate_proj`). If not specified, it will be automatically inferred from the adapters provided in `--lora-paths`. This argument is needed when you expect to dynamically load adapters of different target modules after server startup. You can also set it to `all` to enable LoRA for all supported modules. However, enabling LoRA on additional modules introduces a minor performance overhead. If your application is performance-sensitive, we recommend only specifying the modules for which you plan to load adapters.\n",
|
||||
"\n",
|
||||
"* `--max-lora-chunk-size`: Maximum chunk size for the ChunkedSGMV LoRA backend. Only used when --lora-backend is 'csgmv'. Choosing a larger value might improve performance. Please tune this value based on your hardware and workload as needed. Defaults to 16.\n",
|
||||
"\n",
|
||||
"* `tp_size`: LoRA serving along with Tensor Parallelism is supported by SGLang. `tp_size` controls the number of GPUs for tensor parallelism. More details on the tensor sharding strategy can be found in [S-Lora](https://arxiv.org/pdf/2311.03285) paper.\n",
|
||||
"\n",
|
||||
"From client side, the user needs to provide a list of strings as input batch, and a list of adaptor names that each input sequence corresponds to."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Usage\n",
|
||||
"\n",
|
||||
"### Serving Single Adaptor"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Note:** SGLang supports LoRA adapters through two APIs:\n",
|
||||
"\n",
|
||||
"1. **OpenAI-Compatible API** (`/v1/chat/completions`, `/v1/completions`): Use the `model:adapter-name` syntax. See [OpenAI API with LoRA](../basic_usage/openai_api_completions.ipynb#Using-LoRA-Adapters) for examples.\n",
|
||||
"\n",
|
||||
"2. **Native API** (`/generate`): Pass `lora_path` in the request body (shown below)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"import requests\n",
|
||||
"\n",
|
||||
"from sglang.test.doc_patch import launch_server_cmd\n",
|
||||
"from sglang.utils import wait_for_server, terminate_process"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"server_process, port = launch_server_cmd(\n",
|
||||
" # Here we set max-loras-per-batch to 2: one slot for adaptor and another one for base model\n",
|
||||
" \"\"\"\n",
|
||||
"python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
|
||||
" --enable-lora \\\n",
|
||||
" --lora-paths lora0=algoprog/fact-generation-llama-3.1-8b-instruct-lora \\\n",
|
||||
" --max-loras-per-batch 2 \\\n",
|
||||
" --log-level warning \\\n",
|
||||
"\"\"\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"wait_for_server(f\"http://localhost:{port}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"url = f\"http://127.0.0.1:{port}\"\n",
|
||||
"json_data = {\n",
|
||||
" \"text\": [\n",
|
||||
" \"List 3 countries and their capitals.\",\n",
|
||||
" \"List 3 countries and their capitals.\",\n",
|
||||
" ],\n",
|
||||
" \"sampling_params\": {\"max_new_tokens\": 32, \"temperature\": 0},\n",
|
||||
" # The first input uses lora0, and the second input uses the base model\n",
|
||||
" \"lora_path\": [\"lora0\", None],\n",
|
||||
"}\n",
|
||||
"response = requests.post(\n",
|
||||
" url + \"/generate\",\n",
|
||||
" json=json_data,\n",
|
||||
")\n",
|
||||
"print(f\"Output 0: {response.json()[0]['text']}\")\n",
|
||||
"print(f\"Output 1: {response.json()[1]['text']}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(server_process)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Serving Multiple Adaptors"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"server_process, port = launch_server_cmd(\"\"\"\n",
|
||||
"python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
|
||||
" --enable-lora \\\n",
|
||||
" --lora-paths lora0=algoprog/fact-generation-llama-3.1-8b-instruct-lora \\\n",
|
||||
" lora1=Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16 \\\n",
|
||||
" --max-loras-per-batch 2 \\\n",
|
||||
" --log-level warning \\\n",
|
||||
"\"\"\")\n",
|
||||
"\n",
|
||||
"wait_for_server(f\"http://localhost:{port}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"url = f\"http://127.0.0.1:{port}\"\n",
|
||||
"json_data = {\n",
|
||||
" \"text\": [\n",
|
||||
" \"List 3 countries and their capitals.\",\n",
|
||||
" \"List 3 countries and their capitals.\",\n",
|
||||
" ],\n",
|
||||
" \"sampling_params\": {\"max_new_tokens\": 32, \"temperature\": 0},\n",
|
||||
" # The first input uses lora0, and the second input uses lora1\n",
|
||||
" \"lora_path\": [\"lora0\", \"lora1\"],\n",
|
||||
"}\n",
|
||||
"response = requests.post(\n",
|
||||
" url + \"/generate\",\n",
|
||||
" json=json_data,\n",
|
||||
")\n",
|
||||
"print(f\"Output 0: {response.json()[0]['text']}\")\n",
|
||||
"print(f\"Output 1: {response.json()[1]['text']}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(server_process)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Dynamic LoRA loading"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Instead of specifying all adapters during server startup via `--lora-paths`. You can also load & unload LoRA adapters dynamically via the `/load_lora_adapter` and `/unload_lora_adapter` API.\n",
|
||||
"\n",
|
||||
"When using dynamic LoRA loading, it's recommended to explicitly specify both `--max-lora-rank` and `--lora-target-modules` at startup. For backward compatibility, SGLang will infer these values from `--lora-paths` if they are not explicitly provided. However, in that case, you would have to ensure that all dynamically loaded adapters share the same shape (rank and target modules) as those in the initial `--lora-paths` or are strictly \"smaller\"."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"lora0 = \"Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16\" # rank - 4, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj\n",
|
||||
"lora1 = \"algoprog/fact-generation-llama-3.1-8b-instruct-lora\" # rank - 64, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj\n",
|
||||
"lora0_new = \"philschmid/code-llama-3-1-8b-text-to-sql-lora\" # rank - 256, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# The `--target-lora-modules` param below is technically not needed, as the server will infer it from lora0 which already has all the target modules specified.\n",
|
||||
"# We are adding it here just to demonstrate usage.\n",
|
||||
"server_process, port = launch_server_cmd(\"\"\"\n",
|
||||
" python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
|
||||
" --enable-lora \\\n",
|
||||
" --cuda-graph-max-bs 2 \\\n",
|
||||
" --max-loras-per-batch 2 \\\n",
|
||||
" --max-lora-rank 256\n",
|
||||
" --lora-target-modules all\n",
|
||||
" --log-level warning\n",
|
||||
" \"\"\")\n",
|
||||
"\n",
|
||||
"url = f\"http://127.0.0.1:{port}\"\n",
|
||||
"wait_for_server(url)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Load adapter lora0"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"response = requests.post(\n",
|
||||
" url + \"/load_lora_adapter\",\n",
|
||||
" json={\n",
|
||||
" \"lora_name\": \"lora0\",\n",
|
||||
" \"lora_path\": lora0,\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"if response.status_code == 200:\n",
|
||||
" print(\"LoRA adapter loaded successfully.\", response.json())\n",
|
||||
"else:\n",
|
||||
" print(\"Failed to load LoRA adapter.\", response.json())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Load adapter lora1:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"response = requests.post(\n",
|
||||
" url + \"/load_lora_adapter\",\n",
|
||||
" json={\n",
|
||||
" \"lora_name\": \"lora1\",\n",
|
||||
" \"lora_path\": lora1,\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"if response.status_code == 200:\n",
|
||||
" print(\"LoRA adapter loaded successfully.\", response.json())\n",
|
||||
"else:\n",
|
||||
" print(\"Failed to load LoRA adapter.\", response.json())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Check inference output:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"url = f\"http://127.0.0.1:{port}\"\n",
|
||||
"json_data = {\n",
|
||||
" \"text\": [\n",
|
||||
" \"List 3 countries and their capitals.\",\n",
|
||||
" \"List 3 countries and their capitals.\",\n",
|
||||
" ],\n",
|
||||
" \"sampling_params\": {\"max_new_tokens\": 32, \"temperature\": 0},\n",
|
||||
" # The first input uses lora0, and the second input uses lora1\n",
|
||||
" \"lora_path\": [\"lora0\", \"lora1\"],\n",
|
||||
"}\n",
|
||||
"response = requests.post(\n",
|
||||
" url + \"/generate\",\n",
|
||||
" json=json_data,\n",
|
||||
")\n",
|
||||
"print(f\"Output from lora0: \\n{response.json()[0]['text']}\\n\")\n",
|
||||
"print(f\"Output from lora1 (updated): \\n{response.json()[1]['text']}\\n\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Unload lora0 and replace it with a different adapter:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"response = requests.post(\n",
|
||||
" url + \"/unload_lora_adapter\",\n",
|
||||
" json={\n",
|
||||
" \"lora_name\": \"lora0\",\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"response = requests.post(\n",
|
||||
" url + \"/load_lora_adapter\",\n",
|
||||
" json={\n",
|
||||
" \"lora_name\": \"lora0\",\n",
|
||||
" \"lora_path\": lora0_new,\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"if response.status_code == 200:\n",
|
||||
" print(\"LoRA adapter loaded successfully.\", response.json())\n",
|
||||
"else:\n",
|
||||
" print(\"Failed to load LoRA adapter.\", response.json())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Check output again:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"url = f\"http://127.0.0.1:{port}\"\n",
|
||||
"json_data = {\n",
|
||||
" \"text\": [\n",
|
||||
" \"List 3 countries and their capitals.\",\n",
|
||||
" \"List 3 countries and their capitals.\",\n",
|
||||
" ],\n",
|
||||
" \"sampling_params\": {\"max_new_tokens\": 32, \"temperature\": 0},\n",
|
||||
" # The first input uses lora0, and the second input uses lora1\n",
|
||||
" \"lora_path\": [\"lora0\", \"lora1\"],\n",
|
||||
"}\n",
|
||||
"response = requests.post(\n",
|
||||
" url + \"/generate\",\n",
|
||||
" json=json_data,\n",
|
||||
")\n",
|
||||
"print(f\"Output from lora0: \\n{response.json()[0]['text']}\\n\")\n",
|
||||
"print(f\"Output from lora1 (updated): \\n{response.json()[1]['text']}\\n\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(server_process)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### OpenAI-compatible API usage\n",
|
||||
"\n",
|
||||
"You can use LoRA adapters via the OpenAI-compatible APIs by specifying the adapter in the `model` field using the `base-model:adapter-name` syntax (for example, `qwen/qwen2.5-0.5b-instruct:adapter_a`). For more details and examples, see the “Using LoRA Adapters” section in the OpenAI API documentation: [openai_api_completions](../basic_usage/openai_api_completions).\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### LoRA GPU Pinning"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Another advanced option is to specify adapters as `pinned` during loading. When an adapter is pinned, it is permanently assigned to one of the available GPU pool slots (as configured by `--max-loras-per-batch`) and will not be evicted from GPU memory during runtime. Instead, it remains resident until it is explicitly unloaded.\n",
|
||||
"\n",
|
||||
"This can improve performance in scenarios where the same adapter is frequently used across requests, by avoiding repeated memory transfers and reinitialization overhead. However, since GPU pool slots are limited, pinning adapters reduces the flexibility of the system to dynamically load other adapters on demand. If too many adapters are pinned, it may lead to degraded performance, or in the most extreme case (`Number of pinned adapters == max-loras-per-batch`), halt all unpinned requests. Therefore, currently SGLang limits maximal number of pinned adapters to `max-loras-per-batch - 1` to prevent unexpected starvations. \n",
|
||||
"\n",
|
||||
"In the example below, we start a server with `lora1` loaded as pinned, `lora2` and `lora3` loaded as regular (unpinned) adapters. Please note that, we intentionally specify `lora2` and `lora3` in two different formats to demonstrate that both are supported."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"server_process, port = launch_server_cmd(\"\"\"\n",
|
||||
" python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
|
||||
" --enable-lora \\\n",
|
||||
" --cuda-graph-max-bs 8 \\\n",
|
||||
" --max-loras-per-batch 3 \\\n",
|
||||
" --max-lora-rank 256 \\\n",
|
||||
" --lora-target-modules all \\\n",
|
||||
" --lora-paths \\\n",
|
||||
" {\"lora_name\":\"lora0\",\"lora_path\":\"Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16\",\"pinned\":true} \\\n",
|
||||
" {\"lora_name\":\"lora1\",\"lora_path\":\"algoprog/fact-generation-llama-3.1-8b-instruct-lora\"} \\\n",
|
||||
" lora2=philschmid/code-llama-3-1-8b-text-to-sql-lora\n",
|
||||
" --log-level warning\n",
|
||||
" \"\"\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"url = f\"http://127.0.0.1:{port}\"\n",
|
||||
"wait_for_server(url)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"You can also specify adapter as pinned during dynamic adapter loading. In the example below, we reload `lora2` as pinned adapter:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"response = requests.post(\n",
|
||||
" url + \"/unload_lora_adapter\",\n",
|
||||
" json={\n",
|
||||
" \"lora_name\": \"lora1\",\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"response = requests.post(\n",
|
||||
" url + \"/load_lora_adapter\",\n",
|
||||
" json={\n",
|
||||
" \"lora_name\": \"lora1\",\n",
|
||||
" \"lora_path\": \"algoprog/fact-generation-llama-3.1-8b-instruct-lora\",\n",
|
||||
" \"pinned\": True, # Pin the adapter to GPU\n",
|
||||
" },\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Verify that the results are expected:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"url = f\"http://127.0.0.1:{port}\"\n",
|
||||
"json_data = {\n",
|
||||
" \"text\": [\n",
|
||||
" \"List 3 countries and their capitals.\",\n",
|
||||
" \"List 3 countries and their capitals.\",\n",
|
||||
" \"List 3 countries and their capitals.\",\n",
|
||||
" ],\n",
|
||||
" \"sampling_params\": {\"max_new_tokens\": 32, \"temperature\": 0},\n",
|
||||
" # The first input uses lora0, and the second input uses lora1\n",
|
||||
" \"lora_path\": [\"lora0\", \"lora1\", \"lora2\"],\n",
|
||||
"}\n",
|
||||
"response = requests.post(\n",
|
||||
" url + \"/generate\",\n",
|
||||
" json=json_data,\n",
|
||||
")\n",
|
||||
"print(f\"Output from lora0 (pinned): \\n{response.json()[0]['text']}\\n\")\n",
|
||||
"print(f\"Output from lora1 (pinned): \\n{response.json()[1]['text']}\\n\")\n",
|
||||
"print(f\"Output from lora2 (not pinned): \\n{response.json()[2]['text']}\\n\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(server_process)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Choosing LoRA Backend\n",
|
||||
"\n",
|
||||
"SGLang supports two LoRA backends that you can choose from using the `--lora-backend` argument:\n",
|
||||
"\n",
|
||||
"- `triton`: Basic Triton-based backend.\n",
|
||||
"- `csgmv`: Default chunked SGMV backend optimized for high concurrency scenarios.\n",
|
||||
"\n",
|
||||
"The `csgmv` backend was recently introduced to improve performance especially at high-concurrency scenarios. Our benchmark shows that it achieves 20% to 80% latency improvements over the basic triton backend."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"server_process, port = launch_server_cmd(\"\"\"\n",
|
||||
" python3 -m sglang.launch_server \\\n",
|
||||
" --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
|
||||
" --enable-lora \\\n",
|
||||
" --lora-backend csgmv \\\n",
|
||||
" --max-loras-per-batch 16 \\\n",
|
||||
" --lora-paths lora1=path/to/lora1 lora2=path/to/lora2\n",
|
||||
" \"\"\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(server_process)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## LoRA Overlap Loading"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"By using the `--enable-lora-overlap-loading` server argument, the SGLang engine is able to overlap the loading of LoRA weights with prefill and decode compute, essentially hiding the data movement for LoRA weights behind GPU computation. Our benchmarks show that under adversarial conditions, enabling this feature can result in a ~35% reduction in median TTFT - (see the [LoRA overlap loading PR](https://github.com/sgl-project/sglang/pull/15512) for detailed benchmarks)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"lora0 = \"Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16\"\n",
|
||||
"lora1 = \"algoprog/fact-generation-llama-3.1-8b-instruct-lora\"\n",
|
||||
"lora2 = \"philschmid/code-llama-3-1-8b-text-to-sql-lora\"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"server_process, port = launch_server_cmd(\"\"\"\n",
|
||||
" python3 -m sglang.launch_server \\\n",
|
||||
" --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
|
||||
" --enable-lora \\\n",
|
||||
" --enable-lora-overlap-loading \\\n",
|
||||
" --lora-paths lora0=Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16 \\\n",
|
||||
" lora1=algoprog/fact-generation-llama-3.1-8b-instruct-lora \\\n",
|
||||
" lora2=philschmid/code-llama-3-1-8b-text-to-sql-lora \\\n",
|
||||
" --max-lora-rank 256 \\\n",
|
||||
" --max-loras-per-batch 2 \\\n",
|
||||
" --max-loaded-loras 4\n",
|
||||
" \"\"\")\n",
|
||||
"\n",
|
||||
"url = f\"http://127.0.0.1:{port}\"\n",
|
||||
"wait_for_server(url)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"json_data = {\n",
|
||||
" \"text\": [\n",
|
||||
" \"Write a very long fairy-tale.\",\n",
|
||||
" \"List 3 countries and their capitals.\",\n",
|
||||
" \"List 3 countries and their capitals.\",\n",
|
||||
" ],\n",
|
||||
" \"sampling_params\": [\n",
|
||||
" {\"max_new_tokens\": 1024, \"temperature\": 0},\n",
|
||||
" {\"max_new_tokens\": 64, \"temperature\": 0},\n",
|
||||
" {\"max_new_tokens\": 64, \"temperature\": 0},\n",
|
||||
" ],\n",
|
||||
" \"lora_path\": [\"lora0\", \"lora1\", \"lora2\"],\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"# lora0 and lora1 will be loaded into the memory pool first, and because max_loras_per_batch = 2, lora2's request will remain in the queue.\n",
|
||||
"# lora1's request will likely finish first, and once it does, lora2 will be loaded. With --enable-lora-overlap-loading, this loading will\n",
|
||||
"# occur asynchronously and thus decoding for lora0's request won't be blocked.\n",
|
||||
"response = requests.post(\n",
|
||||
" url + \"/generate\",\n",
|
||||
" json=json_data,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"for i in range(3):\n",
|
||||
" print(f\"Output from lora{i}: \\n{response.json()[i]['text']}\\n\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(server_process)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Limitations of LoRA Overlap Loading"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"However, LoRA overlap loading is not free and comes with two important caveats:\n",
|
||||
"\n",
|
||||
"1. **Pinned CPU memory requirement**:\n",
|
||||
" Asynchronous H2D memory copies require LoRA weights to be pinned in CPU memory, which is a finite system resource. To mitigate excessive pinned-memory usage, SGLang currently restricts `max_loaded_loras` to be at most 2× `max_loras_per_batch` when LoRA overlap loading is enabled.\n",
|
||||
"\n",
|
||||
"2. **Reduced multi-adapter prefill batching**:\n",
|
||||
" With overlap loading, adapters become available on the GPU at different times because each adapter is loaded asynchronously. This can reduce the scheduler’s ability to form multi-adapter prefill batches, since only requests whose adapters are currently loaded can be grouped together. As a result, requests for different adapters will be scheduled in separate (or smaller) prefill batches, which can increase TTFT when adapter load time is small compared to prefill compute time. This is why LoRA overlap loading is disabled by default: it should only be enabled when users have determined that LoRA weight loading is a bottleneck (EG high adapter churn, heavy adapter weights, or PCIe-bottlenecked workloads).\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Example When Overlap Loading Results in Higher Latency"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"For instance, suppose we have four LoRA adapters: `lora0`, `lora1`, `lora2`, and `lora3`. Loading any adapter takes 2ms, while the prefill step for requests for that adapter takes 20ms.\n",
|
||||
"\n",
|
||||
"1. **Baseline**:\n",
|
||||
" The engine loads all four adapters synchronously, then runs one combined prefill batch, giving us a total time of ≈ `2 * 4 + 20 = 28ms`\n",
|
||||
"\n",
|
||||
"2. **With LoRA overlap loading enabled**:\n",
|
||||
" The engine begins loading `lora0` and, once it is ready, schedules a prefill batch containing only `lora0` while `lora1` loads in the background. Then it schedules `lora1`’s prefill while `lora2` loads, and so on. In the worst case where prefill cannot be batched across adapters, total time is ≈ `2 + 4 * 20 = 82ms`\n",
|
||||
"\n",
|
||||
"In this scenario, overlap loading reduces adapter-load overhead, but the loss of multi-adapter prefill batching dominates and leads to higher TTFT."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Future Works\n",
|
||||
"\n",
|
||||
"The development roadmap for LoRA-related features can be found in this [issue](https://github.com/sgl-project/sglang/issues/2929). Other features, including Embedding Layer, Unified Paging, Cutlass backend are still under development."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,507 @@
|
||||
---
|
||||
title: "LoRA Serving"
|
||||
metatags:
|
||||
description: "SGLang multi-LoRA serving: S-LoRA and Punica techniques, dynamic adapter loading, GPU pinning, overlap loading, Triton and CSGMV backends."
|
||||
---
|
||||
SGLang enables the use of [LoRA adapters](https://arxiv.org/abs/2106.09685) with a base model. By incorporating techniques from [S-LoRA](https://arxiv.org/pdf/2311.03285) and [Punica](https://arxiv.org/pdf/2310.18547), SGLang can efficiently support multiple LoRA adapters for different sequences within a single batch of inputs.
|
||||
|
||||
|
||||
## Arguments for LoRA Serving
|
||||
|
||||
|
||||
The following server arguments are relevant for multi-LoRA serving:
|
||||
|
||||
* `enable_lora`: Enable LoRA support for the model. This argument is automatically set to True if `--lora-paths` is provided for backward compatibility.
|
||||
|
||||
* `enable_lora_overlap_loading`: Enable asynchronous LoRA weight loading in order to overlap H2D transfers with GPU compute. This should be enabled if you find that your LoRA workloads are bottlenecked by adapter weight loading, for example when frequently loading large LoRA adapters.
|
||||
|
||||
* `lora_paths`: The list of LoRA adapters to load. Each adapter must be specified in one of the following formats: <PATH> | <NAME>=<PATH> | JSON with schema {"lora_name":str,"lora_path":str,"pinned":bool}.
|
||||
|
||||
* `max_loras_per_batch`: Maximum number of adaptors used by each batch. This argument can affect the amount of GPU memory reserved for multi-LoRA serving, so it should be set to a smaller value when memory is scarce. Defaults to be 8.
|
||||
|
||||
* `max_loaded_loras`: If specified, it limits the maximum number of LoRA adapters loaded in CPU memory at a time. The value must be greater than or equal to `max-loras-per-batch`.
|
||||
|
||||
* `lora_eviction_policy`: LoRA adapter eviction policy when GPU memory pool is full. `lru`: Least Recently Used (default, better cache efficiency). `fifo`: First-In-First-Out.
|
||||
|
||||
* `lora_backend`: The backend of running GEMM kernels for Lora modules. Currently we support Triton LoRA backend (`triton`) and Chunked SGMV backend (`csgmv`). In the future, faster backend built upon Cutlass or Cuda kernels will be added.
|
||||
|
||||
* `max_lora_rank`: The maximum LoRA rank that should be supported. If not specified, it will be automatically inferred from the adapters provided in `--lora-paths`. This argument is needed when you expect to dynamically load adapters of larger LoRA rank after server startup.
|
||||
|
||||
* `lora_target_modules`: The union set of all target modules where LoRA should be applied (e.g., `q_proj`, `k_proj`, `gate_proj`). If not specified, it will be automatically inferred from the adapters provided in `--lora-paths`. This argument is needed when you expect to dynamically load adapters of different target modules after server startup. You can also set it to `all` to enable LoRA for all supported modules. However, enabling LoRA on additional modules introduces a minor performance overhead. If your application is performance-sensitive, we recommend only specifying the modules for which you plan to load adapters.
|
||||
|
||||
* `--max-lora-chunk-size`: Maximum chunk size for the ChunkedSGMV LoRA backend. Only used when --lora-backend is 'csgmv'. Choosing a larger value might improve performance. Please tune this value based on your hardware and workload as needed. Defaults to 16.
|
||||
|
||||
* `tp_size`: LoRA serving along with Tensor Parallelism is supported by SGLang. `tp_size` controls the number of GPUs for tensor parallelism. More details on the tensor sharding strategy can be found in [S-Lora](https://arxiv.org/pdf/2311.03285) paper.
|
||||
|
||||
From client side, the user needs to provide a list of strings as input batch, and a list of adaptor names that each input sequence corresponds to.
|
||||
|
||||
|
||||
## Usage
|
||||
|
||||
### Serving Single Adaptor
|
||||
|
||||
|
||||
**Note:** SGLang supports LoRA adapters through two APIs:
|
||||
|
||||
1. **OpenAI-Compatible API** (`/v1/chat/completions`, `/v1/completions`): Use the `model:adapter-name` syntax. See [OpenAI API with LoRA](../basic_usage/openai_api_completions.ipynb#Using-LoRA-Adapters) for examples.
|
||||
|
||||
2. **Native API** (`/generate`): Pass `lora_path` in the request body (shown below).
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import json
|
||||
import requests
|
||||
|
||||
from sglang.test.doc_patch import launch_server_cmd
|
||||
from sglang.utils import wait_for_server, terminate_process
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
server_process, port = launch_server_cmd(
|
||||
# Here we set max-loras-per-batch to 2: one slot for adaptor and another one for base model
|
||||
"""
|
||||
python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--enable-lora \
|
||||
--lora-paths lora0=algoprog/fact-generation-llama-3.1-8b-instruct-lora \
|
||||
--max-loras-per-batch 2 \
|
||||
--log-level warning \
|
||||
"""
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
url = f"http://127.0.0.1:{port}"
|
||||
json_data = {
|
||||
"text": [
|
||||
"List 3 countries and their capitals.",
|
||||
"List 3 countries and their capitals.",
|
||||
],
|
||||
"sampling_params": {"max_new_tokens": 32, "temperature": 0},
|
||||
# The first input uses lora0, and the second input uses the base model
|
||||
"lora_path": ["lora0", None],
|
||||
}
|
||||
response = requests.post(
|
||||
url + "/generate",
|
||||
json=json_data,
|
||||
)
|
||||
print(f"Output 0: {response.json()[0]['text']}")
|
||||
print(f"Output 1: {response.json()[1]['text']}")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
terminate_process(server_process)
|
||||
```
|
||||
|
||||
### Serving Multiple Adaptors
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
server_process, port = launch_server_cmd(
|
||||
"""
|
||||
python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--enable-lora \
|
||||
--lora-paths lora0=algoprog/fact-generation-llama-3.1-8b-instruct-lora \
|
||||
lora1=Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16 \
|
||||
--max-loras-per-batch 2 \
|
||||
--log-level warning \
|
||||
"""
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
url = f"http://127.0.0.1:{port}"
|
||||
json_data = {
|
||||
"text": [
|
||||
"List 3 countries and their capitals.",
|
||||
"List 3 countries and their capitals.",
|
||||
],
|
||||
"sampling_params": {"max_new_tokens": 32, "temperature": 0},
|
||||
# The first input uses lora0, and the second input uses lora1
|
||||
"lora_path": ["lora0", "lora1"],
|
||||
}
|
||||
response = requests.post(
|
||||
url + "/generate",
|
||||
json=json_data,
|
||||
)
|
||||
print(f"Output 0: {response.json()[0]['text']}")
|
||||
print(f"Output 1: {response.json()[1]['text']}")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
terminate_process(server_process)
|
||||
```
|
||||
|
||||
### Dynamic LoRA loading
|
||||
|
||||
|
||||
Instead of specifying all adapters during server startup via `--lora-paths`. You can also load & unload LoRA adapters dynamically via the `/load_lora_adapter` and `/unload_lora_adapter` API.
|
||||
|
||||
When using dynamic LoRA loading, it's recommended to explicitly specify both `--max-lora-rank` and `--lora-target-modules` at startup. For backward compatibility, SGLang will infer these values from `--lora-paths` if they are not explicitly provided. However, in that case, you would have to ensure that all dynamically loaded adapters share the same shape (rank and target modules) as those in the initial `--lora-paths` or are strictly "smaller".
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
lora0 = "Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16" # rank - 4, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj
|
||||
lora1 = "algoprog/fact-generation-llama-3.1-8b-instruct-lora" # rank - 64, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
|
||||
lora0_new = "philschmid/code-llama-3-1-8b-text-to-sql-lora" # rank - 256, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
|
||||
|
||||
|
||||
# The `--target-lora-modules` param below is technically not needed, as the server will infer it from lora0 which already has all the target modules specified.
|
||||
# We are adding it here just to demonstrate usage.
|
||||
server_process, port = launch_server_cmd(
|
||||
"""
|
||||
python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--enable-lora \
|
||||
--cuda-graph-max-bs 2 \
|
||||
--max-loras-per-batch 2 \
|
||||
--max-lora-rank 256
|
||||
--lora-target-modules all
|
||||
--log-level warning
|
||||
"""
|
||||
)
|
||||
|
||||
url = f"http://127.0.0.1:{port}"
|
||||
wait_for_server(url)
|
||||
```
|
||||
|
||||
Load adapter lora0
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
response = requests.post(
|
||||
url + "/load_lora_adapter",
|
||||
json={
|
||||
"lora_name": "lora0",
|
||||
"lora_path": lora0,
|
||||
},
|
||||
)
|
||||
|
||||
if response.status_code == 200:
|
||||
print("LoRA adapter loaded successfully.", response.json())
|
||||
else:
|
||||
print("Failed to load LoRA adapter.", response.json())
|
||||
```
|
||||
|
||||
Load adapter lora1:
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
response = requests.post(
|
||||
url + "/load_lora_adapter",
|
||||
json={
|
||||
"lora_name": "lora1",
|
||||
"lora_path": lora1,
|
||||
},
|
||||
)
|
||||
|
||||
if response.status_code == 200:
|
||||
print("LoRA adapter loaded successfully.", response.json())
|
||||
else:
|
||||
print("Failed to load LoRA adapter.", response.json())
|
||||
```
|
||||
|
||||
Check inference output:
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
url = f"http://127.0.0.1:{port}"
|
||||
json_data = {
|
||||
"text": [
|
||||
"List 3 countries and their capitals.",
|
||||
"List 3 countries and their capitals.",
|
||||
],
|
||||
"sampling_params": {"max_new_tokens": 32, "temperature": 0},
|
||||
# The first input uses lora0, and the second input uses lora1
|
||||
"lora_path": ["lora0", "lora1"],
|
||||
}
|
||||
response = requests.post(
|
||||
url + "/generate",
|
||||
json=json_data,
|
||||
)
|
||||
print(f"Output from lora0: \n{response.json()[0]['text']}\n")
|
||||
print(f"Output from lora1 (updated): \n{response.json()[1]['text']}\n")
|
||||
```
|
||||
|
||||
Unload lora0 and replace it with a different adapter:
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
response = requests.post(
|
||||
url + "/unload_lora_adapter",
|
||||
json={
|
||||
"lora_name": "lora0",
|
||||
},
|
||||
)
|
||||
|
||||
response = requests.post(
|
||||
url + "/load_lora_adapter",
|
||||
json={
|
||||
"lora_name": "lora0",
|
||||
"lora_path": lora0_new,
|
||||
},
|
||||
)
|
||||
|
||||
if response.status_code == 200:
|
||||
print("LoRA adapter loaded successfully.", response.json())
|
||||
else:
|
||||
print("Failed to load LoRA adapter.", response.json())
|
||||
```
|
||||
|
||||
Check output again:
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
url = f"http://127.0.0.1:{port}"
|
||||
json_data = {
|
||||
"text": [
|
||||
"List 3 countries and their capitals.",
|
||||
"List 3 countries and their capitals.",
|
||||
],
|
||||
"sampling_params": {"max_new_tokens": 32, "temperature": 0},
|
||||
# The first input uses lora0, and the second input uses lora1
|
||||
"lora_path": ["lora0", "lora1"],
|
||||
}
|
||||
response = requests.post(
|
||||
url + "/generate",
|
||||
json=json_data,
|
||||
)
|
||||
print(f"Output from lora0: \n{response.json()[0]['text']}\n")
|
||||
print(f"Output from lora1 (updated): \n{response.json()[1]['text']}\n")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
terminate_process(server_process)
|
||||
```
|
||||
|
||||
### OpenAI-compatible API usage
|
||||
|
||||
You can use LoRA adapters via the OpenAI-compatible APIs by specifying the adapter in the `model` field using the `base-model:adapter-name` syntax (for example, `qwen/qwen2.5-0.5b-instruct:adapter_a`). For more details and examples, see the “Using LoRA Adapters” section in the OpenAI API documentation: [openai_api_completions](../basic_usage/openai_api_completions).
|
||||
|
||||
|
||||
|
||||
### LoRA GPU Pinning
|
||||
|
||||
|
||||
Another advanced option is to specify adapters as `pinned` during loading. When an adapter is pinned, it is permanently assigned to one of the available GPU pool slots (as configured by `--max-loras-per-batch`) and will not be evicted from GPU memory during runtime. Instead, it remains resident until it is explicitly unloaded.
|
||||
|
||||
This can improve performance in scenarios where the same adapter is frequently used across requests, by avoiding repeated memory transfers and reinitialization overhead. However, since GPU pool slots are limited, pinning adapters reduces the flexibility of the system to dynamically load other adapters on demand. If too many adapters are pinned, it may lead to degraded performance, or in the most extreme case (`Number of pinned adapters == max-loras-per-batch`), halt all unpinned requests. Therefore, currently SGLang limits maximal number of pinned adapters to `max-loras-per-batch - 1` to prevent unexpected starvations.
|
||||
|
||||
In the example below, we start a server with `lora1` loaded as pinned, `lora2` and `lora3` loaded as regular (unpinned) adapters. Please note that, we intentionally specify `lora2` and `lora3` in two different formats to demonstrate that both are supported.
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
server_process, port = launch_server_cmd(
|
||||
"""
|
||||
python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--enable-lora \
|
||||
--cuda-graph-max-bs 8 \
|
||||
--max-loras-per-batch 3 \
|
||||
--max-lora-rank 256 \
|
||||
--lora-target-modules all \
|
||||
--lora-paths \
|
||||
{"lora_name":"lora0","lora_path":"Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16","pinned":true} \
|
||||
{"lora_name":"lora1","lora_path":"algoprog/fact-generation-llama-3.1-8b-instruct-lora"} \
|
||||
lora2=philschmid/code-llama-3-1-8b-text-to-sql-lora
|
||||
--log-level warning
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
url = f"http://127.0.0.1:{port}"
|
||||
wait_for_server(url)
|
||||
```
|
||||
|
||||
You can also specify adapter as pinned during dynamic adapter loading. In the example below, we reload `lora2` as pinned adapter:
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
response = requests.post(
|
||||
url + "/unload_lora_adapter",
|
||||
json={
|
||||
"lora_name": "lora1",
|
||||
},
|
||||
)
|
||||
|
||||
response = requests.post(
|
||||
url + "/load_lora_adapter",
|
||||
json={
|
||||
"lora_name": "lora1",
|
||||
"lora_path": "algoprog/fact-generation-llama-3.1-8b-instruct-lora",
|
||||
"pinned": True, # Pin the adapter to GPU
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
Verify that the results are expected:
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
url = f"http://127.0.0.1:{port}"
|
||||
json_data = {
|
||||
"text": [
|
||||
"List 3 countries and their capitals.",
|
||||
"List 3 countries and their capitals.",
|
||||
"List 3 countries and their capitals.",
|
||||
],
|
||||
"sampling_params": {"max_new_tokens": 32, "temperature": 0},
|
||||
# The first input uses lora0, and the second input uses lora1
|
||||
"lora_path": ["lora0", "lora1", "lora2"],
|
||||
}
|
||||
response = requests.post(
|
||||
url + "/generate",
|
||||
json=json_data,
|
||||
)
|
||||
print(f"Output from lora0 (pinned): \n{response.json()[0]['text']}\n")
|
||||
print(f"Output from lora1 (pinned): \n{response.json()[1]['text']}\n")
|
||||
print(f"Output from lora2 (not pinned): \n{response.json()[2]['text']}\n")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
terminate_process(server_process)
|
||||
```
|
||||
|
||||
## Choosing LoRA Backend
|
||||
|
||||
SGLang supports two LoRA backends that you can choose from using the `--lora-backend` argument:
|
||||
|
||||
- `triton`: Basic Triton-based backend.
|
||||
- `csgmv`: Default chunked SGMV backend optimized for high concurrency scenarios.
|
||||
|
||||
The `csgmv` backend was recently introduced to improve performance especially at high-concurrency scenarios. Our benchmark shows that it achieves 20% to 80% latency improvements over the basic triton backend.
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
server_process, port = launch_server_cmd(
|
||||
"""
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--enable-lora \
|
||||
--lora-backend csgmv \
|
||||
--max-loras-per-batch 16 \
|
||||
--lora-paths lora1=path/to/lora1 lora2=path/to/lora2
|
||||
"""
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
terminate_process(server_process)
|
||||
```
|
||||
|
||||
## LoRA Overlap Loading
|
||||
|
||||
|
||||
By using the `--enable-lora-overlap-loading` server argument, the SGLang engine is able to overlap the loading of LoRA weights with prefill and decode compute, essentially hiding the data movement for LoRA weights behind GPU computation. Our benchmarks show that under adversarial conditions, enabling this feature can result in a ~35% reduction in median TTFT - (see the [LoRA overlap loading PR](https://github.com/sgl-project/sglang/pull/15512) for detailed benchmarks).
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
lora0 = "Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16"
|
||||
lora1 = "algoprog/fact-generation-llama-3.1-8b-instruct-lora"
|
||||
lora2 = "philschmid/code-llama-3-1-8b-text-to-sql-lora"
|
||||
|
||||
|
||||
server_process, port = launch_server_cmd(
|
||||
"""
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--enable-lora \
|
||||
--enable-lora-overlap-loading \
|
||||
--lora-paths lora0=Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16 \
|
||||
lora1=algoprog/fact-generation-llama-3.1-8b-instruct-lora \
|
||||
lora2=philschmid/code-llama-3-1-8b-text-to-sql-lora \
|
||||
--max-lora-rank 256 \
|
||||
--max-loras-per-batch 2 \
|
||||
--max-loaded-loras 4
|
||||
"""
|
||||
)
|
||||
|
||||
url = f"http://127.0.0.1:{port}"
|
||||
wait_for_server(url)
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
json_data = {
|
||||
"text": [
|
||||
"Write a very long fairy-tale.",
|
||||
"List 3 countries and their capitals.",
|
||||
"List 3 countries and their capitals.",
|
||||
],
|
||||
"sampling_params": [
|
||||
{"max_new_tokens": 1024, "temperature": 0},
|
||||
{"max_new_tokens": 64, "temperature": 0},
|
||||
{"max_new_tokens": 64, "temperature": 0},
|
||||
],
|
||||
"lora_path": ["lora0", "lora1", "lora2"],
|
||||
}
|
||||
|
||||
# lora0 and lora1 will be loaded into the memory pool first, and because max_loras_per_batch = 2, lora2's request will remain in the queue.
|
||||
# lora1's request will likely finish first, and once it does, lora2 will be loaded. With --enable-lora-overlap-loading, this loading will
|
||||
# occur asynchronously and thus decoding for lora0's request won't be blocked.
|
||||
response = requests.post(
|
||||
url + "/generate",
|
||||
json=json_data,
|
||||
)
|
||||
|
||||
for i in range(3):
|
||||
print(f"Output from lora{i}: \n{response.json()[i]['text']}\n")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
terminate_process(server_process)
|
||||
```
|
||||
|
||||
#### Limitations of LoRA Overlap Loading
|
||||
|
||||
|
||||
However, LoRA overlap loading is not free and comes with two important caveats:
|
||||
|
||||
1. **Pinned CPU memory requirement**:
|
||||
Asynchronous H2D memory copies require LoRA weights to be pinned in CPU memory, which is a finite system resource. To mitigate excessive pinned-memory usage, SGLang currently restricts `max_loaded_loras` to be at most 2× `max_loras_per_batch` when LoRA overlap loading is enabled.
|
||||
|
||||
2. **Reduced multi-adapter prefill batching**:
|
||||
With overlap loading, adapters become available on the GPU at different times because each adapter is loaded asynchronously. This can reduce the scheduler’s ability to form multi-adapter prefill batches, since only requests whose adapters are currently loaded can be grouped together. As a result, requests for different adapters will be scheduled in separate (or smaller) prefill batches, which can increase TTFT when adapter load time is small compared to prefill compute time. This is why LoRA overlap loading is disabled by default: it should only be enabled when users have determined that LoRA weight loading is a bottleneck (EG high adapter churn, heavy adapter weights, or PCIe-bottlenecked workloads).
|
||||
|
||||
|
||||
|
||||
#### Example When Overlap Loading Results in Higher Latency
|
||||
|
||||
|
||||
For instance, suppose we have four LoRA adapters: `lora0`, `lora1`, `lora2`, and `lora3`. Loading any adapter takes 2ms, while the prefill step for requests for that adapter takes 20ms.
|
||||
|
||||
1. **Baseline**:
|
||||
The engine loads all four adapters synchronously, then runs one combined prefill batch, giving us a total time of ≈ `2 * 4 + 20 = 28ms`
|
||||
|
||||
2. **With LoRA overlap loading enabled**:
|
||||
The engine begins loading `lora0` and, once it is ready, schedules a prefill batch containing only `lora0` while `lora1` loads in the background. Then it schedules `lora1`’s prefill while `lora2` loads, and so on. In the worst case where prefill cannot be batched across adapters, total time is ≈ `2 + 4 * 20 = 82ms`
|
||||
|
||||
In this scenario, overlap loading reduces adapter-load overhead, but the loss of multi-adapter prefill batching dominates and leads to higher TTFT.
|
||||
|
||||
|
||||
## Future Works
|
||||
|
||||
The development roadmap for LoRA-related features can be found in this [issue](https://github.com/sgl-project/sglang/issues/2929). Other features, including Embedding Layer, Unified Paging, Cutlass backend are still under development.
|
||||
@@ -0,0 +1,142 @@
|
||||
---
|
||||
title: "Loading Models from Object Storage"
|
||||
metatags:
|
||||
description: "Load SGLang models directly from S3, Google Cloud Storage, Azure Blob, and S3-compatible object storage with runai_streamer."
|
||||
---
|
||||
|
||||
SGLang supports direct loading of models from object storage (S3 and Google Cloud Storage) without requiring a full local download. This feature uses the `runai_streamer` load format to stream model weights directly from cloud storage, significantly reducing startup time and local storage requirements.
|
||||
|
||||
## Overview
|
||||
|
||||
When loading models from object storage, SGLang uses a two-phase approach:
|
||||
|
||||
1. **Metadata Download** (once, before process launch): Configuration files and tokenizer files are downloaded to a local cache
|
||||
2. **Weight Streaming** (lazy, during model loading): Model weights are streamed directly from object storage as needed
|
||||
|
||||
## Supported Storage Backends
|
||||
|
||||
1. **Amazon S3**: `s3://bucket-name/path/to/model/`
|
||||
2. **Google Cloud Storage**: `gs://bucket-name/path/to/model/`
|
||||
3. **Azure Blob**: `az://some-azure-container/path/`
|
||||
4. **S3 compatible**: `s3://bucket-name/path/to/model/`
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Simply provide an object storage URI as the model path:
|
||||
|
||||
```bash
|
||||
# S3
|
||||
python -m sglang.launch_server \
|
||||
--model-path s3://my-bucket/models/llama-3-8b/ \
|
||||
--load-format runai_streamer
|
||||
|
||||
# Google Cloud Storage
|
||||
python -m sglang.launch_server \
|
||||
--model-path gs://my-bucket/models/llama-3-8b/ \
|
||||
--load-format runai_streamer
|
||||
```
|
||||
|
||||
**Note**: The `--load-format runai_streamer` is automatically detected when using object storage URIs, so you can omit it:
|
||||
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model-path s3://my-bucket/models/llama-3-8b/
|
||||
```
|
||||
|
||||
### With Tensor Parallelism
|
||||
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model-path gs://my-bucket/models/llama-70b/ \
|
||||
--tp 4 \
|
||||
--model-loader-extra-config '{"distributed": true}'
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Load Format
|
||||
|
||||
The `runai_streamer` load format is specifically designed for object storage, ssd and shared file systems
|
||||
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model-path s3://bucket/model/ \
|
||||
--load-format runai_streamer
|
||||
```
|
||||
|
||||
### Extended Configuration Parameters
|
||||
|
||||
Use `--model-loader-extra-config` to pass additional configuration as a JSON string:
|
||||
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model-path s3://bucket/model/ \
|
||||
--model-loader-extra-config '{
|
||||
"distributed": true,
|
||||
"concurrency": 8,
|
||||
"memory_limit": 2147483648
|
||||
}'
|
||||
```
|
||||
|
||||
#### Available Parameters
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "22%"}} />
|
||||
<col style={{width: "16%"}} />
|
||||
<col style={{width: "44%"}} />
|
||||
<col style={{width: "18%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Default</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>distributed</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Enable distributed streaming for multi-GPU setups. Automatically set to <code>true</code> for object storage paths and cuda alike devices.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Auto-detected</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>concurrency</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>int</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Number of concurrent download streams. Higher values can improve throughput for large models.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>4</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>memory_limit</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>int</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Memory limit (in bytes) for the streaming buffer.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>System-dependent</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Distributed Streaming
|
||||
|
||||
For multi-GPU setups, enable distributed streaming to parallelize weight loading between the processes:
|
||||
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model-path s3://bucket/model/ \
|
||||
--tp 8 \
|
||||
--model-loader-extra-config '{"distributed": true}'
|
||||
```
|
||||
|
||||
## Limitations
|
||||
|
||||
- **Supported Formats**: Currently only supports `.safetensors` weight format (recommended format)
|
||||
- **Supported Device**: Distributed streaming is supported on cuda alike devices. Otherwise fallback to non distributed streaming
|
||||
|
||||
## See Also
|
||||
|
||||
- [Runai model streamer documentation](https://github.com/run-ai/runai-model-streamer)
|
||||
@@ -0,0 +1,38 @@
|
||||
---
|
||||
title: "Observability"
|
||||
metatags:
|
||||
description: "SGLang observability: Prometheus metrics, request logging, request dump and replay, crash dump debugging."
|
||||
---
|
||||
## Production Metrics
|
||||
SGLang exposes the following metrics via Prometheus. You can enable them by adding `--enable-metrics` when launching the server.
|
||||
You can query them by:
|
||||
```bash Command
|
||||
curl http://localhost:30000/metrics
|
||||
```
|
||||
|
||||
See [Production Metrics](../references/production_metrics) and [Production Request Tracing](../references/production_request_trace) for more details.
|
||||
|
||||
## Logging
|
||||
|
||||
By default, SGLang does not log any request contents. You can log them by using `--log-requests`.
|
||||
You can control the verbosity by using `--log-request-level`.
|
||||
See [Logging](./server_arguments#logging) for more details.
|
||||
|
||||
## Request Dump and Replay
|
||||
|
||||
You can dump all requests and replay them later for benchmarking or other purposes.
|
||||
|
||||
To start dumping, use the following command to send a request to a server:
|
||||
```bash Command
|
||||
python3 -m sglang.srt.managers.configure_logging --url http://localhost:30000 --dump-requests-folder /tmp/sglang_request_dump --dump-requests-threshold 100
|
||||
```
|
||||
The server will dump the requests into a pickle file for every 100 requests.
|
||||
|
||||
To replay the request dump, use `scripts/playground/replay_request_dump.py`.
|
||||
|
||||
## Crash Dump and Replay
|
||||
Sometimes the server might crash, and you may want to debug the cause of the crash.
|
||||
SGLang supports crash dumping, which will dump all requests from the 5 minutes before the crash, allowing you to replay the requests and debug the reason later.
|
||||
|
||||
To enable crash dumping, use `--crash-dump-folder /tmp/crash_dump`.
|
||||
To replay the crash dump, use `scripts/playground/replay_request_dump.py`.
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
title: Advanced Features
|
||||
description: Advanced configuration, optimization, and deployment features for SGLang.
|
||||
---
|
||||
|
||||
- [Server Arguments](./server_arguments)
|
||||
- [Hyperparameter Tuning](./hyperparameter_tuning)
|
||||
- [Attention Backend](./attention_backend)
|
||||
- [Speculative Decoding](./speculative_decoding)
|
||||
- [Structured Outputs](./structured_outputs)
|
||||
- [Quantization](./quantization)
|
||||
- [Expert Parallelism](./expert_parallelism)
|
||||
- [LoRA](./lora)
|
||||
- [PD Disaggregation](./pd_disaggregation)
|
||||
- [Pipeline Parallelism](./pipeline_parallelism)
|
||||
- [HiCache](./hicache_best_practices)
|
||||
- [Observability](./observability)
|
||||
- [And more…](./server_arguments)
|
||||
@@ -0,0 +1,404 @@
|
||||
---
|
||||
title: "PD Disaggregation"
|
||||
metatags:
|
||||
description: "SGLang PD disaggregation: separate prefill and decode phases, Mooncake and NIXL transfer engines, multi-node DeepSeek deployment."
|
||||
---
|
||||
## Why and What is PD Disaggregation?
|
||||
|
||||
Large Language Model (LLM) inference comprises two distinct phases: **Prefill** and **Decode**. The Prefill phase is computation-intensive, processing the entire input sequence, while the Decode phase is memory-intensive, managing the Key-Value (KV) cache for token generation. Traditionally, these phases are handled within a unified engine, where combined scheduling of prefill and decode batches introduces inefficiencies. To address these challenges, we introduce **Prefill and Decoding (PD) Disaggregation** in SGLang.
|
||||
|
||||
### Issues with Unified Scheduling
|
||||
|
||||
The conventional unified engine, which processes prefill and decode batches together, results in two significant problems:
|
||||
|
||||
1. **Prefill Interruption**: Incoming prefill batches frequently interrupt ongoing decode batches, causing substantial delays in token generation.
|
||||
2. **DP Attention Imbalance**: In data-parallel (DP) attention, one DP worker may process a prefill batch while another handles a decode batch simultaneously, leading to increased decode latency.
|
||||
|
||||
PD Disaggregation resolves these by separating the two stages, enabling tailored optimizations for each.
|
||||
|
||||
For the design details, please refer to [link](https://docs.google.com/document/d/1rQXJwKd5b9b1aOzLh98mnyMhBMhlxXA5ATZTHoQrwvc/edit?tab=t.0).
|
||||
|
||||
Currently, we support Mooncake and NIXL as the transfer engine.
|
||||
|
||||
## Profiling in PD Disaggregation Mode
|
||||
|
||||
When you need to profile prefill or decode workers in PD disaggregation mode, please refer to the [Profile In PD Disaggregation Mode](../developer_guide/benchmark_and_profiling#profile-in-pd-disaggregation-mode) section in the Benchmark and Profiling guide. Due to torch profiler limitations, prefill and decode workers must be profiled separately using dedicated command-line options.
|
||||
|
||||
## Router Integration
|
||||
|
||||
For deploying PD disaggregation at scale with load balancing and fault tolerance, SGLang provides a router. The router can distribute requests between prefill and decode instances using various routing policies. For detailed information on setting up routing with PD disaggregation, including configuration options and deployment patterns, see the [SGLang Model Gateway (former Router)](../advanced_features/sgl_model_gateway.md#prefill-decode-disaggregation).
|
||||
|
||||
|
||||
## Mooncake
|
||||
### Requirements
|
||||
|
||||
```bash
|
||||
uv pip install mooncake-transfer-engine
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
### Llama Single Node
|
||||
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
||||
--disaggregation-mode prefill \
|
||||
--port 30000 \
|
||||
--disaggregation-ib-device mlx5_roce0
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
||||
--disaggregation-mode decode \
|
||||
--port 30001 \
|
||||
--base-gpu-id 1 \
|
||||
--disaggregation-ib-device mlx5_roce0
|
||||
python -m sglang_router.launch_router --pd-disaggregation --prefill http://127.0.0.1:30000 --decode http://127.0.0.1:30001 --host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
### DeepSeek Multi-Node
|
||||
|
||||
```bash
|
||||
# prefill 0
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3-0324 \
|
||||
--disaggregation-ib-device ${device_name} \
|
||||
--disaggregation-mode prefill \
|
||||
--host ${local_ip} \
|
||||
--port 30000 \
|
||||
--trust-remote-code \
|
||||
--dist-init-addr ${prefill_master_ip}:5000 \
|
||||
--nnodes 2 \
|
||||
--node-rank 0 \
|
||||
--tp-size 16 \
|
||||
--dp-size 8 \
|
||||
--enable-dp-attention \
|
||||
--moe-a2a-backend deepep \
|
||||
--mem-fraction-static 0.8
|
||||
# prefill 1
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3-0324 \
|
||||
--disaggregation-ib-device ${device_name} \
|
||||
--disaggregation-mode prefill \
|
||||
--host ${local_ip} \
|
||||
--port 30000 \
|
||||
--trust-remote-code \
|
||||
--dist-init-addr ${prefill_master_ip}:5000 \
|
||||
--nnodes 2 \
|
||||
--node-rank 1 \
|
||||
--tp-size 16 \
|
||||
--dp-size 8 \
|
||||
--enable-dp-attention \
|
||||
--moe-a2a-backend deepep \
|
||||
--mem-fraction-static 0.8
|
||||
# decode 0
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3-0324 \
|
||||
--disaggregation-ib-device ${device_name} \
|
||||
--disaggregation-mode decode \
|
||||
--host ${local_ip} \
|
||||
--port 30001 \
|
||||
--trust-remote-code \
|
||||
--dist-init-addr ${decode_master_ip}:5000 \
|
||||
--nnodes 2 \
|
||||
--node-rank 0 \
|
||||
--tp-size 16 \
|
||||
--dp-size 8 \
|
||||
--enable-dp-attention \
|
||||
--moe-a2a-backend deepep \
|
||||
--mem-fraction-static 0.8 \
|
||||
--max-running-requests 128
|
||||
# decode 1
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3-0324 \
|
||||
--disaggregation-ib-device ${device_name} \
|
||||
--disaggregation-mode decode \
|
||||
--host ${local_ip} \
|
||||
--port 30001 \
|
||||
--trust-remote-code \
|
||||
--dist-init-addr ${decode_master_ip}:5000 \
|
||||
--nnodes 2 \
|
||||
--node-rank 1 \
|
||||
--tp-size 16 \
|
||||
--dp-size 8 \
|
||||
--enable-dp-attention \
|
||||
--moe-a2a-backend deepep \
|
||||
--mem-fraction-static 0.8 \
|
||||
--max-running-requests 128
|
||||
```
|
||||
### Advanced Configuration
|
||||
|
||||
PD Disaggregation with Mooncake supports the following environment variables for fine-grained control over system behavior.
|
||||
|
||||
#### NVLink Transport Configuration
|
||||
To enable NVLink transport for KV cache transfers with the mooncake backend (recommended for NVL72 deployments), set the following environment variables. Note that auxiliary data transfer will still use TCP as a temporary workaround.
|
||||
|
||||
```bash
|
||||
export SGLANG_MOONCAKE_CUSTOM_MEM_POOL=True
|
||||
export MC_FORCE_MNNVL=True
|
||||
```
|
||||
|
||||
#### Prefill Server Configuration
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**`SGLANG_DISAGGREGATION_THREAD_POOL_SIZE`**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Controls the total number of worker threads for KVCache transfer operations per TP rank</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>A dynamic value calculated by `int(0.75 * os.cpu_count()) // 8)`, which is limited to be larger than 4 and less than 12 to ensure efficiency and prevent thread race conditions</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**`SGLANG_DISAGGREGATION_QUEUE_SIZE`**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Sets the number of parallel transfer queues. KVCache transfer requests from multiple decode instances will be sharded into these queues so that they can share the threads and the transfer bandwidth at the same time. If it is set to `1`, then we transfer requests one by one according to fcfs strategy</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`4`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**`SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT`**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Timeout (seconds) for receiving destination KV indices during request initialization</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`300`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
If a greater mean TTFT is acceptable, you can `export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600` (10 minutes) to relax the timeout condition.
|
||||
Please be aware that this setting will cause prefill instances to take a longer time to clean up the affected memory resources when a running decode node loses connection.
|
||||
|
||||
#### Decode Server Configuration
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**`SGLANG_DISAGGREGATION_HEARTBEAT_INTERVAL`**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Interval (seconds) between health checks to prefill bootstrap servers</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`5.0`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**`SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE`**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Consecutive heartbeat failures before marking prefill server offline</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`2`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**`SGLANG_DISAGGREGATION_WAITING_TIMEOUT`**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Timeout (seconds) for receiving KV Cache after request initialization</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`300`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
If a greater mean TTFT is acceptable, you can `export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=600` (10 minutes) to relax the timeout condition.
|
||||
|
||||
|
||||
## NIXL
|
||||
### Requirements
|
||||
|
||||
Install via pip.
|
||||
|
||||
```bash
|
||||
pip install nixl
|
||||
```
|
||||
|
||||
Or build from source - may be required if you already have UCX installed.
|
||||
|
||||
```bash
|
||||
git clone https://github.com/ai-dynamo/nixl.git
|
||||
cd nixl
|
||||
pip install . --config-settings=setup-args="-Ducx_path=/path/to/ucx"
|
||||
```
|
||||
|
||||
|
||||
### Usage
|
||||
|
||||
### Llama Single Node
|
||||
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
||||
--disaggregation-mode prefill \
|
||||
--port 30000 \
|
||||
--disaggregation-transfer-backend nixl
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
||||
--disaggregation-mode decode \
|
||||
--port 30001 \
|
||||
--base-gpu-id 1 \
|
||||
--disaggregation-transfer-backend nixl
|
||||
python -m sglang_router.launch_router --pd-disaggregation --prefill http://127.0.0.1:30000 --decode http://127.0.0.1:30001 --host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
### DeepSeek Multi-Node
|
||||
|
||||
```bash
|
||||
# prefill 0
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3-0324 \
|
||||
--disaggregation-transfer-backend nixl \
|
||||
--disaggregation-mode prefill \
|
||||
--host ${local_ip} \
|
||||
--port 30000 \
|
||||
--trust-remote-code \
|
||||
--dist-init-addr ${prefill_master_ip}:5000 \
|
||||
--nnodes 2 \
|
||||
--node-rank 0 \
|
||||
--tp-size 16 \
|
||||
--dp-size 8 \
|
||||
--enable-dp-attention \
|
||||
--moe-a2a-backend deepep \
|
||||
--mem-fraction-static 0.8
|
||||
# prefill 1
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3-0324 \
|
||||
--disaggregation-transfer-backend nixl \
|
||||
--disaggregation-mode prefill \
|
||||
--host ${local_ip} \
|
||||
--port 30000 \
|
||||
--trust-remote-code \
|
||||
--dist-init-addr ${prefill_master_ip}:5000 \
|
||||
--nnodes 2 \
|
||||
--node-rank 1 \
|
||||
--tp-size 16 \
|
||||
--dp-size 8 \
|
||||
--enable-dp-attention \
|
||||
--moe-a2a-backend deepep \
|
||||
--mem-fraction-static 0.8
|
||||
# decode 0
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3-0324 \
|
||||
--disaggregation-transfer-backend nixl \
|
||||
--disaggregation-mode decode \
|
||||
--host ${local_ip} \
|
||||
--port 30001 \
|
||||
--trust-remote-code \
|
||||
--dist-init-addr ${decode_master_ip}:5000 \
|
||||
--nnodes 2 \
|
||||
--node-rank 0 \
|
||||
--tp-size 16 \
|
||||
--dp-size 8 \
|
||||
--enable-dp-attention \
|
||||
--moe-a2a-backend deepep \
|
||||
--mem-fraction-static 0.8 \
|
||||
--max-running-requests 128
|
||||
# decode 1
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3-0324 \
|
||||
--disaggregation-transfer-backend nixl \
|
||||
--disaggregation-mode decode \
|
||||
--host ${local_ip} \
|
||||
--port 30001 \
|
||||
--trust-remote-code \
|
||||
--dist-init-addr ${decode_master_ip}:5000 \
|
||||
--nnodes 2 \
|
||||
--node-rank 1 \
|
||||
--tp-size 16 \
|
||||
--dp-size 8 \
|
||||
--enable-dp-attention \
|
||||
--moe-a2a-backend deepep \
|
||||
--mem-fraction-static 0.8 \
|
||||
--max-running-requests 128
|
||||
```
|
||||
|
||||
### Advanced Configuration
|
||||
|
||||
#### NIXL Backend Selection
|
||||
|
||||
By default, NIXL uses the **UCX** backend for KV cache transfers. You can select a different NIXL plugin backend depending on your infrastructure using the environment variable `SGLANG_DISAGGREGATION_NIXL_BACKEND`.
|
||||
|
||||
Example: `export SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC`
|
||||
|
||||
**Available backends:** UCX (default), LIBFABRIC, or any installed NIXL plugin.
|
||||
|
||||
Example usage:
|
||||
```bash
|
||||
export SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
||||
--disaggregation-mode prefill \
|
||||
--disaggregation-transfer-backend nixl \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
## ASCEND
|
||||
|
||||
### Usage
|
||||
|
||||
Use ascend backend with [memfabric_hybrid](https://gitcode.com/Ascend/memfabric_hybrid) and ASCEND_MF_STORE_URL being set
|
||||
|
||||
```bash
|
||||
pip install memfabric-hybrid==1.0.5
|
||||
export ASCEND_MF_STORE_URL="tcp://xxx.xx.xxx.xxx:xxxx"
|
||||
```
|
||||
Use mooncake backend, more details can be found in mooncake section.
|
||||
```bash
|
||||
export ENABLE_ASCEND_TRANSFER_WITH_MOONCAKE=true
|
||||
```
|
||||
ASCEND_NPU_PHY_ID need to be set in container env
|
||||
```bash
|
||||
export ASCEND_NPU_PHY_ID=xxx
|
||||
```
|
||||
|
||||
|
||||
### Llama Single Node
|
||||
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
||||
--disaggregation-mode prefill \
|
||||
--port 30000 \
|
||||
--disaggregation-transfer-backend ascend
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
||||
--disaggregation-mode decode \
|
||||
--port 30001 \
|
||||
--base-gpu-id 1 \
|
||||
--disaggregation-transfer-backend ascend
|
||||
python -m sglang_router.launch_router --pd-disaggregation --prefill http://127.0.0.1:30000 --decode http://127.0.0.1:30001 --host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
### DeepSeek Multi-Node
|
||||
|
||||
```bash
|
||||
# prefill 0
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3-0324 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--disaggregation-mode prefill \
|
||||
--host ${local_ip} \
|
||||
--port 30000 \
|
||||
--trust-remote-code \
|
||||
--dist-init-addr ${prefill_master_ip}:5000 \
|
||||
--nnodes 1 \
|
||||
--node-rank 0 \
|
||||
--tp-size 16
|
||||
# decode 0
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3-0324 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--disaggregation-mode decode \
|
||||
--host ${local_ip} \
|
||||
--port 30001 \
|
||||
--trust-remote-code \
|
||||
--dist-init-addr ${decode_master_ip}:5000 \
|
||||
--nnodes 1 \
|
||||
--node-rank 0 \
|
||||
--tp-size 16
|
||||
```
|
||||
@@ -0,0 +1,299 @@
|
||||
---
|
||||
title: "Piecewise CUDA Graph"
|
||||
metatags:
|
||||
description: "Use Piecewise CUDA Graph to reduce prefill and extend kernel launch overhead while supporting dynamic token shapes."
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
||||
Standard CUDA graphs capture the entire model forward pass as a single graph. This works well for decode (fixed batch size), but not for extend/prefill where the number of tokens varies across iterations.
|
||||
|
||||
Piecewise CUDA Graph (PCG) solves this by splitting the model's computation graph into pieces (roughly one per layer) at "split points" (e.g., MoE dispatch ops). Each piece is captured as a separate CUDA graph for a set of pre-defined token lengths. At runtime, the input is padded to the nearest captured size, and each piece is replayed. This eliminates kernel launch overhead for prefill/extend while still supporting dynamic shapes.
|
||||
|
||||
Recently we **enabled PCG by default**, which means that the old `--enable-piecewise-cuda-graph` flag is deprecated. Use `--disable-piecewise-cuda-graph` to turn it off.
|
||||
|
||||
## Usage
|
||||
|
||||
PCG is enabled by default for supported configurations. No extra flags needed:
|
||||
|
||||
```bash
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct
|
||||
```
|
||||
|
||||
### Disable PCG
|
||||
|
||||
```bash
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
||||
--disable-piecewise-cuda-graph
|
||||
```
|
||||
|
||||
### Custom capture sizes
|
||||
|
||||
```bash
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
||||
--piecewise-cuda-graph-max-tokens 2048
|
||||
```
|
||||
|
||||
### Server Args
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "32%"}} />
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "48%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--disable-piecewise-cuda-graph</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>False</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Disable PCG for extend/prefill.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--enforce-piecewise-cuda-graph</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>False</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Force-enable PCG, skipping all auto-disable conditions. For testing only.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--piecewise-cuda-graph-max-tokens</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>None</code> (auto)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Maximum token count to capture. Defaults to <code>chunked_prefill_size</code> (non-MLA) or <code>2048</code> (MLA).</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--piecewise-cuda-graph-tokens</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>None</code> (auto)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Explicit list of token lengths to capture. Auto-generated if not set.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--piecewise-cuda-graph-compiler</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>"eager"</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Compiler backend for the captured subgraphs. Choices: <code>eager</code>, <code>inductor</code>.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><del><code>--enable-piecewise-cuda-graph</code></del></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Deprecated.</strong> PCG is now enabled by default. Use <code>--enforce-piecewise-cuda-graph</code> to skip auto-disable conditions.</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Bug Report
|
||||
|
||||
PCG is enabled by default but is still in an experimental stage. Since PCG relies on `torch.compile` to trace the model's forward pass, most bugs are introduced by torch compile tracing failures (e.g., untraceable ops, dynamic control flow, or graph breaks). If you encounter any issues related to PCG, please disable it by adding `--disable-piecewise-cuda-graph` to your launch command and report the bug at [GitHub Issues](https://github.com/sgl-project/sglang/issues/new/choose). We greatly appreciate your help in improving this feature.
|
||||
|
||||
### For Users
|
||||
|
||||
If you see an error message like the following during server startup, it is a PCG bug:
|
||||
|
||||
```
|
||||
Piecewise CUDA Graph is enabled by default as an experimental feature.
|
||||
To work around this error, add --disable-piecewise-cuda-graph to your launch command.
|
||||
Please report this issue at https://github.com/sgl-project/sglang/issues/new/choose
|
||||
```
|
||||
|
||||
To work around it, add `--disable-piecewise-cuda-graph` to your launch command. When filing a bug report, please include:
|
||||
1. The full error traceback
|
||||
2. Model name and quantization method
|
||||
3. Launch command with all arguments
|
||||
4. GPU type and driver version
|
||||
|
||||
### For Developers
|
||||
|
||||
Since PCG relies on `torch.compile` to trace the model's forward pass, newly developed CUDA kernels (both JIT kernels and sgl-kernels) are typically not compatible with `torch.compile` out of the box. The tracing will fail on untraceable operations such as JIT compilation, file I/O, or dynamic module loading inside the kernel.
|
||||
|
||||
To make a kernel compatible with PCG, you need to register it as a custom op using `register_custom_op` from `sglang.srt.utils.custom_op`. This wraps the kernel as an opaque node in the compiled graph so that `torch.compile` will not trace inside it.
|
||||
|
||||
**Example usage (JIT kernel):**
|
||||
|
||||
```python
|
||||
from sglang.srt.utils.custom_op import register_custom_op
|
||||
|
||||
# Inplace operator (no return value)
|
||||
@register_custom_op(mutates_args=["output_q", "output_s"])
|
||||
def per_token_group_quant_8bit(
|
||||
input: torch.Tensor,
|
||||
output_q: torch.Tensor,
|
||||
output_s: torch.Tensor,
|
||||
) -> None:
|
||||
# kernel implementation ...
|
||||
```
|
||||
|
||||
**Example usage (operator with output):**
|
||||
|
||||
```python
|
||||
# out_shape indicates which argument has the same shape as the output
|
||||
@register_custom_op(mutates_args=["x"], out_shape=0)
|
||||
def add(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor:
|
||||
return x.add_(y)
|
||||
```
|
||||
|
||||
For wrapping external library functions (e.g., FlashInfer kernels), use `register_custom_op_from_extern` instead. See `python/sglang/srt/utils/custom_op.py` for full API documentation.
|
||||
|
||||
## How it works
|
||||
|
||||
### Torch compile backend
|
||||
|
||||
PCG uses `torch.compile` with a custom backend (`SGLangBackend`) to split and compile the model's forward pass. The flow is:
|
||||
|
||||
```
|
||||
model.forward wrapper
|
||||
→ torch.compile(..., backend=SGLangBackend)
|
||||
→ FX graph
|
||||
→ split_graph() at registered split ops
|
||||
→ split_gm (top-level graph that chains the pieces)
|
||||
→ replace capturable submodules with CUDAPiecewiseBackend
|
||||
→ runtime dispatch: eager split ops + per-piece capture/replay
|
||||
```
|
||||
|
||||
- **Install**: `install_torch_compiled()` replaces `model.forward` with a wrapper function. When `is_in_piecewise_cuda_graph()` returns True, the wrapper dispatches to the compiled callable; otherwise it falls back to the original forward. The first invocation through this path triggers Dynamo tracing and graph compilation — CUDA graph replay only happens after the capture phase completes.
|
||||
|
||||
- **Split**: When `torch.compile` traces the model, `SGLangBackend` receives the FX graph and calls `split_graph()`. Ops listed in `CompilationConfig.split_ops` are treated as split points, so the graph is cut at each one. These split-op submodules are left to run eagerly at runtime, while the surrounding submodules are compiled and wrapped by `CUDAPiecewiseBackend`. The result is a top-level "stitching graph" (`split_gm`) with children such as `submod_0`, `submod_1`, … interleaving capturable subgraphs and eager split-op submodules.
|
||||
|
||||
- **Replace**: `PiecewiseCompileInterpreter` iterates over each capturable submodule in `split_gm`, compiles it for general (dynamic) shapes, and replaces it in-place with a `CUDAPiecewiseBackend` instance. Split-op submodules (e.g., attention, all-reduce) are left as-is and run eagerly at runtime.
|
||||
|
||||
- **Dispatch**: At runtime, calling `split_gm` executes the stitching graph, which calls each submodule in order. Split-op submodules run eagerly. Each `CUDAPiecewiseBackend` submodule goes through three phases:
|
||||
- **Compile warmup** — runs the general-shape compiled path.
|
||||
- **Capture** — for each capture size, runs one warmup pass then records a CUDA graph.
|
||||
- **Steady-state replay** — replays the captured CUDA graph for each forward pass.
|
||||
|
||||
### Piecewise cuda graph runner
|
||||
|
||||
`PiecewiseCudaGraphRunner` orchestrates the full lifecycle through three phases:
|
||||
|
||||
- **Compile** — Warms up JIT kernels with a dummy forward pass, then wraps the model with `torch.compile`, triggering Dynamo tracing to split the FX graph and create `CUDAPiecewiseBackend` instances for each subgraph piece.
|
||||
|
||||
- **Capture** — Iterates over capture sizes in reverse order (largest first). For each size, runs the forward pass twice (one warmup, one CUDA graph capture).
|
||||
|
||||
- **Replay** — At runtime, finds the smallest captured size >= actual token count via binary search, copies inputs into static buffers with zero-padding, replays the captured CUDA graphs, and slices outputs back to the actual token count.
|
||||
|
||||
### Memory optimization
|
||||
|
||||
The memory cost of PCG comes from two parts: **torch memory allocator** and **non-torch memory**.
|
||||
|
||||
The torch memory allocator overhead is trivial thanks to several optimizations: a global shared memory pool is reused across all CUDA graph runners and capture sizes, capture is done in reverse order (large to small) so smaller graphs reuse memory allocated by larger ones, and output tensors of the last subgraph are stored as weak references to maximize memory reuse.
|
||||
|
||||
The main memory overhead comes from non-torch memory — the CUDA graph objects themselves require GPU memory to store the recorded kernel launch parameters and internal state. This overhead scales with the number of captured sizes, which is why `piecewise_cuda_graph_max_tokens` is capped conservatively by default.
|
||||
|
||||
### Shape configuration
|
||||
|
||||
Piecewise CUDA graph pre-captures graphs for a set of token counts. At runtime, the actual token count is rounded up to the nearest captured size (via binary search), and the corresponding graph is replayed. If the token count exceeds the largest captured size, the runtime falls back to the normal (non-graph) forward path.
|
||||
|
||||
The default capture schedule is auto-generated with increasing granularity:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "50%"}} />
|
||||
<col style={{width: "50%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Token range</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Step size</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>4 – 32</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>4</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>48 – 256</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>288 – 512</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>32</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>576 – 1024</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>64</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>1280 – 4096</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>256</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>4096+</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>512</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
For the auto-generated schedule, sizes are capped at `--piecewise-cuda-graph-max-tokens`. The default cap is `chunked_prefill_size` for non-MLA models and `2048` for MLA backend models. If `--max-total-tokens` is set, the cap is further limited to not exceed it. Additionally, Llama-2 models are auto-capped at 4096 tokens as a temporary workaround.
|
||||
|
||||
## Compatibility
|
||||
|
||||
PCG is auto-disabled in the following scenarios. We are actively working on expanding compatibility — support for many of these will be coming soon.
|
||||
|
||||
- Disabled model architectures (e.g., `DeepseekV32ForCausalLM`)
|
||||
- Speculative decoding
|
||||
- DP attention
|
||||
- Pipeline parallelism (`pp_size > 1`)
|
||||
- Non-CUDA hardware (AMD ROCm, Ascend NPU)
|
||||
- MoE A2A backend
|
||||
- LoRA
|
||||
- Multimodal / VLM models
|
||||
- DLLM (diffusion LLM)
|
||||
- Deterministic inference
|
||||
- PD disaggregation
|
||||
- Expert distribution recorder / EPLB
|
||||
|
||||
Use `--enforce-piecewise-cuda-graph` to skip all auto-disable checks (for testing/debugging only).
|
||||
|
||||
## Code Reference
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "48%"}} />
|
||||
<col style={{width: "52%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>File</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/model_executor/piecewise_cuda_graph_runner.py</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Main runner: init, capture, replay</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/compilation/compile.py</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>install_torch_compiled</code> trampoline</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/compilation/backend.py</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>SGLangBackend</code>, graph splitting, piecewise compilation</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/compilation/cuda_piecewise_backend.py</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Per-subgraph CUDA graph capture/replay</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/compilation/piecewise_context_manager.py</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Global context flags and <code>ForwardContext</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/compilation/compilation_config.py</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Capture sizes, split ops, compiler config</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/utils/custom_op.py</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>register_custom_op</code> for torch.compile compatibility</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/server_args.py</code></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Server arguments and auto-disable logic</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
@@ -0,0 +1,119 @@
|
||||
---
|
||||
title: "Pipeline Parallelism for Long Context"
|
||||
metatags:
|
||||
description: "SGLang pipeline parallelism: reduce TTFT for ultra-long sequences, dynamic chunking, async P2P communication, multi-node deployment."
|
||||
---
|
||||
## Why Pipeline Parallelism?
|
||||
|
||||
As Large Language Models (LLMs) scale toward trillion-parameter architectures and "infinite" context windows, the underlying serving infrastructure must evolve toward more granular, cross-node parallelization strategies. While KV cache techniques effectively mitigate redundant computation, they cannot circumvent the prohibitive Time to First Token (TTFT) inherent in ultra-long sequences with extremely large initial Input Token Length (ITL). Although Tensor Parallelism (TP) remains the conventional approach for intra-node scaling, it frequently encounters communication bottlenecks during multi-node deployments. On the other hand, pipeline parallelism only requires cross-node communication at the boundaries of each pipeline stage, which can achieve better computation-communication overlap compared to a large TP. Therefore, it is also a promising parallelization strategy for improving throughput.
|
||||
|
||||
Detailed analysis can be found in this [blog](https://lmsys.org/blog/2026-01-15-chunked-pipeline/).
|
||||
|
||||
## Implementation Refactoring based on Async Communication
|
||||
With Dynamic Chunked Prefill, pipeline parallelism has the potential to reduce the TTFT of long-context inputs. For each request, its input tokens can be partitioned into multiple chunks, each no longer than the chunked prefill size. Different chunks of the same request can be processed simultaneously by different nodes, thus parallelizing the processing and reducing TTFT. SGLang has supported Pipeline Parallelism (#5724) for some time and made it compatible with the PD Disaggregation feature (#8846), but the implementation was not perfect and had significant room for performance improvements.
|
||||
|
||||
To eliminate this performance hazard, SGLang implements a Micro-batching Event Loop with non-blocking asynchronous peer-to-peer (P2P) communication to overlap GPU computation with CPU metadata processing and PP communication. This ensures that while one micro-batch is being computed on the GPU, the next one is already being prepared and moved into position effectively, ensuring the pipeline remains as saturated as possible. This approach was first proposed in #7979 and has been redesigned and included in #11852.
|
||||
|
||||
The key mechanisms of the implementation include:
|
||||
|
||||
* **Decoupled Sync/Async Logic in the Event Loop:** The scheduler uses `async_send` in `_pp_send_pyobj_to_next_stage`. Instead of waiting for a transfer to complete, it returns a `P2PWork` handle. The actual synchronization (`P2PWork.work.wait()`) is deferred until `_pp_commit_comm_work` is called, allowing the CPU to perform other work—like scheduling the next batch or processing metadata—while data is in flight.
|
||||
* **Multi-Stream Execution:** In addition to the main `default_stream`, which serves as the synchronization stream, SGLang utilizes dedicated `forward_stream` and `copy_stream` to execute forward pass GPU computation and Data-to-Host (D2H) memory transfers separately for better overlapping. While `_pp_launch_batch` is executing the current micro-batch on the GPU for the current stage, the CPU processes the previous micro-batch's results using `_pp_process_batch_result`.
|
||||
|
||||
## Guidance about Dynamic Chunking
|
||||
|
||||
### Why Dynamic Chunking
|
||||
Chunked prefill with a fixed size can cause bubbles in the pipeline, especially when the pp size is large. The main reason behind this phenomenon is that the model has a non-uniform running time, even though each chunk size is identical (brought by the Transformer structure). The larger the prefix sequence length, the longer the running time of the chunk. And these bubbles will be propagated to the next stage, and will significantly degrade the scale efficiency of larger pp ranks.
|
||||
|
||||
To address this issue, SGLang introduces a dynamic chunking mechanism to predict the optimal size for the next chunk such that it satisfies this condition:
|
||||
|
||||
Runtime(L + Next Chunk Size) - Runtime(L) = Runtime(Initial Chunk Size)
|
||||
|
||||
where ***L*** denotes the Prefix Sequence Length. By profiling a series of requests with different ITLs, we model the cumulative runtime as a quadratic function of sequence length. Using this model, we solve the optimal next chunk size for any given prefix length ***L***. Since the computation complexity of the Attention mechanism scales with ***L***, the next chunk size will be progressively reduced as ***L*** grows to maintain an aligned chunk execution time across pipeline stages.
|
||||
|
||||
Based on this method, the scheduler can predict and dynamically reduce the chunk size during runtime to minimize the bubbles caused by the stage misalignment. To be noticed, the scheduler does not use the raw predicted value. To facilitate efficient KVCache memory management and ensure affinity with hardware execution efficiency, the value is aligned downward to the nearest multiple of max(`--page-size`, 64).
|
||||
|
||||
|
||||
### Chunked Prefill Size and Smoothing Factor
|
||||
|
||||
When `--enable-dynamic-chunking` is enabled, each chunk size of a sequence is determined dynamically based on the quadratic model that predicts the next chunk size based on the estimated runtime of the initial chunk length. In this case, we use `--chunked-prefill-size` to set up the initial chunk size. When switching to the dynamic chunking mode, the initial chunk size (`--chunked-prefill-size`) should be set to a larger value comparable to the original chunked prefill size, so that there won't be too many chunks.
|
||||
|
||||
**`SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR`** is an environmental variable that controls the smoothing factor for the dynamic chunking algorithm, defaulting to 0.75. It determines how much the chunk size can change during the prefill phase. A larger value means a more aggressive chunk size change, which may lead to better performance but also to greater chunk size changes (the chunk size at the end may become very small, which could lead to performance degradation) and more total chunks. When it is set to 1, the chunk size will be adjusted strictly based on the aforementioned quadratic model that predicts the next chunk size. A smaller value means a more conservative chunk size change, which may lead to smaller chunk size changes and fewer total chunks. When it is set to 0, the chunk size will not be adjusted dynamically, so it is identical to the traditional way with a fixed chunked prefill size.
|
||||
|
||||
Due to the variation in hardware, models, and target workloads, a static configuration is seldom optimal across all scenarios. Consequently, achieving peak performance necessitates a degree of hyperparameter tuning when switching to the dynamic chunking mode.
|
||||
|
||||
**Tuning Guidance for Dynamic Chunked Prefill**
|
||||
|
||||
* **Step 1 \- Iterate to find the optimal fixed chunked prefill size for the targeted PP size**: Different PP sizes for targeted ITL may have different optimal chunked prefill sizes. Therefore, users should iterate to obtain the baseline according to the available resources for scaling.
|
||||
* **Step 2 \- Initial Chunk Size Selection for Dynamic Chunking**: Set the initial size to 2× or 3× the optimal fixed chunked prefill size. This reduces the total number of chunks and prevents "tail chunks" from underutilizing hardware. To maintain efficiency for extremely large Input Token Lengths (ITL), the dynamic predictor automatically ensures subsequent chunks are at least 1/4 of this initial size. In addition, it is recommended to use a larger initial chunk size (e.g., 4× the optimal fixed chunked prefill size) for such cases as well.
|
||||
* **Step 3 \- Smooth Factor Adjustment**: This factor controls how strictly the chunk size adjusts the prediction given by the quadratic performance fitting model.
|
||||
* 1.0: Follows the model strictly.
|
||||
* **0.6 – 0.85 (Recommended)**: Typical range for the best balance between dynamic scaling and hardware stability. Through experiments, we find that a range between 0.6 and 0.85 typically yields the best performance for dynamic chunking.
|
||||
* 0: Disables dynamic adjustment, reverting to traditional fixed-size chunking.
|
||||
* **Another small optimization tip:** Put the larger partition in the higher PP rank when the layers are not evenly divisible across ranks. It can increase the GPU utilization when a larger PP rank is waiting for the previous stage’s result, hence reducing the bubbles on higher PP ranks. If we take DeepSeek-V3.1 as an example, `SGLANG_PP_LAYER_PARTITION=15,15,15,16` usually performs better than `16,15,15,15`.
|
||||
|
||||
## Best Practice for Long Context
|
||||
|
||||
### Tuning the Chunked Prefill Size
|
||||
Optimizing the chunked prefill size is crucial for balancing pipeline efficiency and resource utilization. The ideal size depends on factors including model architecture, hardware configuration, and typical input lengths. We recommend starting with a small chunk size, such as 4K, and gradually increasing it until you find the optimal size for your specific use case (Different targeted ITL and PP Sizes may have different optimal chunked prefill sizes. Therefore, users should iterate to obtain the baseline according to the available resources for scaling). Alternatively, you can analyze the hardware capacity and determine the optimal chunk size based on the roofline model.
|
||||
|
||||
### Enable Dynamic Chunking and Adjust Smoothing Factor for Ultra-long ITL
|
||||
SGLang also offers a dynamic chunking solution that could further improve performance. This feature is currently an experimental feature that requires a certain amount of tuning experimentation and may not be suitable for all workloads. In addition, fine-tuning the smoothing factor can help optimize performance for specific workloads and model characteristics.
|
||||
|
||||
### Case Study on NVIDIA H20
|
||||
|
||||
When evaluating pipeline parallelism with fixed chunked prefill sizes from 2K to 16K, experiment results show that a 4K chunk size delivered optimal prefill TTFT performance for the DeepSeek-V3.1, and a 6K chunk size delivered optimal prefill TTFT performance for the Qwen3-235B-A22B-FP8.
|
||||
|
||||
When enabling dynamic chunking, we first scale the optimal fixed chunked prefill size by a factor of 3 as the initial chunk size. Through experimentation, we found that a multiplier of 2-3 provides an appropriate balance—avoiding excessive initial pipeline bubbles while ensuring that subsequent chunks don't become too small as context length increases. With the default dynamic chunking smoothing factor of 0.75, we performed parameter tuning and determined that a value of 0.65 works optimally with the 12K initial chunk size for the DeepSeek-V3.1, while a value of 0.8 works optimally with the 18K initial chunk size for the Qwen3-235B-A22B-FP8.
|
||||
|
||||
#### DeepSeek-V3.1 with 128K Input Token Length
|
||||
```bash Command
|
||||
# prefill node 0 (fixed chunked prefill size)
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3.1 --trust-remote-code \
|
||||
--nnodes 4 --node-rank 0 --tp 8 --pp-size 4 \
|
||||
--port 30000 --dist-init-addr <MASTER_NODE_IP> \
|
||||
--disable-radix-cache --mem-fraction-static 0.8 \
|
||||
--attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \
|
||||
--max-running-requests 128 --chunked-prefill-size 4096
|
||||
```
|
||||
|
||||
```bash Command
|
||||
# prefill node 0 (with dynamic chunking)
|
||||
export SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR=0.65
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3.1 --trust-remote-code \
|
||||
--nnodes 4 --node-rank 0 --tp 8 --pp-size 4 \
|
||||
--port 30000 --dist-init-addr <MASTER_NODE_IP> \
|
||||
--disable-radix-cache --mem-fraction-static 0.8 \
|
||||
--attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \
|
||||
--max-running-requests 128 --chunked-prefill-size 12288 --enable-dynamic-chunking
|
||||
```
|
||||
|
||||
#### Qwen3-235B-A22B-FP8 with 128K Input Token Length
|
||||
```bash Command
|
||||
# prefill node 0 (fixed chunked prefill size)
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-235B-A22B-FP8 --trust-remote-code \
|
||||
--nnodes 4 --node-rank 0 --tp 4 --pp-size 8 \
|
||||
--port 30000 --dist-init-addr <MASTER_NODE_IP> \
|
||||
--disable-radix-cache --mem-fraction-static 0.8 \
|
||||
--attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \
|
||||
--max-running-requests 128 --chunked-prefill-size 6144
|
||||
```
|
||||
|
||||
```bash Command
|
||||
# prefill node 0 (with dynamic chunking)
|
||||
export SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR=0.8
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-235B-A22B-FP8 --trust-remote-code \
|
||||
--nnodes 4 --node-rank 0 --tp 4 --pp-size 8 \
|
||||
--port 30000 --dist-init-addr <MASTER_NODE_IP> \
|
||||
--disable-radix-cache --mem-fraction-static 0.8 \
|
||||
--attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \
|
||||
--max-running-requests 128 --chunked-prefill-size 18432 --enable-dynamic-chunking
|
||||
```
|
||||
|
||||
Note: `--disable-radix-cache` is enabled only for reproducible benchmarking purposes. It is not recommended to use it in production.
|
||||
|
||||
## Best Practice for Pipeline Parallelism with PD Disaggregation
|
||||
To be added. Stay tuned for the latest updates on Pipeline Parallelism with PD Disaggregation.
|
||||
@@ -0,0 +1,395 @@
|
||||
---
|
||||
title: "Quantization"
|
||||
metatags:
|
||||
description: "SGLang quantization: FP8, FP4, AWQ, GPTQ, ModelOpt, torchao. Offline and online quantization methods for efficient LLM inference."
|
||||
---
|
||||
SGLang supports various quantization methods, including offline quantization and online dynamic quantization.
|
||||
|
||||
Offline quantization loads pre-quantized model weights directly during inference. This is required for quantization methods
|
||||
such as GPTQ and AWQ, which collect and pre-compute various statistics from the original weights using the calibration dataset.
|
||||
|
||||
Online quantization dynamically computes scaling parameters—such as the maximum/minimum values of model weights—during runtime.
|
||||
Like NVIDIA FP8 training's [delayed scaling](https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/examples/fp8_primer.html#Mixed-precision-training-with-FP8) mechanism, online quantization calculates the appropriate scaling factors
|
||||
on-the-fly to convert high-precision weights into a lower-precision format.
|
||||
|
||||
**Note: For better performance, usability and convenience, offline quantization is recommended over online quantization.**
|
||||
|
||||
If you use a pre-quantized model, do not add `--quantization` to enable online quantization at the same time.
|
||||
For popular pre-quantized models, please visit [Unsloth](https://huggingface.co/unsloth), [NVIDIA ModelOpt](https://huggingface.co/collections/nvidia/inference-optimized-checkpoints-with-model-optimizer)
|
||||
or [NeuralMagic](https://huggingface.co/collections/neuralmagic) collections on HF for some
|
||||
popular quality validated quantized models. Quantized models must be validated via benchmarks post-quantization
|
||||
to guard against abnormal quantization loss regressions.
|
||||
|
||||
## Offline Quantization
|
||||
|
||||
To load already quantized models, simply load the model weights and config. **Again, if the model has been quantized offline,
|
||||
there's no need to add `--quantization` argument when starting the engine. The quantization method will be parsed from the
|
||||
downloaded Hugging Face config. For example, DeepSeek V3/R1 models are already in FP8, so do not add redundant parameters.**
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 \
|
||||
--port 30000 --host 0.0.0.0
|
||||
```
|
||||
|
||||
Take note, if your model is **per-channel quantized (INT8 or FP8) with per-token dynamic quantization activation**, you can opt to include `--quantization w8a8_int8` or `--quantization w8a8_fp8` to invoke the corresponding CUTLASS int8_kernel or fp8_kernel in sgl-kernel. This action will ignore the Hugging Face config's quantization settings. For instance, with `neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic`, if you execute with `--quantization w8a8_fp8`, the system will use the `W8A8Fp8Config` from SGLang to invoke the sgl-kernel, rather than the `CompressedTensorsConfig` for vLLM kernels.
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic \
|
||||
--quantization w8a8_fp8 \
|
||||
--port 30000 --host 0.0.0.0
|
||||
```
|
||||
|
||||
### Examples of Offline Model Quantization
|
||||
|
||||
#### Using [Unsloth](https://docs.unsloth.ai/basics/inference-and-deployment/sglang-guide)
|
||||
|
||||
We strongly suggest the use of Unsloth to quantize and load the model. Please refer to [SGLang Deployment & Inference Guide with Unsloth](https://docs.unsloth.ai/basics/inference-and-deployment/sglang-guide).
|
||||
|
||||
#### Using [auto-round](https://github.com/intel/auto-round)
|
||||
|
||||
```bash Command
|
||||
# Install
|
||||
pip install auto-round
|
||||
```
|
||||
|
||||
- LLM quantization
|
||||
|
||||
```py Example
|
||||
# for LLM
|
||||
from auto_round import AutoRound
|
||||
model_id = "meta-llama/Llama-3.2-1B-Instruct"
|
||||
quant_path = "Llama-3.2-1B-Instruct-autoround-4bit"
|
||||
# Scheme examples: "W2A16", "W3A16", "W4A16", "W8A16", "NVFP4", "MXFP4" (no real kernels), "GGUF:Q4_K_M", etc.
|
||||
scheme = "W4A16"
|
||||
format = "auto_round"
|
||||
autoround = AutoRound(model_id, scheme=scheme)
|
||||
autoround.quantize_and_save(quant_path, format=format) # quantize and save
|
||||
|
||||
```
|
||||
|
||||
- VLM quantization
|
||||
```py Example
|
||||
# for VLMs
|
||||
from auto_round import AutoRoundMLLM
|
||||
model_name = "Qwen/Qwen2-VL-2B-Instruct"
|
||||
quant_path = "Qwen2-VL-2B-Instruct-autoround-4bit"
|
||||
scheme = "W4A16"
|
||||
format = "auto_round"
|
||||
autoround = AutoRoundMLLM(model_name, scheme)
|
||||
autoround.quantize_and_save(quant_path, format=format) # quantize and save
|
||||
|
||||
```
|
||||
|
||||
- Command Line Usage (Gaudi/CPU/Intel GPU/CUDA)
|
||||
|
||||
```bash Command
|
||||
auto-round \
|
||||
--model meta-llama/Llama-3.2-1B-Instruct \
|
||||
--bits 4 \
|
||||
--group_size 128 \
|
||||
--format "auto_round" \
|
||||
--output_dir ./tmp_autoround
|
||||
```
|
||||
|
||||
- known issues
|
||||
|
||||
Several limitations currently affect offline quantized model loading in sglang, These issues might be resolved in future updates of sglang. If you experience any problems, consider using Hugging Face Transformers as an alternative.
|
||||
|
||||
1. Mixed-bit Quantization Limitations
|
||||
|
||||
Mixed-bit quantization is not fully supported. Due to vLLM's layer fusion (e.g., QKV fusion), applying different bit-widths to components within the same fused layer can lead to compatibility issues.
|
||||
|
||||
|
||||
2. Limited Support for Quantized MoE Models
|
||||
|
||||
Quantized MoE models may encounter inference issues due to kernel limitations (e.g., lack of support for mlp.gate layer quantization). please try to skip quantizing these layers to avoid such errors.
|
||||
|
||||
|
||||
3. Limited Support for Quantized VLMs
|
||||
<Accordion title="Details">
|
||||
{/* VLM failure cases */}
|
||||
|
||||
Qwen2.5-VL-7B
|
||||
|
||||
auto_round:auto_gptq format: Accuracy is close to zero.
|
||||
|
||||
GPTQ format: Fails with:
|
||||
```text Output
|
||||
The output size is not aligned with the quantized weight shape
|
||||
```
|
||||
auto_round:auto_awq and AWQ format: These work as expected.
|
||||
</Accordion>
|
||||
|
||||
#### Using [GPTQModel](https://github.com/ModelCloud/GPTQModel)
|
||||
|
||||
```bash Command
|
||||
# install
|
||||
pip install gptqmodel --no-build-isolation -v
|
||||
```
|
||||
|
||||
```py Example
|
||||
from datasets import load_dataset
|
||||
from gptqmodel import GPTQModel, QuantizeConfig
|
||||
|
||||
model_id = "meta-llama/Llama-3.2-1B-Instruct"
|
||||
quant_path = "Llama-3.2-1B-Instruct-gptqmodel-4bit"
|
||||
|
||||
calibration_dataset = load_dataset(
|
||||
"allenai/c4", data_files="en/c4-train.00001-of-01024.json.gz",
|
||||
split="train"
|
||||
).select(range(1024))["text"]
|
||||
|
||||
quant_config = QuantizeConfig(bits=4, group_size=128) # quantization config
|
||||
model = GPTQModel.load(model_id, quant_config) # load model
|
||||
|
||||
model.quantize(calibration_dataset, batch_size=2) # quantize
|
||||
model.save(quant_path) # save model
|
||||
```
|
||||
|
||||
#### Using [LLM Compressor](https://github.com/vllm-project/llm-compressor/)
|
||||
|
||||
```bash Command
|
||||
# install
|
||||
pip install llmcompressor
|
||||
```
|
||||
|
||||
Here, we take quantize `meta-llama/Meta-Llama-3-8B-Instruct` to `FP8` as an example to elaborate on how to do offline quantization.
|
||||
|
||||
```python Example
|
||||
from transformers import AutoTokenizer
|
||||
from llmcompressor.transformers import SparseAutoModelForCausalLM
|
||||
from llmcompressor.transformers import oneshot
|
||||
from llmcompressor.modifiers.quantization import QuantizationModifier
|
||||
|
||||
# Step 1: Load the original model.
|
||||
MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct"
|
||||
|
||||
model = SparseAutoModelForCausalLM.from_pretrained(
|
||||
MODEL_ID, device_map="auto", torch_dtype="auto")
|
||||
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
|
||||
|
||||
# Step 2: Perform offline quantization.
|
||||
# Step 2.1: Configure the simple PTQ quantization.
|
||||
recipe = QuantizationModifier(
|
||||
targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"])
|
||||
|
||||
# Step 2.2: Apply the quantization algorithm.
|
||||
oneshot(model=model, recipe=recipe)
|
||||
|
||||
# Step 3: Save the model.
|
||||
SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-Dynamic"
|
||||
model.save_pretrained(SAVE_DIR)
|
||||
tokenizer.save_pretrained(SAVE_DIR)
|
||||
```
|
||||
|
||||
Then, you can directly use the quantized model with `SGLang`, by using the following command:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path $PWD/Meta-Llama-3-8B-Instruct-FP8-Dynamic \
|
||||
--port 30000 --host 0.0.0.0
|
||||
```
|
||||
|
||||
#### Using [NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer)
|
||||
|
||||
NVIDIA Model Optimizer (ModelOpt) provides advanced quantization techniques optimized for NVIDIA hardware. SGLang includes a streamlined workflow for quantizing models with ModelOpt and automatically exporting them for deployment.
|
||||
|
||||
##### Installation
|
||||
|
||||
First, install ModelOpt. You can either install it directly or as an optional SGLang dependency:
|
||||
|
||||
```bash Command
|
||||
# Option 1: Install ModelOpt directly
|
||||
pip install nvidia-modelopt
|
||||
|
||||
# Option 2: Install SGLang with ModelOpt support (recommended)
|
||||
pip install sglang[modelopt]
|
||||
```
|
||||
|
||||
##### Quantization and Export Workflow
|
||||
|
||||
SGLang provides an example script that demonstrates the complete ModelOpt quantization and export workflow:
|
||||
|
||||
```bash Command
|
||||
# Quantize and export a model using ModelOpt FP8 quantization
|
||||
python examples/usage/modelopt_quantize_and_export.py quantize \
|
||||
--model-path TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
|
||||
--export-dir ./quantized_tinyllama_fp8 \
|
||||
--quantization-method modelopt_fp8
|
||||
|
||||
# For FP4 quantization
|
||||
python examples/usage/modelopt_quantize_and_export.py quantize \
|
||||
--model-path TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
|
||||
--export-dir ./quantized_tinyllama_fp4 \
|
||||
--quantization-method modelopt_fp4
|
||||
```
|
||||
|
||||
##### Available Quantization Methods
|
||||
|
||||
- `modelopt_fp8`: FP8 quantization with optimal performance on NVIDIA Hopper and Blackwell GPUs
|
||||
- `modelopt_fp4`: FP4 quantization with optimal performance on Nvidia Blackwell GPUs
|
||||
|
||||
##### Python API Usage
|
||||
|
||||
You can also use ModelOpt quantization programmatically:
|
||||
|
||||
```python Example
|
||||
import sglang as sgl
|
||||
from sglang.srt.configs.device_config import DeviceConfig
|
||||
from sglang.srt.configs.load_config import LoadConfig
|
||||
from sglang.srt.configs.model_config import ModelConfig
|
||||
from sglang.srt.model_loader.loader import get_model_loader
|
||||
|
||||
# Configure model with ModelOpt quantization and export
|
||||
model_config = ModelConfig(
|
||||
model_path="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
|
||||
quantization="modelopt_fp8", # or "modelopt_fp4"
|
||||
trust_remote_code=True,
|
||||
)
|
||||
|
||||
load_config = LoadConfig(
|
||||
modelopt_export_path="./exported_model",
|
||||
modelopt_checkpoint_save_path="./checkpoint.pth", # optional, fake quantized checkpoint
|
||||
)
|
||||
device_config = DeviceConfig(device="cuda")
|
||||
|
||||
# Load and quantize the model (export happens automatically)
|
||||
model_loader = get_model_loader(load_config, model_config)
|
||||
quantized_model = model_loader.load_model(
|
||||
model_config=model_config,
|
||||
device_config=device_config,
|
||||
)
|
||||
```
|
||||
|
||||
##### Deploying Quantized Models
|
||||
|
||||
After quantization and export, you can deploy the model with SGLang:
|
||||
|
||||
```bash Command
|
||||
# Deploy the exported quantized model
|
||||
python -m sglang.launch_server \
|
||||
--model-path ./quantized_tinyllama_fp8 \
|
||||
--quantization modelopt \
|
||||
--port 30000 --host 0.0.0.0
|
||||
```
|
||||
|
||||
Or using the Python API:
|
||||
|
||||
```python Example
|
||||
import sglang as sgl
|
||||
|
||||
# Deploy exported ModelOpt quantized model
|
||||
llm = sgl.Engine(
|
||||
model_path="./quantized_tinyllama_fp8",
|
||||
quantization="modelopt"
|
||||
)
|
||||
|
||||
# Run inference
|
||||
prompts = ["Hello, how are you?", "What is the capital of France?"]
|
||||
sampling_params = {"temperature": 0.8, "top_p": 0.95, "max_new_tokens": 100}
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
|
||||
for i, output in enumerate(outputs):
|
||||
print(f"Prompt: {prompts[i]}")
|
||||
print(f"Output: {output.outputs[0].text}")
|
||||
```
|
||||
|
||||
##### Advanced Features
|
||||
|
||||
**Checkpoint Management**: Save and restore fake quantized checkpoints for reuse:
|
||||
|
||||
```bash Command
|
||||
# Save the fake quantized checkpoint during quantization
|
||||
python examples/usage/modelopt_quantize_and_export.py quantize \
|
||||
--model-path meta-llama/Llama-3.2-1B-Instruct \
|
||||
--export-dir ./quantized_model \
|
||||
--quantization-method modelopt_fp8 \
|
||||
--checkpoint-save-path ./my_checkpoint.pth
|
||||
|
||||
# The checkpoint can be reused for future quantization runs and skip calibration
|
||||
```
|
||||
|
||||
**Export-only Workflow**: If you have a pre-existing fake quantized ModelOpt checkpoint, you can export it directly:
|
||||
|
||||
```python Example
|
||||
from sglang.srt.configs.device_config import DeviceConfig
|
||||
from sglang.srt.configs.load_config import LoadConfig
|
||||
from sglang.srt.configs.model_config import ModelConfig
|
||||
from sglang.srt.model_loader.loader import get_model_loader
|
||||
|
||||
model_config = ModelConfig(
|
||||
model_path="meta-llama/Llama-3.2-1B-Instruct",
|
||||
quantization="modelopt_fp8",
|
||||
trust_remote_code=True,
|
||||
)
|
||||
|
||||
load_config = LoadConfig(
|
||||
modelopt_checkpoint_restore_path="./my_checkpoint.pth",
|
||||
modelopt_export_path="./exported_model",
|
||||
)
|
||||
|
||||
# Load and export the model
|
||||
model_loader = get_model_loader(load_config, model_config)
|
||||
model_loader.load_model(model_config=model_config, device_config=DeviceConfig())
|
||||
```
|
||||
|
||||
##### Benefits of ModelOpt
|
||||
|
||||
- **Hardware Optimization**: Specifically optimized for NVIDIA GPU architectures
|
||||
- **Advanced Quantization**: Supports cutting-edge FP8 and FP4 quantization techniques
|
||||
- **Seamless Integration**: Automatic export to HuggingFace format for easy deployment
|
||||
- **Calibration-based**: Uses calibration datasets for optimal quantization quality
|
||||
- **Production Ready**: Enterprise-grade quantization with NVIDIA support
|
||||
|
||||
## Online Quantization
|
||||
|
||||
To enable online quantization, you can simply specify `--quantization` in the command line. For example, you can launch the server with the following command to enable `FP8` quantization for model `meta-llama/Meta-Llama-3.1-8B-Instruct`:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--quantization fp8 \
|
||||
--port 30000 --host 0.0.0.0
|
||||
```
|
||||
|
||||
Our team is working on supporting more online quantization methods. SGLang will soon support methods including but not limited to `["awq", "gptq", "marlin", "gptq_marlin", "awq_marlin", "bitsandbytes", "gguf"]`.
|
||||
|
||||
### torchao online quantization method
|
||||
|
||||
SGLang also supports quantization methods based on [torchao](https://github.com/pytorch/ao). You can simply specify `--torchao-config` in the command line to support this feature. For example, if you want to enable `int4wo-128` for model `meta-llama/Meta-Llama-3.1-8B-Instruct`, you can launch the server with the following command:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--torchao-config int4wo-128 \
|
||||
--port 30000 --host 0.0.0.0
|
||||
```
|
||||
|
||||
SGLang supports the following quantization methods based on torchao `["int8dq", "int8wo", "fp8wo", "fp8dq-per_tensor", "fp8dq-per_row", "int4wo-32", "int4wo-64", "int4wo-128", "int4wo-256"]`.
|
||||
|
||||
Note: According to [this issue](https://github.com/sgl-project/sglang/issues/2219#issuecomment-2561890230), `"int8dq"` method currently has some bugs when using together with cuda graph capture. So we suggest to disable cuda graph capture when using `"int8dq"` method. Namely, please use the following command:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--torchao-config int8dq \
|
||||
--disable-cuda-graph \
|
||||
--port 30000 --host 0.0.0.0
|
||||
```
|
||||
|
||||
### `quark_int4fp8_moe` online quantization method
|
||||
|
||||
SGLang running on AMD GPUs (CDNA3 or CDNA4 architecture) supports the quantization method `--quantization quark_int4fp8_moe`, that will replace [MoE layers](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/layers/moe/fused_moe_triton/layer.py) originally in high precision (bfloat16, float16 or float32) to use weights dynamically quantized to int4, that are upcasted to float8 during inference to run compute in float8 precision with activations dynamically quantized on the fly to float8.
|
||||
|
||||
Other layers (e.g. projections in the attention layers) have their weights quantized online to float8 directly.
|
||||
|
||||
## Reference
|
||||
|
||||
- [GPTQModel](https://github.com/ModelCloud/GPTQModel)
|
||||
- [LLM Compressor](https://github.com/vllm-project/llm-compressor/)
|
||||
- [NVIDIA Model Optimizer (ModelOpt)](https://github.com/NVIDIA/Model-Optimizer)
|
||||
- [Torchao: PyTorch Architecture Optimization](https://github.com/pytorch/ao)
|
||||
- [vLLM Quantization](https://docs.vllm.ai/en/latest/quantization/)
|
||||
- [auto-round](https://github.com/intel/auto-round)
|
||||
@@ -0,0 +1,256 @@
|
||||
---
|
||||
title: "Quantized KV Cache"
|
||||
metatags:
|
||||
description: "SGLang quantized KV cache: FP8 E4M3/E5M2 and FP4 E2M1 formats, memory savings up to 3.56x, scaling factors, accuracy benchmarks."
|
||||
---
|
||||
Quantized KV cache reduces the memory footprint of key-value cache storage by using lower-precision data types (FP8 or FP4) instead of the default model precision in BF16. During autoregressive generation, LLMs cache previously computed key-value pairs to avoid redundant calculations. The KV cache typically consumes a significant portion of GPU memory, especially for long sequences.
|
||||
|
||||
Quantized KV cache is a memory optimization technique that primarily benefits throughput by allowing more tokens to be cached, but may introduce minimal accuracy degradation depending on the quantization format used.
|
||||
|
||||
<Warning>
|
||||
**Performance Warning**: When quantized KV cache must be dequantized before use in attention operations, performance can be extremely slow if dequantization is not fused with the attention kernel. Always verify that your chosen attention backend supports quantized KV cache. Backends without fused support may experience significant throughput degradation, potentially negating the memory benefits.
|
||||
|
||||
**Backend Support**: Not all attention backends support quantized KV cache. Refer to [Attention Backend](./attention_backend) for which backends support it.
|
||||
</Warning>
|
||||
|
||||
## Supported Formats
|
||||
|
||||
SGLang supports the following quantized KV cache formats:
|
||||
|
||||
### FP8 Format
|
||||
|
||||
[OCP (Open Compute Project)](https://www.opencompute.org) specifies two common 8-bit floating point formats:
|
||||
|
||||
- **E5M2** (5 exponent bits, 2 mantissa bits): Larger dynamic range (±57344.0), lower precision
|
||||
- **E4M3** (4 exponent bits, 3 mantissa bits): Higher precision, smaller dynamic range (±240.0)
|
||||
|
||||
### FP4 Format
|
||||
|
||||
<Warning>
|
||||
FP4 quantization is currently experimental.
|
||||
</Warning>
|
||||
|
||||
[OCP (Open Compute Project)](https://www.opencompute.org) specifies MXFP4 (Microscaling FP4), a 4-bit floating-point format:
|
||||
|
||||
- **E2M1** (1 sign bit, 2 exponent bits, 1 mantissa bit): Uses block-based microscaling where tensors are divided into blocks of consecutive elements, with each block sharing a single 8-bit exponential scaling factor. While OCP specifies blocks of 32 elements, SGLang's current implementation uses blocks of 16 elements for KV cache quantization.
|
||||
|
||||
## Usage
|
||||
|
||||
### Enabling Quantized KV Cache
|
||||
|
||||
To enable quantized KV cache, use the `--kv-cache-dtype` argument when launching the server:
|
||||
|
||||
```bash Command
|
||||
# Enable FP8 E5M2 KV cache
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-R1-0528 \
|
||||
--kv-cache-dtype fp8_e5m2 \
|
||||
|
||||
# Enable FP8 E4M3 KV cache
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-R1-0528 \
|
||||
--kv-cache-dtype fp8_e4m3 \
|
||||
|
||||
# Enable FP4 E2M1 KV cache
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path nvidia/DeepSeek-R1-0528-NVFP4 \
|
||||
--kv-cache-dtype fp4_e2m1 \
|
||||
```
|
||||
|
||||
### Scaling Factors
|
||||
|
||||
FP8 quantization requires scaling factors to properly quantize and dequantize the KV cache.
|
||||
|
||||
<Note>
|
||||
Currently, only per-tensor (scalar) scaling factors are supported.
|
||||
</Note>
|
||||
|
||||
Scaling factors can be:
|
||||
|
||||
- **Loaded from checkpoints**: Pre-quantized models (e.g., ModelOpt) may include `k_scale` and `v_scale` parameters that are automatically loaded
|
||||
- **Provided via JSON**: Supply scaling factors via `--quantization-param-path`.
|
||||
|
||||
The JSON file should follow this format:
|
||||
|
||||
```json Config
|
||||
{
|
||||
"kv_cache": {
|
||||
"dtype": "float8_e4m3fn",
|
||||
"scaling_factor": {
|
||||
"0": {
|
||||
"0": 1.0,
|
||||
"1": 1.0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Where the outer keys in `scaling_factor` are tensor parallel ranks and inner keys are layer indices.
|
||||
|
||||
<Warning>
|
||||
If scaling factors are not provided and not found in the checkpoint, it will default to 1.0, which may cause accuracy issues.
|
||||
</Warning>
|
||||
|
||||
<Tip>
|
||||
**FP4 (MXFP4)**: Unlike FP8, FP4 quantization handles scaling factors automatically on-the-fly during quantization and dequantization. No pre-quantized models or external scaling factor files are required—the block-based scaling factors are computed dynamically as needed.
|
||||
</Tip>
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Memory Savings
|
||||
|
||||
Quantized KV cache provides significant memory savings:
|
||||
- **BF16 → FP4**: Supports approximately 3.56× more tokens than BF16 (accounting for scaling factor overhead)
|
||||
|
||||
<Note>
|
||||
FP4 and FP8 quantization require additional memory for block-based scaling factors, which reduces the effective memory savings compared to the raw bit-width reduction. FP4 with block size 16 supports approximately 1.78× more tokens than FP8, and approximately 3.56× more tokens than BF16. The relative token capacity between FP8 and BF16 can be derived from these ratios.
|
||||
</Note>
|
||||
|
||||
This enables longer context lengths or more concurrent requests within the same memory budget.
|
||||
|
||||
### Accuracy Impact
|
||||
|
||||
#### FP8 Accuracy
|
||||
|
||||
FP8 E4M3 quantization typically introduces minimal accuracy degradation. The impact depends on model architecture, sequence length, and quantization format (generally, E4M3 has better accuracy than E5M2).
|
||||
|
||||
#### FP4 Accuracy
|
||||
|
||||
FP4 (MXFP4) quantization provides significant memory savings with varying accuracy impact depending on model size and dataset complexity. Preliminary accuracy test results from [PR #10078](https://github.com/sgl-project/sglang/pull/10078) (MLA) and [PR #12612](https://github.com/sgl-project/sglang/pull/12612) (MHA) show:
|
||||
|
||||
**Large Models (e.g., Qwen3-235B-A22B, DeepSeek-R1-0528)**
|
||||
|
||||
On large-scale models, FP4 maintains accuracy close to FP8/BF16, especially on simpler datasets:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "20%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Dataset</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>KV16</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>KV8 (FP8 E4M3)</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>KV4 (FP4 E2M1)</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3-235B-A22B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>gsm8k</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.9168</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.9181</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.9186</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3-235B-A22B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>aime25</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.7733</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.7333</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.6000</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3-235B-A22B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>gpqa_diamond</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.7010</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.6899</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.6778</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-R1-0528</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>gsm8k</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.9157</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.9154</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.9124</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-R1-0528</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>aime25</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.5067</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.4934</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.4000</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-R1-0528</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>gpqa_diamond</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.7707</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.7697</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.7273</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Smaller Models (e.g., GPT-OSS-120B)**
|
||||
|
||||
On smaller models, FP4 shows more pronounced accuracy drops, particularly on challenging datasets:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "20%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Dataset</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>KV16</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>KV8 (FP8 E4M3)</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>KV4 (FP4 E2M1)</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GPT-OSS-120B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>gsm8k</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.9161</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.9163</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.9152</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GPT-OSS-120B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>aime25</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.7533</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.7667</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.3533</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GPT-OSS-120B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>gpqa_diamond</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.5081</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.5434</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.3202</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Key Observations:**
|
||||
|
||||
- **Simple datasets (e.g., gsm8k)**: FP4 maintains accuracy close to FP8/BF16 across model sizes
|
||||
- **Model size matters**: Large models (200B+ parameters) generally tolerate FP4 quantization better than smaller models
|
||||
- **Context length**: Accuracy degradation may be more pronounced in long-context scenarios, as the accumulation of the quantization error may become significant.
|
||||
|
||||
<Tip>
|
||||
Evaluate FP4 accuracy on your specific model and workload. Large models on simpler tasks typically show minimal degradation, while smaller models or complex reasoning tasks may require FP8 or BF16 for acceptable accuracy.
|
||||
</Tip>
|
||||
|
||||
## Best Practices
|
||||
|
||||
- **Use pre-quantized models**: Prefer models quantized offline with scaling factors included in the checkpoint.
|
||||
- **Choose the right format**: Use `fp8_e4m3` for better accuracy (recommended), `fp8_e5m2` for larger dynamic range, or `fp4_e2m1` for maximum memory savings (experimental)
|
||||
- **Check backend compatibility**: Verify that your chosen attention backend supports quantized KV cache
|
||||
|
||||
<Note>
|
||||
See also:
|
||||
- [Quantization](./quantization)
|
||||
- [Attention Backend](./attention_backend)
|
||||
- [Server Arguments](./server_arguments)
|
||||
</Note>
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
title: "R-Fork"
|
||||
metatags:
|
||||
description: "SGLang R-Fork: zero-copy GPU-to-GPU weight loading, reduce boot-up time from minutes to seconds. NCCL and TransferEngine backends."
|
||||
---
|
||||
R-Fork (Tensor Remote Fork) is a novel weight loading methodology that leverages efficient inter-node GPU-to-GPU data transfer path to load tensors from a running SGLang instance to a new instance with zero-copy. It can significantly optimize the SGLang instance boot-up time by reducing model weights loading from several minutes to mere seconds.
|
||||
|
||||
To learn more details about R-Fork, please check **[R-Fork blog](https://lmsys.org/blog/2025-12-10-rfork/)**
|
||||
|
||||
## Usage
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "50%"}} />
|
||||
<col style={{width: "50%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Usage</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>load-format</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>set to `remote_instance` to enable R-Fork.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>remote-instance-weight-loader-backend</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`nccl` or `transfer_engine`, default value is `nccl`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>remote-instance-weight-loader-seed-instance-ip</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>IP address of the seed instance who will provide the model weight</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>remote-instance-weight-loader-seed-instance-service-port</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>the port that the seed instance's HTTP server is listening on</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>remote-instance-weight-loader-send-weights-group-ports</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>the list of available ports on the seed instance that will be used to build NCCL communication groups between seed and client instance. This argument is only needed by `nccl` backend.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>remote-instance-weight-loader-start-seed-via-transfer-engine</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>set to start seed service that supports TransferEngine as backend. It is needed for seed instances when using `transfer_engine` as backend.</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
### NCCL as backend
|
||||
|
||||
seed instance:
|
||||
```shell Command
|
||||
python -m sglang.launch_server [args]
|
||||
```
|
||||
|
||||
client instance:
|
||||
```shell Command
|
||||
python -m sglang.launch_server [args] \
|
||||
--load-format remote_instance \
|
||||
--remote-instance-weight-loader-seed-instance-ip [seed_instance_ip] \
|
||||
--remote-instance-weight-loader-seed-instance-service-port [seed_instance_service_port] \
|
||||
--remote-instance-weight-loader-send-weights-group-ports [send_weights_nccl_group_ports_list] \
|
||||
--remote-instance-weight-loader-backend nccl
|
||||
```
|
||||
|
||||
### TransferEngine as backend
|
||||
|
||||
seed instance:
|
||||
```shell Command
|
||||
python -m sglang.launch_server [args] \
|
||||
--remote-instance-weight-loader-start-seed-via-transfer-engine
|
||||
```
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server [args] \
|
||||
--load-format remote_instance \
|
||||
--remote-instance-weight-loader-seed-instance-ip [seed_instance_ip] \
|
||||
--remote-instance-weight-loader-seed-instance-service-port [seed_instance_service_port] \
|
||||
--remote-instance-weight-loader-backend transfer_engine
|
||||
```
|
||||
@@ -0,0 +1,377 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Reasoning Parser\n",
|
||||
"\n",
|
||||
"SGLang supports parsing reasoning content out from \"normal\" content for reasoning models such as [DeepSeek R1](https://huggingface.co/deepseek-ai/DeepSeek-R1).\n",
|
||||
"\n",
|
||||
"## Supported Models & Parsers\n",
|
||||
"\n",
|
||||
"| Model | Reasoning tags | Parser | Notes |\n",
|
||||
"|---------|-----------------------------|------------------|-------|\n",
|
||||
"| [DeepSeek‑R1 series](https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d) | `<think>` … `</think>` | `deepseek-r1` | Supports all variants (R1, R1-0528, R1-Distill) |\n",
|
||||
"| [DeepSeek‑V3 series](https://huggingface.co/deepseek-ai/DeepSeek-V3.1) | `<think>` … `</think>` | `deepseek-v3` | Including [DeepSeek‑V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp). Supports `thinking` parameter |\n",
|
||||
"| [Standard Qwen3 models](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f) | `<think>` … `</think>` | `qwen3` | Supports `enable_thinking` parameter |\n",
|
||||
"| [Qwen3-Thinking models](https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507) | `<think>` … `</think>` | `qwen3` or `qwen3-thinking` | Always generates thinking content |\n",
|
||||
"| [Kimi K2 Thinking](https://huggingface.co/moonshotai/Kimi-K2-Thinking) | `◁think▷` … `◁/think▷` | `kimi_k2` | Uses special thinking delimiters. Also requires `--tool-call-parser kimi_k2` for tool use. |\n",
|
||||
"| [GPT OSS](https://huggingface.co/openai/gpt-oss-120b) | `<\\|channel\\|>analysis<\\|message\\|>` … `<\\|end\\|>` | `gpt-oss` | N/A |\n",
|
||||
"### Model-Specific Behaviors\n",
|
||||
"\n",
|
||||
"**DeepSeek-R1 Family:**\n",
|
||||
"- DeepSeek-R1: No `<think>` start tag, jumps directly to thinking content\n",
|
||||
"- DeepSeek-R1-0528: Generates both `<think>` start and `</think>` end tags\n",
|
||||
"- Both are handled by the same `deepseek-r1` parser\n",
|
||||
"\n",
|
||||
"**DeepSeek-V3 Family:**\n",
|
||||
"- DeepSeek-V3.1/V3.2: Hybrid model supporting both thinking and non-thinking modes, use the `deepseek-v3` parser and `thinking` parameter (NOTE: not `enable_thinking`)\n",
|
||||
"\n",
|
||||
"**Qwen3 Family:**\n",
|
||||
"- Standard Qwen3 (e.g., Qwen3-2507): Use `qwen3` parser, supports `enable_thinking` in chat templates\n",
|
||||
"- Qwen3-Thinking (e.g., Qwen3-235B-A22B-Thinking-2507): Use `qwen3` or `qwen3-thinking` parser, always thinks\n",
|
||||
"\n",
|
||||
"**Kimi K2:**\n",
|
||||
"- Kimi K2 Thinking: Uses special `◁think▷` and `◁/think▷` tags. For agentic tool use, also specify `--tool-call-parser kimi_k2`.\n",
|
||||
"\n",
|
||||
"**GPT OSS:**\n",
|
||||
"- GPT OSS: Uses special `<|channel|>analysis<|message|>` and `<|end|>` tags"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Usage\n",
|
||||
"\n",
|
||||
"### Launching the Server"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Specify the `--reasoning-parser` option."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import requests\n",
|
||||
"from openai import OpenAI\n",
|
||||
"from sglang.test.doc_patch import launch_server_cmd\n",
|
||||
"from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
|
||||
"\n",
|
||||
"server_process, port = launch_server_cmd(\n",
|
||||
" \"python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --host 0.0.0.0 --reasoning-parser deepseek-r1 --log-level warning\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"wait_for_server(f\"http://localhost:{port}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Note that `--reasoning-parser` defines the parser used to interpret responses."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### OpenAI Compatible API\n",
|
||||
"\n",
|
||||
"Using the OpenAI compatible API, the contract follows the [DeepSeek API design](https://api-docs.deepseek.com/guides/reasoning_model) established with the release of DeepSeek-R1:\n",
|
||||
"\n",
|
||||
"- `reasoning_content`: The content of the CoT.\n",
|
||||
"- `content`: The content of the final answer."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Initialize OpenAI-like client\n",
|
||||
"client = OpenAI(api_key=\"None\", base_url=f\"http://0.0.0.0:{port}/v1\")\n",
|
||||
"model_name = client.models.list().data[0].id\n",
|
||||
"\n",
|
||||
"messages = [\n",
|
||||
" {\n",
|
||||
" \"role\": \"user\",\n",
|
||||
" \"content\": \"What is 1+3?\",\n",
|
||||
" }\n",
|
||||
"]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Non-Streaming Request"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"response_non_stream = client.chat.completions.create(\n",
|
||||
" model=model_name,\n",
|
||||
" messages=messages,\n",
|
||||
" temperature=0.6,\n",
|
||||
" top_p=0.95,\n",
|
||||
" stream=False, # Non-streaming\n",
|
||||
" extra_body={\"separate_reasoning\": True},\n",
|
||||
")\n",
|
||||
"print_highlight(\"==== Reasoning ====\")\n",
|
||||
"print_highlight(response_non_stream.choices[0].message.reasoning_content)\n",
|
||||
"\n",
|
||||
"print_highlight(\"==== Text ====\")\n",
|
||||
"print_highlight(response_non_stream.choices[0].message.content)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Streaming Request"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"response_stream = client.chat.completions.create(\n",
|
||||
" model=model_name,\n",
|
||||
" messages=messages,\n",
|
||||
" temperature=0.6,\n",
|
||||
" top_p=0.95,\n",
|
||||
" stream=True, # Non-streaming\n",
|
||||
" extra_body={\"separate_reasoning\": True},\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"reasoning_content = \"\"\n",
|
||||
"content = \"\"\n",
|
||||
"for chunk in response_stream:\n",
|
||||
" if chunk.choices[0].delta.content:\n",
|
||||
" content += chunk.choices[0].delta.content\n",
|
||||
" if chunk.choices[0].delta.reasoning_content:\n",
|
||||
" reasoning_content += chunk.choices[0].delta.reasoning_content\n",
|
||||
"\n",
|
||||
"print_highlight(\"==== Reasoning ====\")\n",
|
||||
"print_highlight(reasoning_content)\n",
|
||||
"\n",
|
||||
"print_highlight(\"==== Text ====\")\n",
|
||||
"print_highlight(content)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Optionally, you can buffer the reasoning content to the last reasoning chunk (or the first chunk after the reasoning content)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"response_stream = client.chat.completions.create(\n",
|
||||
" model=model_name,\n",
|
||||
" messages=messages,\n",
|
||||
" temperature=0.6,\n",
|
||||
" top_p=0.95,\n",
|
||||
" stream=True, # Non-streaming\n",
|
||||
" extra_body={\"separate_reasoning\": True, \"stream_reasoning\": False},\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"reasoning_content = \"\"\n",
|
||||
"content = \"\"\n",
|
||||
"for chunk in response_stream:\n",
|
||||
" if chunk.choices[0].delta.content:\n",
|
||||
" content += chunk.choices[0].delta.content\n",
|
||||
" if chunk.choices[0].delta.reasoning_content:\n",
|
||||
" reasoning_content += chunk.choices[0].delta.reasoning_content\n",
|
||||
"\n",
|
||||
"print_highlight(\"==== Reasoning ====\")\n",
|
||||
"print_highlight(reasoning_content)\n",
|
||||
"\n",
|
||||
"print_highlight(\"==== Text ====\")\n",
|
||||
"print_highlight(content)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The reasoning separation is enable by default when specify . \n",
|
||||
"**To disable it, set the `separate_reasoning` option to `False` in request.**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"response_non_stream = client.chat.completions.create(\n",
|
||||
" model=model_name,\n",
|
||||
" messages=messages,\n",
|
||||
" temperature=0.6,\n",
|
||||
" top_p=0.95,\n",
|
||||
" stream=False, # Non-streaming\n",
|
||||
" extra_body={\"separate_reasoning\": False},\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(\"==== Original Output ====\")\n",
|
||||
"print_highlight(response_non_stream.choices[0].message.content)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### SGLang Native API "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from transformers import AutoTokenizer\n",
|
||||
"\n",
|
||||
"tokenizer = AutoTokenizer.from_pretrained(\"deepseek-ai/DeepSeek-R1-Distill-Qwen-7B\")\n",
|
||||
"input = tokenizer.apply_chat_template(\n",
|
||||
" messages, tokenize=False, add_generation_prompt=True, return_dict=False\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"gen_url = f\"http://localhost:{port}/generate\"\n",
|
||||
"gen_data = {\n",
|
||||
" \"text\": input,\n",
|
||||
" \"sampling_params\": {\n",
|
||||
" \"skip_special_tokens\": False,\n",
|
||||
" \"max_new_tokens\": 1024,\n",
|
||||
" \"temperature\": 0.6,\n",
|
||||
" \"top_p\": 0.95,\n",
|
||||
" },\n",
|
||||
"}\n",
|
||||
"gen_response = requests.post(gen_url, json=gen_data).json()[\"text\"]\n",
|
||||
"\n",
|
||||
"print_highlight(\"==== Original Output ====\")\n",
|
||||
"print_highlight(gen_response)\n",
|
||||
"\n",
|
||||
"parse_url = f\"http://localhost:{port}/separate_reasoning\"\n",
|
||||
"separate_reasoning_data = {\n",
|
||||
" \"text\": gen_response,\n",
|
||||
" \"reasoning_parser\": \"deepseek-r1\",\n",
|
||||
"}\n",
|
||||
"separate_reasoning_response_json = requests.post(\n",
|
||||
" parse_url, json=separate_reasoning_data\n",
|
||||
").json()\n",
|
||||
"print_highlight(\"==== Reasoning ====\")\n",
|
||||
"print_highlight(separate_reasoning_response_json[\"reasoning_text\"])\n",
|
||||
"print_highlight(\"==== Text ====\")\n",
|
||||
"print_highlight(separate_reasoning_response_json[\"text\"])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(server_process)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Offline Engine API"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sglang as sgl\n",
|
||||
"from sglang.srt.parser.reasoning_parser import ReasoningParser\n",
|
||||
"from sglang.utils import print_highlight\n",
|
||||
"\n",
|
||||
"llm = sgl.Engine(model_path=\"deepseek-ai/DeepSeek-R1-Distill-Qwen-7B\")\n",
|
||||
"tokenizer = AutoTokenizer.from_pretrained(\"deepseek-ai/DeepSeek-R1-Distill-Qwen-7B\")\n",
|
||||
"input = tokenizer.apply_chat_template(\n",
|
||||
" messages, tokenize=False, add_generation_prompt=True, return_dict=False\n",
|
||||
")\n",
|
||||
"sampling_params = {\n",
|
||||
" \"max_new_tokens\": 1024,\n",
|
||||
" \"skip_special_tokens\": False,\n",
|
||||
" \"temperature\": 0.6,\n",
|
||||
" \"top_p\": 0.95,\n",
|
||||
"}\n",
|
||||
"result = llm.generate(prompt=input, sampling_params=sampling_params)\n",
|
||||
"\n",
|
||||
"generated_text = result[\"text\"] # Assume there is only one prompt\n",
|
||||
"\n",
|
||||
"print_highlight(\"==== Original Output ====\")\n",
|
||||
"print_highlight(generated_text)\n",
|
||||
"\n",
|
||||
"parser = ReasoningParser(\"deepseek-r1\")\n",
|
||||
"reasoning_text, text = parser.parse_non_stream(generated_text)\n",
|
||||
"print_highlight(\"==== Reasoning ====\")\n",
|
||||
"print_highlight(reasoning_text)\n",
|
||||
"print_highlight(\"==== Text ====\")\n",
|
||||
"print_highlight(text)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"llm.shutdown()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Supporting New Reasoning Model Schemas\n",
|
||||
"\n",
|
||||
"For future reasoning models, you can implement the reasoning parser as a subclass of `BaseReasoningFormatDetector` in `python/sglang/srt/reasoning_parser.py` and specify the reasoning parser for new reasoning model schemas accordingly."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
@@ -0,0 +1,317 @@
|
||||
---
|
||||
title: "Reasoning Parser"
|
||||
metatags:
|
||||
description: "SGLang reasoning parser: separate thinking content from output for DeepSeek R1, Qwen3, Kimi K2, GPT-OSS reasoning models."
|
||||
---
|
||||
SGLang supports parsing reasoning content out from "normal" content for reasoning models such as [DeepSeek R1](https://huggingface.co/deepseek-ai/DeepSeek-R1).
|
||||
|
||||
## Supported Models & Parsers
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Reasoning tags</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parser</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Notes</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[DeepSeek‑R1 series](https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`<think>` … `</think>`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`deepseek-r1`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Supports all variants (R1, R1-0528, R1-Distill)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[DeepSeek‑V3 series](https://huggingface.co/deepseek-ai/DeepSeek-V3.1)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`<think>` … `</think>`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`deepseek-v3`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Including [DeepSeek‑V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp). Supports `thinking` parameter</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[Standard Qwen3 models](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`<think>` … `</think>`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`qwen3`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Supports `enable_thinking` parameter</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[Qwen3-Thinking models](https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`<think>` … `</think>`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`qwen3` or `qwen3-thinking`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Always generates thinking content</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[Kimi K2 Thinking](https://huggingface.co/moonshotai/Kimi-K2-Thinking)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`◁think▷` … `◁/think▷`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`kimi_k2`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Uses special thinking delimiters. Also requires `--tool-call-parser kimi_k2` for tool use.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[GPT OSS](https://huggingface.co/openai/gpt-oss-120b)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`<|channel|>analysis<|message|>` … `<|end|>`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`gpt-oss`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
### Model-Specific Behaviors
|
||||
|
||||
**DeepSeek-R1 Family:**
|
||||
- DeepSeek-R1: No `<think>` start tag, jumps directly to thinking content
|
||||
- DeepSeek-R1-0528: Generates both `<think>` start and `</think>` end tags
|
||||
- Both are handled by the same `deepseek-r1` parser
|
||||
|
||||
**DeepSeek-V3 Family:**
|
||||
- DeepSeek-V3.1/V3.2: Hybrid model supporting both thinking and non-thinking modes, use the `deepseek-v3` parser and `thinking` parameter (NOTE: not `enable_thinking`)
|
||||
|
||||
**Qwen3 Family:**
|
||||
- Standard Qwen3 (e.g., Qwen3-2507): Use `qwen3` parser, supports `enable_thinking` in chat templates
|
||||
- Qwen3-Thinking (e.g., Qwen3-235B-A22B-Thinking-2507): Use `qwen3` or `qwen3-thinking` parser, always thinks
|
||||
|
||||
**Kimi K2:**
|
||||
- Kimi K2 Thinking: Uses special `◁think▷` and `◁/think▷` tags. For agentic tool use, also specify `--tool-call-parser kimi_k2`.
|
||||
|
||||
**GPT OSS:**
|
||||
- GPT OSS: Uses special `<|channel|>analysis<|message|>` and `<|end|>` tags
|
||||
|
||||
|
||||
## Usage
|
||||
|
||||
### Launching the Server
|
||||
|
||||
|
||||
Specify the `--reasoning-parser` option.
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
from openai import OpenAI
|
||||
from sglang.test.doc_patch import launch_server_cmd
|
||||
from sglang.utils import wait_for_server, print_highlight, terminate_process
|
||||
|
||||
server_process, port = launch_server_cmd(
|
||||
"python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --host 0.0.0.0 --reasoning-parser deepseek-r1 --log-level warning"
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
```
|
||||
|
||||
Note that `--reasoning-parser` defines the parser used to interpret responses.
|
||||
|
||||
|
||||
### OpenAI Compatible API
|
||||
|
||||
Using the OpenAI compatible API, the contract follows the [DeepSeek API design](https://api-docs.deepseek.com/guides/reasoning_model) established with the release of DeepSeek-R1:
|
||||
|
||||
- `reasoning_content`: The content of the CoT.
|
||||
- `content`: The content of the final answer.
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
# Initialize OpenAI-like client
|
||||
client = OpenAI(api_key="None", base_url=f"http://0.0.0.0:{port}/v1")
|
||||
model_name = client.models.list().data[0].id
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "What is 1+3?",
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
#### Non-Streaming Request
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
response_non_stream = client.chat.completions.create(
|
||||
model=model_name,
|
||||
messages=messages,
|
||||
temperature=0.6,
|
||||
top_p=0.95,
|
||||
stream=False, # Non-streaming
|
||||
extra_body={"separate_reasoning": True},
|
||||
)
|
||||
print_highlight("==== Reasoning ====")
|
||||
print_highlight(response_non_stream.choices[0].message.reasoning_content)
|
||||
|
||||
print_highlight("==== Text ====")
|
||||
print_highlight(response_non_stream.choices[0].message.content)
|
||||
```
|
||||
|
||||
#### Streaming Request
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
response_stream = client.chat.completions.create(
|
||||
model=model_name,
|
||||
messages=messages,
|
||||
temperature=0.6,
|
||||
top_p=0.95,
|
||||
stream=True, # Non-streaming
|
||||
extra_body={"separate_reasoning": True},
|
||||
)
|
||||
|
||||
reasoning_content = ""
|
||||
content = ""
|
||||
for chunk in response_stream:
|
||||
if chunk.choices[0].delta.content:
|
||||
content += chunk.choices[0].delta.content
|
||||
if chunk.choices[0].delta.reasoning_content:
|
||||
reasoning_content += chunk.choices[0].delta.reasoning_content
|
||||
|
||||
print_highlight("==== Reasoning ====")
|
||||
print_highlight(reasoning_content)
|
||||
|
||||
print_highlight("==== Text ====")
|
||||
print_highlight(content)
|
||||
```
|
||||
|
||||
Optionally, you can buffer the reasoning content to the last reasoning chunk (or the first chunk after the reasoning content).
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
response_stream = client.chat.completions.create(
|
||||
model=model_name,
|
||||
messages=messages,
|
||||
temperature=0.6,
|
||||
top_p=0.95,
|
||||
stream=True, # Non-streaming
|
||||
extra_body={"separate_reasoning": True, "stream_reasoning": False},
|
||||
)
|
||||
|
||||
reasoning_content = ""
|
||||
content = ""
|
||||
for chunk in response_stream:
|
||||
if chunk.choices[0].delta.content:
|
||||
content += chunk.choices[0].delta.content
|
||||
if chunk.choices[0].delta.reasoning_content:
|
||||
reasoning_content += chunk.choices[0].delta.reasoning_content
|
||||
|
||||
print_highlight("==== Reasoning ====")
|
||||
print_highlight(reasoning_content)
|
||||
|
||||
print_highlight("==== Text ====")
|
||||
print_highlight(content)
|
||||
```
|
||||
|
||||
The reasoning separation is enable by default when specify .
|
||||
**To disable it, set the `separate_reasoning` option to `False` in request.**
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
response_non_stream = client.chat.completions.create(
|
||||
model=model_name,
|
||||
messages=messages,
|
||||
temperature=0.6,
|
||||
top_p=0.95,
|
||||
stream=False, # Non-streaming
|
||||
extra_body={"separate_reasoning": False},
|
||||
)
|
||||
|
||||
print_highlight("==== Original Output ====")
|
||||
print_highlight(response_non_stream.choices[0].message.content)
|
||||
```
|
||||
|
||||
### SGLang Native API
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-7B")
|
||||
input = tokenizer.apply_chat_template(
|
||||
messages, tokenize=False, add_generation_prompt=True, return_dict=False
|
||||
)
|
||||
|
||||
gen_url = f"http://localhost:{port}/generate"
|
||||
gen_data = {
|
||||
"text": input,
|
||||
"sampling_params": {
|
||||
"skip_special_tokens": False,
|
||||
"max_new_tokens": 1024,
|
||||
"temperature": 0.6,
|
||||
"top_p": 0.95,
|
||||
},
|
||||
}
|
||||
gen_response = requests.post(gen_url, json=gen_data).json()["text"]
|
||||
|
||||
print_highlight("==== Original Output ====")
|
||||
print_highlight(gen_response)
|
||||
|
||||
parse_url = f"http://localhost:{port}/separate_reasoning"
|
||||
separate_reasoning_data = {
|
||||
"text": gen_response,
|
||||
"reasoning_parser": "deepseek-r1",
|
||||
}
|
||||
separate_reasoning_response_json = requests.post(
|
||||
parse_url, json=separate_reasoning_data
|
||||
).json()
|
||||
print_highlight("==== Reasoning ====")
|
||||
print_highlight(separate_reasoning_response_json["reasoning_text"])
|
||||
print_highlight("==== Text ====")
|
||||
print_highlight(separate_reasoning_response_json["text"])
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
terminate_process(server_process)
|
||||
```
|
||||
|
||||
### Offline Engine API
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import sglang as sgl
|
||||
from sglang.srt.parser.reasoning_parser import ReasoningParser
|
||||
from sglang.utils import print_highlight
|
||||
|
||||
llm = sgl.Engine(model_path="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B")
|
||||
tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-7B")
|
||||
input = tokenizer.apply_chat_template(
|
||||
messages, tokenize=False, add_generation_prompt=True, return_dict=False
|
||||
)
|
||||
sampling_params = {
|
||||
"max_new_tokens": 1024,
|
||||
"skip_special_tokens": False,
|
||||
"temperature": 0.6,
|
||||
"top_p": 0.95,
|
||||
}
|
||||
result = llm.generate(prompt=input, sampling_params=sampling_params)
|
||||
|
||||
generated_text = result["text"] # Assume there is only one prompt
|
||||
|
||||
print_highlight("==== Original Output ====")
|
||||
print_highlight(generated_text)
|
||||
|
||||
parser = ReasoningParser("deepseek-r1")
|
||||
reasoning_text, text = parser.parse_non_stream(generated_text)
|
||||
print_highlight("==== Reasoning ====")
|
||||
print_highlight(reasoning_text)
|
||||
print_highlight("==== Text ====")
|
||||
print_highlight(text)
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
llm.shutdown()
|
||||
```
|
||||
|
||||
## Supporting New Reasoning Model Schemas
|
||||
|
||||
For future reasoning models, you can implement the reasoning parser as a subclass of `BaseReasoningFormatDetector` in `python/sglang/srt/reasoning_parser.py` and specify the reasoning parser for new reasoning model schemas accordingly.
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,575 @@
|
||||
---
|
||||
title: "SGLang for RL Systems"
|
||||
metatags:
|
||||
description: "SGLang for RL: engine sleep/wake, weight refit, partial rollout, deterministic inference, cache-aware load balancing for RLHF."
|
||||
---
|
||||
This document is a practical guide for infrastructure teams integrating SGLang into RL and post-training systems. It focuses on the operational pain points in the loop (rollout, evaluation, training, weight sync) and maps them to concrete SGLang APIs, flags, and integration patterns. The focus is on maximizing rollout efficiency, accuracy and stability while keeping rollout-serving behavior aligned in production environments.
|
||||
|
||||
## Why SGLang for RL Lifecycle?
|
||||
|
||||
Let's embrace a guiding principle from early DeepMind's RL engineering:
|
||||
|
||||
**Be a library, not a framework.**
|
||||
|
||||
This philosophy empowers innovation by providing SGLang as flexible tools, not rigid structures. Here are five reasons to use SGLang for your RL lifecycle:
|
||||
|
||||
* **Fine-Grained Engine Sleep and Wake Up**: facilitate maximum-powered rollout and training
|
||||
* **Open-To-Use Refit Functionality**: diverse methods for co-location or disaggregation
|
||||
* **Easy To Postpone Generation**: enable partial rollout and dedicated rollout control
|
||||
* **Deterministic Inference**: achieve deterministic inference to enable zero training-inference mismatch
|
||||
* **Load Balancing Router**: cache-aware load-balancing for high-throughput rollout
|
||||
|
||||
The following sections cover these aspects in detail.
|
||||
|
||||
## Fine-Grained Engine Sleep and Wake Up
|
||||
|
||||
Rollout and training are both memory-intensive, and co-locating them on the same GPUs often leads to memory pressure and slow handoffs. SGLang provides a memory-aware sleep/wake mechanism that releases KV cache and weights while keeping the server process alive, then resumes them for rollout without a full restart. This avoids repeated disk I/O and CUDA graph recapture during each RL step.
|
||||
|
||||
Under the hood, the RL team uses CUDA-graph-aware weight offload via [torch_memory_saver](https://github.com/fzyzcjy/torch_memory_saver) to preserve virtual memory addresses for graph replay. For details, see: [Efficient RL Training - Optimizing Memory Usage in verl](https://hebiao064.github.io/rl-memory-management).
|
||||
|
||||
### Server flag
|
||||
|
||||
Enable memory saver support when launching the server:
|
||||
|
||||
```text Output
|
||||
--enable-memory-saver
|
||||
```
|
||||
|
||||
### Release Memory
|
||||
|
||||
**Endpoint:** `POST /release_memory_occupation`
|
||||
|
||||
**Request body:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Field</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Defaults</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Options</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`tags`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Which memory regions to release. If omitted, all are released.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: list[str], values: `kv_cache`, `weights`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
{/* python/sglang/srt/managers/io_struct.py#L1381 currently only supports `kv_cache`, `weights` */}
|
||||
**Behavior notes:**
|
||||
|
||||
- This call asserts there are no ongoing requests. Ensure the engine is idle before calling it.
|
||||
- If `kv_cache` is released, SGLang flushes cache; subsequent requests will rebuild KV cache as needed.
|
||||
|
||||
### Resume Memory
|
||||
|
||||
**Endpoint:** `POST /resume_memory_occupation`
|
||||
|
||||
**Request body:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Field</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Defaults</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Options</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`tags`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Which memory regions to resume. If omitted, all are resumed.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: list[str], values: `kv_cache`, `weights`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
{/* python/sglang/srt/managers/io_struct.py#L1393 currently only supports `kv_cache`, `weights` */}
|
||||
|
||||
## Open-To-Use Refit Functionality
|
||||
|
||||
After training completes each step, rollout engines must be refit with new weights. SGLang supports three refit strategies so you can match your infrastructure style (co-located vs disaggregated) and scaling needs. Each strategy maps to a concrete API with clear request schemas. For a deeper dive into SGLang's weight update utilities, see [RL System Deep Thinking: Weight Update Mechanisms](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/blob/main/rlhf/sys-design/readme-1-EN).
|
||||
|
||||
**How to choose:**
|
||||
|
||||
- **From disk** is simplest and best for elastic rollout scaling and checkpointing.
|
||||
- **From tensor** is best for co-located training/rollout when you can pass in-memory tensors.
|
||||
- **From distributed** is best for disaggregated training/rollout with dedicated communication groups (NCCL/IB).
|
||||
|
||||
### Update Weights from Disk
|
||||
|
||||
**When to use:**
|
||||
|
||||
- Save checkpoint to disk and update weights from disk
|
||||
- Dynamic scaling (new rollout instances can load from the same checkpoint)
|
||||
|
||||
**Why it works well:**
|
||||
|
||||
This path trades some I/O overhead for simplicity and flexibility. It integrates naturally with checkpointing and makes it trivial to add new rollout engines: point them at the same checkpoint and call the API. It is also the safest option for high availability because the checkpoint itself is the source of truth.
|
||||
|
||||
**Endpoint:** `POST /update_weights_from_disk`
|
||||
|
||||
**Request body:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Field</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Defaults</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Options</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`model_path`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The model path with the new weights.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Required</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`load_format`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The format to load the weights.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`abort_all_requests`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Abort all running requests before update.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`weight_version`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Optional weight version label tracked by the server.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`is_async`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Perform weight load asynchronously.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`torch_empty_cache`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Empty torch cache.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`keep_pause`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Keep scheduler paused after update.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`recapture_cuda_graph`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Recapture CUDA graphs after update.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`token_step`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Trainer step id for rollout bookkeeping.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`0`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`flush_cache`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Flush KV cache after update.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`True`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Response body:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Field</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Defaults</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Options</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`success`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Whether the update succeeded.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>-</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`message`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Status / error message.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>-</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`num_paused_requests`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of paused requests during update.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`0`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Python Engine API:** `engine.update_weights_from_disk(model_path, load_format=None)`
|
||||
|
||||
### Update Weights from Tensor
|
||||
|
||||
**When to use:**
|
||||
|
||||
- Co-located training and rollout, where training can provide tensors directly
|
||||
- Fast in-memory updates
|
||||
|
||||
**Important constraints:**
|
||||
|
||||
This strategy requires the training process and rollout engine to share access to the tensors. Co-located setups must keep the model on GPU; moving tensors to CPU will break the update path. For high-performance MoE or specialized attention kernels, co-location may limit some optimizations compared to disaggregated rollouts.
|
||||
|
||||
**Endpoint:** `POST /update_weights_from_tensor`
|
||||
|
||||
**Request body:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Field</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Defaults</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Options</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`serialized_named_tensors`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Per-TP serialized tensor payloads.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Required</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: list[str</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`load_format`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Optional load format selector.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`, `direct`, `flattened_bucket`, or a custom loader path string</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`flush_cache`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Flush KV cache after update.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`True`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`abort_all_requests`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Abort all running requests before update.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`weight_version`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Optional version label tracked by the server.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Note:** The serialized tensor payloads must be created with `MultiprocessingSerializer.serialize(...)` and should be base64-safe strings.
|
||||
|
||||
**Python Engine API:** `engine.update_weights_from_tensor(named_tensors, load_format=None, flush_cache=True)`
|
||||
|
||||
### Update Weights from Distributed Group
|
||||
|
||||
**When to use:**
|
||||
|
||||
- Disaggregated training and rollout
|
||||
- NCCL or IB-backed weight broadcast from training workers to rollout workers
|
||||
|
||||
**How it works:**
|
||||
|
||||
Training workers gather weights (typically on TP rank 0), broadcast them to the rollout group, and each rollout TP shard loads the parameters it needs. This avoids disk I/O and keeps training and rollout decoupled, at the cost of managing a dedicated communication group.
|
||||
|
||||
**Initialize weight update group**
|
||||
|
||||
**Endpoint:** `POST /init_weights_update_group`
|
||||
|
||||
**Request body:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Field</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Defaults</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Options</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`master_address`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Group master address.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Required</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`master_port`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Group master port.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Required</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`rank_offset`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Offset for local rank mapping.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Required</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`world_size`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Total world size.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Required</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`group_name`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Group name.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`weight_update_group`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`backend`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Communication backend.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`nccl`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Update weight**
|
||||
|
||||
**Endpoint:** `POST /update_weights_from_distributed`
|
||||
|
||||
**Request body:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Field</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Defaults</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Options</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`names`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Parameter names to update.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Required</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: list[str]</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`dtypes`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Dtype strings for each parameter.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Required</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: list[str]</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`shapes`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Tensor shapes.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Required</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: list[list[int]]</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`group_name`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Group name.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`weight_update_group`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`flush_cache`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Flush KV cache after update.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`True`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`abort_all_requests`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Abort all running requests before update.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`weight_version`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Optional version label.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`load_format`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Optional format selector.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None` or `flattened_bucket`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Destroy weights update group**
|
||||
|
||||
**Endpoint:** `POST /destroy_weights_update_group`
|
||||
|
||||
**Request body:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Field</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Defaults</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Options</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`group_name`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Group name.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`weight_update_group`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Python Engine APIs:**
|
||||
|
||||
- `engine.init_weights_update_group(...)`
|
||||
- `engine.update_weights_from_distributed(names, dtypes, shapes, ...)`
|
||||
- `engine.destroy_weights_update_group(group_name)`
|
||||
|
||||
## Easy To Postpone Generation
|
||||
|
||||
Multi-turn RL rollouts often suffer from long-tail requests that block the entire batch. A small number of slow interactions can stall all GPUs, and the long-tail behavior makes profiling and monitoring difficult.
|
||||
|
||||
SGLang exposes explicit pause/resume APIs so you can pause slow requests and continue them later. This pattern matches systems like [APRIL](https://arxiv.org/abs/2509.18521), terminate once enough responses are collected, and recycle incomplete responses in the next step. The result is higher GPU utilization without discarding partial work.
|
||||
|
||||
`pause_generation` --- update weights --- `continue_generation` is the correct execution flow when updating weights from training. An update can only happen when SGLang is not actively processing inference tasks.
|
||||
|
||||
### Pause Generation
|
||||
|
||||
**Endpoint:** `POST /pause_generation`
|
||||
|
||||
**Request body:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Field</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Defaults</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Options</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`mode`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Pause mode.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`abort`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`abort`, `retract`, `in_place`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Modes:**
|
||||
|
||||
- `abort`: Default behavior, identical to `abort` endpoint with `abort_all` set. Pending requests from `waiting_queue` and `running_queue` will be returned immediately to the caller.
|
||||
- `retract`: Put engine in "paused" state. Move running requests back to waiting queue. KV cache can be flushed and recomputed later.
|
||||
- `in_place`: Put engine in "paused" state without changing states of the requests. Running requests rely on availability of KV caches to continue, so any subsequent `flush_cache` call will be unsuccessful.
|
||||
|
||||
### Continue Generation
|
||||
|
||||
**Endpoint:** `POST /continue_generation`
|
||||
|
||||
## Deterministic Inference
|
||||
|
||||
In many RL stacks, rollout and training are implemented with different kernels or batching behavior. Even when weights are identical, token probabilities can drift, silently breaking the on-policy assumption. This is the training–inference mismatch problem.
|
||||
|
||||
SGLang supports a deterministic inference mode that reduces non-determinism across batch shapes. This mitigates variance introduced by runtime batching and kernel selection. To further achieve true on-policy training, you need to modify the training engine to use the same deterministic kernels. For implementation details, see these miles examples: [True On-Policy](https://github.com/radixark/miles/tree/main/examples/true_on_policy) and [True On-Policy for VLM](https://github.com/radixark/miles/tree/main/examples/true_on_policy_vlm). For additional context, see the blog post [Let Speed Be With Stability: All-In-One Solution to Training-Inference Mismatch with Miles](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/blob/main/rlhf/slime/mismatch/blog-en).
|
||||
|
||||
**Server flag:**
|
||||
|
||||
```text Output
|
||||
--enable-deterministic-inference
|
||||
```
|
||||
|
||||
For more details, see [Deterministic Inference](./deterministic_inference)
|
||||
|
||||
## Load Balancing Router
|
||||
|
||||
SGLang Model Gateway is the recommended control plane for large‑scale RL rollouts. It provides async, non‑blocking request handling, cache‑aware load balancing, and fault‑tolerant routing across rollout and reward servers. This lets you keep GPUs saturated while avoiding long‑tail stalls and brittle, engine‑local concurrency logic. It has been deployed in the training of GLM 4.5+ models and proven to be highly efficient in production-level large-scale RL workloads.
|
||||
|
||||
Key benefits for RL infrastructure:
|
||||
|
||||
- **Async non-blocking efficiency**: SGLang’s native async server/router architecture (HTTPS/gRPC) manages concurrency automatically. This guarantees maximum GPU saturation and effective continuous batching without requiring complex, manual implementation by engineers.
|
||||
- **Elasticity and fault tolerance**: By encapsulating the reward model and rollout as independent servers, SGLang decouples them logically and physically. This architecture provides robust disaster recovery for large-scale distributed training; if a server fails, the router automatically redirects traffic to healthy nodes, ensuring the training process continues without interruption.
|
||||
- **Training–Inference alignment**: Using the SGLang Model Gateway for both training and inference ensures "What You See Is What You Get." This eliminates score discrepancies and the painful backend alignment issues often caused by using different engines for training versus deployment.
|
||||
- **Dynamic load balancing and long-tail mitigation**: Unlike static partitioning, the SGLang Model Gateway enables request-level dynamic dispatching for multi-turn RL. It can distribute different turns of a conversation across different servers to balance workloads and eliminate long-tail latency caused by varying sequence lengths.
|
||||
|
||||
For deployment and configuration, see: [SGLang Model Gateway](./sgl_model_gateway)
|
||||
@@ -0,0 +1,360 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Speculative Decoding\n",
|
||||
"\n",
|
||||
"SGLang now provides an EAGLE-based (EAGLE-2/EAGLE-3) speculative decoding option. Our implementation aims to maximize speed and efficiency and is considered to be among the fastest in open-source LLM engines.\n",
|
||||
"\n",
|
||||
"### Performance Highlights\n",
|
||||
"\n",
|
||||
"Please see below for the huge improvements on throughput for LLaMA-Instruct 3.1 8B tested on MT bench that can be achieved via EAGLE3 decoding.\n",
|
||||
"For further details please see the [EAGLE3 paper](https://arxiv.org/pdf/2503.01840).\n",
|
||||
"\n",
|
||||
"| Method | Throughput (tokens/s) |\n",
|
||||
"|--------|----------------|\n",
|
||||
"| SGLang (w/o speculative, 1x H100) | 158.34 tokens/s |\n",
|
||||
"| SGLang + EAGLE-2 (1x H100) | 244.10 tokens/s |\n",
|
||||
"| SGLang + EAGLE-3 (1x H100) | 373.25 tokens/s |"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## EAGLE Decoding\n",
|
||||
"\n",
|
||||
"To enable EAGLE speculative decoding the following parameters are relevant:\n",
|
||||
"* `speculative_draft_model_path`: Specifies draft model. This parameter is required.\n",
|
||||
"* `speculative_num_steps`: Depth of autoregressive drafting. Increases speculation range but risks rejection cascades. Default is 5.\n",
|
||||
"* `speculative_eagle_topk`: Branching factor per step. Improves candidate diversity, will lead to higher acceptance rate, but more lead to higher memory/compute consumption. Default is 4.\n",
|
||||
"* `speculative_num_draft_tokens`: Maximum parallel verification capacity. Allows deeper tree evaluation but will lead to higher GPU memory usage. Default is 8.\n",
|
||||
"\n",
|
||||
"These parameters are the same for EAGLE-2 and EAGLE-3.\n",
|
||||
"\n",
|
||||
"You can find the best combinations of these parameters with [bench_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py).\n",
|
||||
"\n",
|
||||
"In the documentation below, we set `--cuda-graph-max-bs` to be a small value for faster engine startup. For your own workloads, please tune the above parameters together with `--cuda-graph-max-bs`, `--max-running-requests`, `--mem-fraction-static` for the best performance. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### EAGLE-2 decoding\n",
|
||||
"\n",
|
||||
"You can enable EAGLE-2 decoding by setting `--speculative-algorithm EAGLE` and choosing an appropriate model."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sglang.test.doc_patch import launch_server_cmd\n",
|
||||
"from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
|
||||
"\n",
|
||||
"import openai"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"server_process, port = launch_server_cmd(\"\"\"\n",
|
||||
"python3 -m sglang.launch_server --model meta-llama/Llama-2-7b-chat-hf --speculative-algorithm EAGLE \\\n",
|
||||
" --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B --speculative-num-steps 3 \\\n",
|
||||
" --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 --cuda-graph-max-bs 8 --log-level warning\n",
|
||||
"\"\"\")\n",
|
||||
"\n",
|
||||
"wait_for_server(f\"http://localhost:{port}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
|
||||
"\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"meta-llama/Llama-2-7b-chat-hf\",\n",
|
||||
" messages=[\n",
|
||||
" {\"role\": \"user\", \"content\": \"List 3 countries and their capitals.\"},\n",
|
||||
" ],\n",
|
||||
" temperature=0,\n",
|
||||
" max_tokens=64,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(f\"Response: {response}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(server_process)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### EAGLE-2 Decoding with `torch.compile`\n",
|
||||
"\n",
|
||||
"You can also enable `torch.compile` for further optimizations and optionally set `--torch-compile-max-bs`:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"server_process, port = launch_server_cmd(\"\"\"\n",
|
||||
"python3 -m sglang.launch_server --model meta-llama/Llama-2-7b-chat-hf --speculative-algorithm EAGLE \\\n",
|
||||
" --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B --speculative-num-steps 5 \\\n",
|
||||
" --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --mem-fraction 0.6 \\\n",
|
||||
" --enable-torch-compile --torch-compile-max-bs 2 --log-level warning\n",
|
||||
"\"\"\")\n",
|
||||
"\n",
|
||||
"wait_for_server(f\"http://localhost:{port}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
|
||||
"\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"meta-llama/Llama-2-7b-chat-hf\",\n",
|
||||
" messages=[\n",
|
||||
" {\"role\": \"user\", \"content\": \"List 3 countries and their capitals.\"},\n",
|
||||
" ],\n",
|
||||
" temperature=0,\n",
|
||||
" max_tokens=64,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(f\"Response: {response}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(server_process)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### EAGLE-2 Decoding via Frequency-Ranked Speculative Sampling\n",
|
||||
"\n",
|
||||
"By employing a truncated high-frequency token vocabulary in the draft model, Eagle speculative decoding reduces `lm_head` computational overhead while accelerating the pipeline without quality degradation. For more details, checkout [the paper](https://arxiv.org/pdf/arXiv:2502.14856).\n",
|
||||
"\n",
|
||||
"In our implementation, set `--speculative-token-map` to enable the optimization. You can get the high-frequency token in FR-Spec from [this model](https://huggingface.co/thunlp/LLaMA3-Instruct-8B-FR-Spec). Or you can obtain high-frequency token by directly downloading these token from [this repo](https://github.com/thunlp/FR-Spec/tree/main?tab=readme-ov-file#prepare-fr-spec-vocabulary-subset).\n",
|
||||
"\n",
|
||||
"Thanks for the contribution from [Weilin Zhao](https://github.com/Achazwl) and [Zhousx](https://github.com/Zhou-sx). "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"server_process, port = launch_server_cmd(\"\"\"\n",
|
||||
"python3 -m sglang.launch_server --model meta-llama/Meta-Llama-3-8B-Instruct --speculative-algorithm EAGLE \\\n",
|
||||
" --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 \\\n",
|
||||
" --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --speculative-token-map thunlp/LLaMA3-Instruct-8B-FR-Spec/freq_32768.pt \\\n",
|
||||
" --mem-fraction 0.7 --cuda-graph-max-bs 2 --dtype float16 --log-level warning\n",
|
||||
"\"\"\")\n",
|
||||
"\n",
|
||||
"wait_for_server(f\"http://localhost:{port}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
|
||||
"\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"meta-llama/Meta-Llama-3-8B-Instruct\",\n",
|
||||
" messages=[\n",
|
||||
" {\"role\": \"user\", \"content\": \"List 3 countries and their capitals.\"},\n",
|
||||
" ],\n",
|
||||
" temperature=0,\n",
|
||||
" max_tokens=64,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(f\"Response: {response}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(server_process)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### EAGLE-3 Decoding\n",
|
||||
"\n",
|
||||
"You can enable EAGLE-3 decoding by setting `--speculative-algorithm EAGLE3` and choosing an appropriate model."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"server_process, port = launch_server_cmd(\"\"\"\n",
|
||||
"python3 -m sglang.launch_server --model meta-llama/Llama-3.1-8B-Instruct --speculative-algorithm EAGLE3 \\\n",
|
||||
" --speculative-draft-model-path jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B --speculative-num-steps 5 \\\n",
|
||||
" --speculative-eagle-topk 8 --speculative-num-draft-tokens 32 --mem-fraction 0.6 \\\n",
|
||||
" --cuda-graph-max-bs 2 --dtype float16 --log-level warning\n",
|
||||
"\"\"\")\n",
|
||||
"\n",
|
||||
"wait_for_server(f\"http://localhost:{port}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
|
||||
"\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"meta-llama/Meta-Llama-3.1-8B-Instruct\",\n",
|
||||
" messages=[\n",
|
||||
" {\"role\": \"user\", \"content\": \"List 3 countries and their capitals.\"},\n",
|
||||
" ],\n",
|
||||
" temperature=0,\n",
|
||||
" max_tokens=64,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(f\"Response: {response}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(server_process)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Multi Token Prediction\n",
|
||||
"\n",
|
||||
"We support [MTP(Multi-Token Prediction)](https://arxiv.org/pdf/2404.19737) in SGLang by using speculative decoding. We use Xiaomi/MiMo-7B-RL model as example here (deepseek mtp usage refer to [deepseek doc](../basic_usage/deepseek.md#multi-token-prediction))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"server_process, port = launch_server_cmd(\"\"\"\n",
|
||||
" python3 -m sglang.launch_server --model-path XiaomiMiMo/MiMo-7B-RL --host 0.0.0.0 --trust-remote-code \\\n",
|
||||
" --speculative-algorithm EAGLE --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \\\n",
|
||||
" --mem-fraction 0.5 --log-level warning\n",
|
||||
"\"\"\")\n",
|
||||
"\n",
|
||||
"wait_for_server(f\"http://localhost:{port}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import requests\n",
|
||||
"\n",
|
||||
"url = f\"http://localhost:{port}/v1/chat/completions\"\n",
|
||||
"\n",
|
||||
"data = {\n",
|
||||
" \"model\": \"XiaomiMiMo/MiMo-7B-RL\",\n",
|
||||
" \"messages\": [{\"role\": \"user\", \"content\": \"What is the capital of France?\"}],\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"response = requests.post(url, json=data)\n",
|
||||
"print_highlight(response.json())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(server_process)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## References\n",
|
||||
"\n",
|
||||
"EAGLE process is as follows:\n",
|
||||
"\n",
|
||||
"- Within EAGLE the draft model predicts the next feature vector, i.e. the last hidden state of the original LLM, using the feature sequence $(f_1, ..., f_k)$ and the token sequence $(t_2, ..., t_{k+1})$. \n",
|
||||
"- The next token is then sampled from $p_{k+2}=\\text{LMHead}(f_{k+1})$. Afterwards, the two sequences are extended in a tree style—branching out multiple potential continuations, with the branching factor per step controlled by the `speculative_eagle_topk` parameter—to ensure a more coherent connection of context, and are given as input again.\n",
|
||||
"- EAGLE-2 additionally uses the draft model to evaluate how probable certain branches in the draft tree are, dynamically stopping the expansion of unlikely branches. After the expansion phase, reranking is employed to select only the top `speculative_num_draft_tokens` final nodes as draft tokens.\n",
|
||||
"- EAGLE-3 removes the feature prediction objective, incorporates low and mid-layer features, and is trained in an on-policy manner.\n",
|
||||
"\n",
|
||||
"This enhances drafting accuracy by operating on the features instead of tokens for more regular inputs and passing the tokens from the next timestep additionally to minimize randomness effects from sampling. Furthermore the dynamic adjustment of the draft tree and selection of reranked final nodes increases acceptance rate of draft tokens further. For more details see [EAGLE-2](https://arxiv.org/abs/2406.16858) and [EAGLE-3](https://arxiv.org/abs/2503.01840) paper.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"For guidance how to train your own EAGLE model please see the [EAGLE repo](https://github.com/SafeAILab/EAGLE/tree/main?tab=readme-ov-file#train)."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,278 @@
|
||||
---
|
||||
title: "Speculative Decoding"
|
||||
metatags:
|
||||
description: "SGLang EAGLE speculative decoding: EAGLE-2/EAGLE-3, up to 2.4x throughput improvement, draft model configuration, MTP for DeepSeek."
|
||||
---
|
||||
SGLang now provides an EAGLE-based (EAGLE-2/EAGLE-3) speculative decoding option. Our implementation aims to maximize speed and efficiency and is considered to be among the fastest in open-source LLM engines.
|
||||
|
||||
### Performance Highlights
|
||||
|
||||
Please see below for the huge improvements on throughput for LLaMA-Instruct 3.1 8B tested on MT bench that can be achieved via EAGLE3 decoding.
|
||||
For further details please see the [EAGLE3 paper](https://arxiv.org/pdf/2503.01840).
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "50%"}} />
|
||||
<col style={{width: "50%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Method</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Throughput (tokens/s)</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>SGLang (w/o speculative, 1x H100)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>158.34 tokens/s</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>SGLang + EAGLE-2 (1x H100)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>244.10 tokens/s</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>SGLang + EAGLE-3 (1x H100)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>373.25 tokens/s</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
## EAGLE Decoding
|
||||
|
||||
To enable EAGLE speculative decoding the following parameters are relevant:
|
||||
* `speculative_draft_model_path`: Specifies draft model. This parameter is required.
|
||||
* `speculative_num_steps`: Depth of autoregressive drafting. Increases speculation range but risks rejection cascades. Default is 5.
|
||||
* `speculative_eagle_topk`: Branching factor per step. Improves candidate diversity, will lead to higher acceptance rate, but more lead to higher memory/compute consumption. Default is 4.
|
||||
* `speculative_num_draft_tokens`: Maximum parallel verification capacity. Allows deeper tree evaluation but will lead to higher GPU memory usage. Default is 8.
|
||||
|
||||
These parameters are the same for EAGLE-2 and EAGLE-3.
|
||||
|
||||
You can find the best combinations of these parameters with [bench_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py).
|
||||
|
||||
In the documentation below, we set `--cuda-graph-max-bs` to be a small value for faster engine startup. For your own workloads, please tune the above parameters together with `--cuda-graph-max-bs`, `--max-running-requests`, `--mem-fraction-static` for the best performance.
|
||||
|
||||
|
||||
### EAGLE-2 decoding
|
||||
|
||||
You can enable EAGLE-2 decoding by setting `--speculative-algorithm EAGLE` and choosing an appropriate model.
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
from sglang.test.doc_patch import launch_server_cmd
|
||||
from sglang.utils import wait_for_server, print_highlight, terminate_process
|
||||
|
||||
import openai
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
server_process, port = launch_server_cmd(
|
||||
"""
|
||||
python3 -m sglang.launch_server --model meta-llama/Llama-2-7b-chat-hf --speculative-algorithm EAGLE \
|
||||
--speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B --speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 4 --speculative-num-draft-tokens 16 --cuda-graph-max-bs 8 --log-level warning
|
||||
"""
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="meta-llama/Llama-2-7b-chat-hf",
|
||||
messages=[
|
||||
{"role": "user", "content": "List 3 countries and their capitals."},
|
||||
],
|
||||
temperature=0,
|
||||
max_tokens=64,
|
||||
)
|
||||
|
||||
print_highlight(f"Response: {response}")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
terminate_process(server_process)
|
||||
```
|
||||
|
||||
### EAGLE-2 Decoding with `torch.compile`
|
||||
|
||||
You can also enable `torch.compile` for further optimizations and optionally set `--torch-compile-max-bs`:
|
||||
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
server_process, port = launch_server_cmd(
|
||||
"""
|
||||
python3 -m sglang.launch_server --model meta-llama/Llama-2-7b-chat-hf --speculative-algorithm EAGLE \
|
||||
--speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B --speculative-num-steps 5 \
|
||||
--speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --mem-fraction 0.6 \
|
||||
--enable-torch-compile --torch-compile-max-bs 2 --log-level warning
|
||||
"""
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="meta-llama/Llama-2-7b-chat-hf",
|
||||
messages=[
|
||||
{"role": "user", "content": "List 3 countries and their capitals."},
|
||||
],
|
||||
temperature=0,
|
||||
max_tokens=64,
|
||||
)
|
||||
|
||||
print_highlight(f"Response: {response}")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
terminate_process(server_process)
|
||||
```
|
||||
|
||||
### EAGLE-2 Decoding via Frequency-Ranked Speculative Sampling
|
||||
|
||||
By employing a truncated high-frequency token vocabulary in the draft model, Eagle speculative decoding reduces `lm_head` computational overhead while accelerating the pipeline without quality degradation. For more details, checkout [the paper](https://arxiv.org/pdf/arXiv:2502.14856).
|
||||
|
||||
In our implementation, set `--speculative-token-map` to enable the optimization. You can get the high-frequency token in FR-Spec from [this model](https://huggingface.co/thunlp/LLaMA3-Instruct-8B-FR-Spec). Or you can obtain high-frequency token by directly downloading these token from [this repo](https://github.com/thunlp/FR-Spec/tree/main?tab=readme-ov-file#prepare-fr-spec-vocabulary-subset).
|
||||
|
||||
Thanks for the contribution from [Weilin Zhao](https://github.com/Achazwl) and [Zhousx](https://github.com/Zhou-sx).
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
server_process, port = launch_server_cmd(
|
||||
"""
|
||||
python3 -m sglang.launch_server --model meta-llama/Meta-Llama-3-8B-Instruct --speculative-algorithm EAGLE \
|
||||
--speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 \
|
||||
--speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --speculative-token-map thunlp/LLaMA3-Instruct-8B-FR-Spec/freq_32768.pt \
|
||||
--mem-fraction 0.7 --cuda-graph-max-bs 2 --dtype float16 --log-level warning
|
||||
"""
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="meta-llama/Meta-Llama-3-8B-Instruct",
|
||||
messages=[
|
||||
{"role": "user", "content": "List 3 countries and their capitals."},
|
||||
],
|
||||
temperature=0,
|
||||
max_tokens=64,
|
||||
)
|
||||
|
||||
print_highlight(f"Response: {response}")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
terminate_process(server_process)
|
||||
```
|
||||
|
||||
### EAGLE-3 Decoding
|
||||
|
||||
You can enable EAGLE-3 decoding by setting `--speculative-algorithm EAGLE3` and choosing an appropriate model.
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
server_process, port = launch_server_cmd(
|
||||
"""
|
||||
python3 -m sglang.launch_server --model meta-llama/Llama-3.1-8B-Instruct --speculative-algorithm EAGLE3 \
|
||||
--speculative-draft-model-path jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B --speculative-num-steps 5 \
|
||||
--speculative-eagle-topk 8 --speculative-num-draft-tokens 32 --mem-fraction 0.6 \
|
||||
--cuda-graph-max-bs 2 --dtype float16 --log-level warning
|
||||
"""
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
|
||||
messages=[
|
||||
{"role": "user", "content": "List 3 countries and their capitals."},
|
||||
],
|
||||
temperature=0,
|
||||
max_tokens=64,
|
||||
)
|
||||
|
||||
print_highlight(f"Response: {response}")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
terminate_process(server_process)
|
||||
```
|
||||
|
||||
## Multi Token Prediction
|
||||
|
||||
We support [MTP(Multi-Token Prediction)](https://arxiv.org/pdf/2404.19737) in SGLang by using speculative decoding. We use Xiaomi/MiMo-7B-RL model as example here (deepseek mtp usage refer to [deepseek doc](../basic_usage/deepseek_v3#multi-token-prediction))
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
server_process, port = launch_server_cmd(
|
||||
"""
|
||||
python3 -m sglang.launch_server --model-path XiaomiMiMo/MiMo-7B-RL --host 0.0.0.0 --trust-remote-code \
|
||||
--speculative-algorithm EAGLE --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \
|
||||
--mem-fraction 0.5 --log-level warning
|
||||
"""
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
url = f"http://localhost:{port}/v1/chat/completions"
|
||||
|
||||
data = {
|
||||
"model": "XiaomiMiMo/MiMo-7B-RL",
|
||||
"messages": [{"role": "user", "content": "What is the capital of France?"}],
|
||||
}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
print_highlight(response.json())
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
terminate_process(server_process)
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
EAGLE process is as follows:
|
||||
|
||||
- Within EAGLE the draft model predicts the next feature vector, i.e. the last hidden state of the original LLM, using the feature sequence $(f_1, ..., f_k)$ and the token sequence $(t_2, ..., t_{k+1})$.
|
||||
- The next token is then sampled from $p_{k+2}=\text{LMHead}(f_{k+1})$. Afterwards, the two sequences are extended in a tree style—branching out multiple potential continuations, with the branching factor per step controlled by the `speculative_eagle_topk` parameter—to ensure a more coherent connection of context, and are given as input again.
|
||||
- EAGLE-2 additionally uses the draft model to evaluate how probable certain branches in the draft tree are, dynamically stopping the expansion of unlikely branches. After the expansion phase, reranking is employed to select only the top `speculative_num_draft_tokens` final nodes as draft tokens.
|
||||
- EAGLE-3 removes the feature prediction objective, incorporates low and mid-layer features, and is trained in an on-policy manner.
|
||||
|
||||
This enhances drafting accuracy by operating on the features instead of tokens for more regular inputs and passing the tokens from the next timestep additionally to minimize randomness effects from sampling. Furthermore the dynamic adjustment of the draft tree and selection of reranked final nodes increases acceptance rate of draft tokens further. For more details see [EAGLE-2](https://arxiv.org/abs/2406.16858) and [EAGLE-3](https://arxiv.org/abs/2503.01840) paper.
|
||||
|
||||
|
||||
For guidance how to train your own EAGLE model please see the [EAGLE repo](https://github.com/SafeAILab/EAGLE/tree/main?tab=readme-ov-file#train).
|
||||
@@ -0,0 +1,997 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Structured Outputs"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"You can specify a JSON schema, [regular expression](https://en.wikipedia.org/wiki/Regular_expression) or [EBNF](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) to constrain the model output. The model output will be guaranteed to follow the given constraints. Only one constraint parameter (`json_schema`, `regex`, or `ebnf`) can be specified for a request.\n",
|
||||
"\n",
|
||||
"SGLang supports three grammar backends:\n",
|
||||
"\n",
|
||||
"- [XGrammar](https://github.com/mlc-ai/xgrammar)(default): Supports JSON schema, regular expression, and EBNF constraints.\n",
|
||||
"- [Outlines](https://github.com/dottxt-ai/outlines): Supports JSON schema and regular expression constraints.\n",
|
||||
"- [Llguidance](https://github.com/guidance-ai/llguidance): Supports JSON schema, regular expression, and EBNF constraints.\n",
|
||||
"\n",
|
||||
"We suggest using XGrammar for its better performance and utility. XGrammar currently uses the [GGML BNF format](https://github.com/ggerganov/llama.cpp/blob/master/grammars/README). For more details, see [XGrammar technical overview](https://blog.mlc.ai/2024/11/22/achieving-efficient-flexible-portable-structured-generation-with-xgrammar).\n",
|
||||
"\n",
|
||||
"To use Outlines, simply add `--grammar-backend outlines` when launching the server.\n",
|
||||
"To use llguidance, add `--grammar-backend llguidance` when launching the server.\n",
|
||||
"If no backend is specified, XGrammar will be used as the default.\n",
|
||||
"\n",
|
||||
"For better output quality, **It's advisable to explicitly include instructions in the prompt to guide the model to generate the desired format.** For example, you can specify, 'Please generate the output in the following JSON format: ...'.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## OpenAI Compatible API"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import openai\n",
|
||||
"import os\n",
|
||||
"\n",
|
||||
"from sglang.test.doc_patch import launch_server_cmd\n",
|
||||
"from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
|
||||
"\n",
|
||||
"os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"server_process, port = launch_server_cmd(\n",
|
||||
" \"python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --host 0.0.0.0 --log-level warning\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"wait_for_server(f\"http://localhost:{port}\")\n",
|
||||
"client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### JSON\n",
|
||||
"\n",
|
||||
"you can directly define a JSON schema or use [Pydantic](https://docs.pydantic.dev/latest/) to define and validate the response."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Using Pydantic**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from pydantic import BaseModel, Field\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Define the schema using Pydantic\n",
|
||||
"class CapitalInfo(BaseModel):\n",
|
||||
" name: str = Field(..., pattern=r\"^\\w+$\", description=\"Name of the capital city\")\n",
|
||||
" population: int = Field(..., description=\"Population of the capital city\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"meta-llama/Meta-Llama-3.1-8B-Instruct\",\n",
|
||||
" messages=[\n",
|
||||
" {\n",
|
||||
" \"role\": \"user\",\n",
|
||||
" \"content\": \"Please generate the information of the capital of France in the JSON format.\",\n",
|
||||
" },\n",
|
||||
" ],\n",
|
||||
" temperature=0,\n",
|
||||
" max_tokens=128,\n",
|
||||
" response_format={\n",
|
||||
" \"type\": \"json_schema\",\n",
|
||||
" \"json_schema\": {\n",
|
||||
" \"name\": \"foo\",\n",
|
||||
" # convert the pydantic model to json schema\n",
|
||||
" \"schema\": CapitalInfo.model_json_schema(),\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"response_content = response.choices[0].message.content\n",
|
||||
"# validate the JSON response by the pydantic model\n",
|
||||
"capital_info = CapitalInfo.model_validate_json(response_content)\n",
|
||||
"print_highlight(f\"Validated response: {capital_info.model_dump_json()}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**JSON Schema Directly**\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"\n",
|
||||
"json_schema = json.dumps(\n",
|
||||
" {\n",
|
||||
" \"type\": \"object\",\n",
|
||||
" \"properties\": {\n",
|
||||
" \"name\": {\"type\": \"string\", \"pattern\": \"^[\\\\w]+$\"},\n",
|
||||
" \"population\": {\"type\": \"integer\"},\n",
|
||||
" },\n",
|
||||
" \"required\": [\"name\", \"population\"],\n",
|
||||
" }\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"meta-llama/Meta-Llama-3.1-8B-Instruct\",\n",
|
||||
" messages=[\n",
|
||||
" {\n",
|
||||
" \"role\": \"user\",\n",
|
||||
" \"content\": \"Give me the information of the capital of France in the JSON format.\",\n",
|
||||
" },\n",
|
||||
" ],\n",
|
||||
" temperature=0,\n",
|
||||
" max_tokens=128,\n",
|
||||
" response_format={\n",
|
||||
" \"type\": \"json_schema\",\n",
|
||||
" \"json_schema\": {\"name\": \"foo\", \"schema\": json.loads(json_schema)},\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(response.choices[0].message.content)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### EBNF"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"ebnf_grammar = \"\"\"\n",
|
||||
"root ::= city | description\n",
|
||||
"city ::= \"London\" | \"Paris\" | \"Berlin\" | \"Rome\"\n",
|
||||
"description ::= city \" is \" status\n",
|
||||
"status ::= \"the capital of \" country\n",
|
||||
"country ::= \"England\" | \"France\" | \"Germany\" | \"Italy\"\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"meta-llama/Meta-Llama-3.1-8B-Instruct\",\n",
|
||||
" messages=[\n",
|
||||
" {\"role\": \"system\", \"content\": \"You are a helpful geography bot.\"},\n",
|
||||
" {\n",
|
||||
" \"role\": \"user\",\n",
|
||||
" \"content\": \"Give me the information of the capital of France.\",\n",
|
||||
" },\n",
|
||||
" ],\n",
|
||||
" temperature=0,\n",
|
||||
" max_tokens=32,\n",
|
||||
" extra_body={\"ebnf\": ebnf_grammar},\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(response.choices[0].message.content)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Regular expression"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"meta-llama/Meta-Llama-3.1-8B-Instruct\",\n",
|
||||
" messages=[\n",
|
||||
" {\"role\": \"user\", \"content\": \"What is the capital of France?\"},\n",
|
||||
" ],\n",
|
||||
" temperature=0,\n",
|
||||
" max_tokens=128,\n",
|
||||
" extra_body={\"regex\": \"(Paris|London)\"},\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(response.choices[0].message.content)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Structural Tag"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"tool_get_current_weather = {\n",
|
||||
" \"type\": \"function\",\n",
|
||||
" \"function\": {\n",
|
||||
" \"name\": \"get_current_weather\",\n",
|
||||
" \"description\": \"Get the current weather in a given location\",\n",
|
||||
" \"parameters\": {\n",
|
||||
" \"type\": \"object\",\n",
|
||||
" \"properties\": {\n",
|
||||
" \"city\": {\n",
|
||||
" \"type\": \"string\",\n",
|
||||
" \"description\": \"The city to find the weather for, e.g. 'San Francisco'\",\n",
|
||||
" },\n",
|
||||
" \"state\": {\n",
|
||||
" \"type\": \"string\",\n",
|
||||
" \"description\": \"the two-letter abbreviation for the state that the city is\"\n",
|
||||
" \" in, e.g. 'CA' which would mean 'California'\",\n",
|
||||
" },\n",
|
||||
" \"unit\": {\n",
|
||||
" \"type\": \"string\",\n",
|
||||
" \"description\": \"The unit to fetch the temperature in\",\n",
|
||||
" \"enum\": [\"celsius\", \"fahrenheit\"],\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
" \"required\": [\"city\", \"state\", \"unit\"],\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"tool_get_current_date = {\n",
|
||||
" \"type\": \"function\",\n",
|
||||
" \"function\": {\n",
|
||||
" \"name\": \"get_current_date\",\n",
|
||||
" \"description\": \"Get the current date and time for a given timezone\",\n",
|
||||
" \"parameters\": {\n",
|
||||
" \"type\": \"object\",\n",
|
||||
" \"properties\": {\n",
|
||||
" \"timezone\": {\n",
|
||||
" \"type\": \"string\",\n",
|
||||
" \"description\": \"The timezone to fetch the current date and time for, e.g. 'America/New_York'\",\n",
|
||||
" }\n",
|
||||
" },\n",
|
||||
" \"required\": [\"timezone\"],\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"schema_get_current_weather = tool_get_current_weather[\"function\"][\"parameters\"]\n",
|
||||
"schema_get_current_date = tool_get_current_date[\"function\"][\"parameters\"]\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def get_messages():\n",
|
||||
" return [\n",
|
||||
" {\n",
|
||||
" \"role\": \"system\",\n",
|
||||
" \"content\": f\"\"\"\n",
|
||||
"# Tool Instructions\n",
|
||||
"- Always execute python code in messages that you share.\n",
|
||||
"- When looking for real time information use relevant functions if available else fallback to brave_search\n",
|
||||
"You have access to the following functions:\n",
|
||||
"Use the function 'get_current_weather' to: Get the current weather in a given location\n",
|
||||
"{tool_get_current_weather[\"function\"]}\n",
|
||||
"Use the function 'get_current_date' to: Get the current date and time for a given timezone\n",
|
||||
"{tool_get_current_date[\"function\"]}\n",
|
||||
"If a you choose to call a function ONLY reply in the following format:\n",
|
||||
"<{{start_tag}}={{function_name}}>{{parameters}}{{end_tag}}\n",
|
||||
"where\n",
|
||||
"start_tag => `<function`\n",
|
||||
"parameters => a JSON dict with the function argument name as key and function argument value as value.\n",
|
||||
"end_tag => `</function>`\n",
|
||||
"Here is an example,\n",
|
||||
"<function=example_function_name>{{\"example_name\": \"example_value\"}}</function>\n",
|
||||
"Reminder:\n",
|
||||
"- Function calls MUST follow the specified format\n",
|
||||
"- Required parameters MUST be specified\n",
|
||||
"- Only call one function at a time\n",
|
||||
"- Put the entire function call reply on one line\n",
|
||||
"- Always add your sources when using search results to answer the user query\n",
|
||||
"You are a helpful assistant.\"\"\",\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"role\": \"user\",\n",
|
||||
" \"content\": \"You are in New York. Please get the current date and time, and the weather.\",\n",
|
||||
" },\n",
|
||||
" ]\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"messages = get_messages()\n",
|
||||
"\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"meta-llama/Meta-Llama-3.1-8B-Instruct\",\n",
|
||||
" messages=messages,\n",
|
||||
" response_format={\n",
|
||||
" \"type\": \"structural_tag\",\n",
|
||||
" \"structures\": [\n",
|
||||
" {\n",
|
||||
" \"begin\": \"<function=get_current_weather>\",\n",
|
||||
" \"schema\": schema_get_current_weather,\n",
|
||||
" \"end\": \"</function>\",\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"begin\": \"<function=get_current_date>\",\n",
|
||||
" \"schema\": schema_get_current_date,\n",
|
||||
" \"end\": \"</function>\",\n",
|
||||
" },\n",
|
||||
" ],\n",
|
||||
" \"triggers\": [\"<function=\"],\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(response.choices[0].message.content)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Support for XGrammar latest structural tag format\n",
|
||||
"# https://xgrammar.mlc.ai/docs/tutorials/structural_tag.html\n",
|
||||
"\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"meta-llama/Meta-Llama-3.1-8B-Instruct\",\n",
|
||||
" messages=messages,\n",
|
||||
" response_format={\n",
|
||||
" \"type\": \"structural_tag\",\n",
|
||||
" \"format\": {\n",
|
||||
" \"type\": \"triggered_tags\",\n",
|
||||
" \"triggers\": [\"<function=\"],\n",
|
||||
" \"tags\": [\n",
|
||||
" {\n",
|
||||
" \"begin\": \"<function=get_current_weather>\",\n",
|
||||
" \"content\": {\n",
|
||||
" \"type\": \"json_schema\",\n",
|
||||
" \"json_schema\": schema_get_current_weather,\n",
|
||||
" },\n",
|
||||
" \"end\": \"</function>\",\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"begin\": \"<function=get_current_date>\",\n",
|
||||
" \"content\": {\n",
|
||||
" \"type\": \"json_schema\",\n",
|
||||
" \"json_schema\": schema_get_current_date,\n",
|
||||
" },\n",
|
||||
" \"end\": \"</function>\",\n",
|
||||
" },\n",
|
||||
" ],\n",
|
||||
" \"at_least_one\": False,\n",
|
||||
" \"stop_after_first\": False,\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(response.choices[0].message.content)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Native API and SGLang Runtime (SRT)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### JSON"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Using Pydantic**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import requests\n",
|
||||
"import json\n",
|
||||
"from pydantic import BaseModel, Field\n",
|
||||
"\n",
|
||||
"from transformers import AutoTokenizer\n",
|
||||
"\n",
|
||||
"tokenizer = AutoTokenizer.from_pretrained(\"meta-llama/Meta-Llama-3.1-8B-Instruct\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Define the schema using Pydantic\n",
|
||||
"class CapitalInfo(BaseModel):\n",
|
||||
" name: str = Field(..., pattern=r\"^\\w+$\", description=\"Name of the capital city\")\n",
|
||||
" population: int = Field(..., description=\"Population of the capital city\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Make API request\n",
|
||||
"messages = [\n",
|
||||
" {\n",
|
||||
" \"role\": \"user\",\n",
|
||||
" \"content\": \"Here is the information of the capital of France in the JSON format.\\n\",\n",
|
||||
" }\n",
|
||||
"]\n",
|
||||
"text = tokenizer.apply_chat_template(\n",
|
||||
" messages, tokenize=False, add_generation_prompt=True, return_dict=False\n",
|
||||
")\n",
|
||||
"response = requests.post(\n",
|
||||
" f\"http://localhost:{port}/generate\",\n",
|
||||
" json={\n",
|
||||
" \"text\": text,\n",
|
||||
" \"sampling_params\": {\n",
|
||||
" \"temperature\": 0,\n",
|
||||
" \"max_new_tokens\": 64,\n",
|
||||
" \"json_schema\": json.dumps(CapitalInfo.model_json_schema()),\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"print_highlight(response.json())\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"response_data = json.loads(response.json()[\"text\"])\n",
|
||||
"# validate the response by the pydantic model\n",
|
||||
"capital_info = CapitalInfo.model_validate(response_data)\n",
|
||||
"print_highlight(f\"Validated response: {capital_info.model_dump_json()}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**JSON Schema Directly**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"json_schema = json.dumps(\n",
|
||||
" {\n",
|
||||
" \"type\": \"object\",\n",
|
||||
" \"properties\": {\n",
|
||||
" \"name\": {\"type\": \"string\", \"pattern\": \"^[\\\\w]+$\"},\n",
|
||||
" \"population\": {\"type\": \"integer\"},\n",
|
||||
" },\n",
|
||||
" \"required\": [\"name\", \"population\"],\n",
|
||||
" }\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# JSON\n",
|
||||
"response = requests.post(\n",
|
||||
" f\"http://localhost:{port}/generate\",\n",
|
||||
" json={\n",
|
||||
" \"text\": text,\n",
|
||||
" \"sampling_params\": {\n",
|
||||
" \"temperature\": 0,\n",
|
||||
" \"max_new_tokens\": 64,\n",
|
||||
" \"json_schema\": json_schema,\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(response.json())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### EBNF"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"messages = [\n",
|
||||
" {\n",
|
||||
" \"role\": \"user\",\n",
|
||||
" \"content\": \"Give me the information of the capital of France.\",\n",
|
||||
" }\n",
|
||||
"]\n",
|
||||
"text = tokenizer.apply_chat_template(\n",
|
||||
" messages, tokenize=False, add_generation_prompt=True, return_dict=False\n",
|
||||
")\n",
|
||||
"response = requests.post(\n",
|
||||
" f\"http://localhost:{port}/generate\",\n",
|
||||
" json={\n",
|
||||
" \"text\": text,\n",
|
||||
" \"sampling_params\": {\n",
|
||||
" \"max_new_tokens\": 128,\n",
|
||||
" \"temperature\": 0,\n",
|
||||
" \"n\": 3,\n",
|
||||
" \"ebnf\": (\n",
|
||||
" \"root ::= city | description\\n\"\n",
|
||||
" 'city ::= \"London\" | \"Paris\" | \"Berlin\" | \"Rome\"\\n'\n",
|
||||
" 'description ::= city \" is \" status\\n'\n",
|
||||
" 'status ::= \"the capital of \" country\\n'\n",
|
||||
" 'country ::= \"England\" | \"France\" | \"Germany\" | \"Italy\"'\n",
|
||||
" ),\n",
|
||||
" },\n",
|
||||
" \"stream\": False,\n",
|
||||
" \"return_logprob\": False,\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(response.json())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Regular expression"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"messages = [\n",
|
||||
" {\n",
|
||||
" \"role\": \"user\",\n",
|
||||
" \"content\": \"Paris is the capital of\",\n",
|
||||
" }\n",
|
||||
"]\n",
|
||||
"text = tokenizer.apply_chat_template(\n",
|
||||
" messages, tokenize=False, add_generation_prompt=True, return_dict=False\n",
|
||||
")\n",
|
||||
"response = requests.post(\n",
|
||||
" f\"http://localhost:{port}/generate\",\n",
|
||||
" json={\n",
|
||||
" \"text\": text,\n",
|
||||
" \"sampling_params\": {\n",
|
||||
" \"temperature\": 0,\n",
|
||||
" \"max_new_tokens\": 64,\n",
|
||||
" \"regex\": \"(France|England)\",\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"print_highlight(response.json())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Structural Tag"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from transformers import AutoTokenizer\n",
|
||||
"\n",
|
||||
"# generate an answer\n",
|
||||
"tokenizer = AutoTokenizer.from_pretrained(\"meta-llama/Meta-Llama-3.1-8B-Instruct\")\n",
|
||||
"\n",
|
||||
"text = tokenizer.apply_chat_template(\n",
|
||||
" messages, tokenize=False, add_generation_prompt=True, return_dict=False\n",
|
||||
")\n",
|
||||
"payload = {\n",
|
||||
" \"text\": text,\n",
|
||||
" \"sampling_params\": {\n",
|
||||
" \"structural_tag\": json.dumps(\n",
|
||||
" {\n",
|
||||
" \"type\": \"structural_tag\",\n",
|
||||
" \"structures\": [\n",
|
||||
" {\n",
|
||||
" \"begin\": \"<function=get_current_weather>\",\n",
|
||||
" \"schema\": schema_get_current_weather,\n",
|
||||
" \"end\": \"</function>\",\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"begin\": \"<function=get_current_date>\",\n",
|
||||
" \"schema\": schema_get_current_date,\n",
|
||||
" \"end\": \"</function>\",\n",
|
||||
" },\n",
|
||||
" ],\n",
|
||||
" \"triggers\": [\"<function=\"],\n",
|
||||
" }\n",
|
||||
" )\n",
|
||||
" },\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Send POST request to the API endpoint\n",
|
||||
"response = requests.post(f\"http://localhost:{port}/generate\", json=payload)\n",
|
||||
"print_highlight(response.json())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Support for XGrammar latest structural tag format\n",
|
||||
"# https://xgrammar.mlc.ai/docs/tutorials/structural_tag.html\n",
|
||||
"\n",
|
||||
"payload = {\n",
|
||||
" \"text\": text,\n",
|
||||
" \"sampling_params\": {\n",
|
||||
" \"structural_tag\": json.dumps(\n",
|
||||
" {\n",
|
||||
" \"type\": \"structural_tag\",\n",
|
||||
" \"format\": {\n",
|
||||
" \"type\": \"triggered_tags\",\n",
|
||||
" \"triggers\": [\"<function=\"],\n",
|
||||
" \"tags\": [\n",
|
||||
" {\n",
|
||||
" \"begin\": \"<function=get_current_weather>\",\n",
|
||||
" \"content\": {\n",
|
||||
" \"type\": \"json_schema\",\n",
|
||||
" \"json_schema\": schema_get_current_weather,\n",
|
||||
" },\n",
|
||||
" \"end\": \"</function>\",\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"begin\": \"<function=get_current_date>\",\n",
|
||||
" \"content\": {\n",
|
||||
" \"type\": \"json_schema\",\n",
|
||||
" \"json_schema\": schema_get_current_date,\n",
|
||||
" },\n",
|
||||
" \"end\": \"</function>\",\n",
|
||||
" },\n",
|
||||
" ],\n",
|
||||
" \"at_least_one\": False,\n",
|
||||
" \"stop_after_first\": False,\n",
|
||||
" },\n",
|
||||
" }\n",
|
||||
" )\n",
|
||||
" },\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Send POST request to the API endpoint\n",
|
||||
"response = requests.post(f\"http://localhost:{port}/generate\", json=payload)\n",
|
||||
"print_highlight(response.json())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(server_process)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Offline Engine API"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sglang as sgl\n",
|
||||
"\n",
|
||||
"llm = sgl.Engine(\n",
|
||||
" model_path=\"meta-llama/Meta-Llama-3.1-8B-Instruct\", grammar_backend=\"xgrammar\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### JSON"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Using Pydantic**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"from pydantic import BaseModel, Field\n",
|
||||
"\n",
|
||||
"prompts = [\n",
|
||||
" \"Give me the information of the capital of China in the JSON format.\",\n",
|
||||
" \"Give me the information of the capital of France in the JSON format.\",\n",
|
||||
" \"Give me the information of the capital of Ireland in the JSON format.\",\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Define the schema using Pydantic\n",
|
||||
"class CapitalInfo(BaseModel):\n",
|
||||
" name: str = Field(..., pattern=r\"^\\w+$\", description=\"Name of the capital city\")\n",
|
||||
" population: int = Field(..., description=\"Population of the capital city\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"sampling_params = {\n",
|
||||
" \"temperature\": 0.1,\n",
|
||||
" \"top_p\": 0.95,\n",
|
||||
" \"json_schema\": json.dumps(CapitalInfo.model_json_schema()),\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"outputs = llm.generate(prompts, sampling_params)\n",
|
||||
"for prompt, output in zip(prompts, outputs):\n",
|
||||
" print_highlight(\"===============================\")\n",
|
||||
" print_highlight(f\"Prompt: {prompt}\") # validate the output by the pydantic model\n",
|
||||
" capital_info = CapitalInfo.model_validate_json(output[\"text\"])\n",
|
||||
" print_highlight(f\"Validated output: {capital_info.model_dump_json()}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**JSON Schema Directly**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"prompts = [\n",
|
||||
" \"Give me the information of the capital of China in the JSON format.\",\n",
|
||||
" \"Give me the information of the capital of France in the JSON format.\",\n",
|
||||
" \"Give me the information of the capital of Ireland in the JSON format.\",\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"json_schema = json.dumps(\n",
|
||||
" {\n",
|
||||
" \"type\": \"object\",\n",
|
||||
" \"properties\": {\n",
|
||||
" \"name\": {\"type\": \"string\", \"pattern\": \"^[\\\\w]+$\"},\n",
|
||||
" \"population\": {\"type\": \"integer\"},\n",
|
||||
" },\n",
|
||||
" \"required\": [\"name\", \"population\"],\n",
|
||||
" }\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"sampling_params = {\"temperature\": 0.1, \"top_p\": 0.95, \"json_schema\": json_schema}\n",
|
||||
"\n",
|
||||
"outputs = llm.generate(prompts, sampling_params)\n",
|
||||
"for prompt, output in zip(prompts, outputs):\n",
|
||||
" print_highlight(\"===============================\")\n",
|
||||
" print_highlight(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### EBNF\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"prompts = [\n",
|
||||
" \"Give me the information of the capital of France.\",\n",
|
||||
" \"Give me the information of the capital of Germany.\",\n",
|
||||
" \"Give me the information of the capital of Italy.\",\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"sampling_params = {\n",
|
||||
" \"temperature\": 0.8,\n",
|
||||
" \"top_p\": 0.95,\n",
|
||||
" \"ebnf\": (\n",
|
||||
" \"root ::= city | description\\n\"\n",
|
||||
" 'city ::= \"London\" | \"Paris\" | \"Berlin\" | \"Rome\"\\n'\n",
|
||||
" 'description ::= city \" is \" status\\n'\n",
|
||||
" 'status ::= \"the capital of \" country\\n'\n",
|
||||
" 'country ::= \"England\" | \"France\" | \"Germany\" | \"Italy\"'\n",
|
||||
" ),\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"outputs = llm.generate(prompts, sampling_params)\n",
|
||||
"for prompt, output in zip(prompts, outputs):\n",
|
||||
" print_highlight(\"===============================\")\n",
|
||||
" print_highlight(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Regular expression"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"prompts = [\n",
|
||||
" \"Please provide information about London as a major global city:\",\n",
|
||||
" \"Please provide information about Paris as a major global city:\",\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"sampling_params = {\"temperature\": 0.8, \"top_p\": 0.95, \"regex\": \"(France|England)\"}\n",
|
||||
"\n",
|
||||
"outputs = llm.generate(prompts, sampling_params)\n",
|
||||
"for prompt, output in zip(prompts, outputs):\n",
|
||||
" print_highlight(\"===============================\")\n",
|
||||
" print_highlight(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Structural Tag"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"text = tokenizer.apply_chat_template(\n",
|
||||
" messages, tokenize=False, add_generation_prompt=True, return_dict=False\n",
|
||||
")\n",
|
||||
"prompts = [text]\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"sampling_params = {\n",
|
||||
" \"temperature\": 0.8,\n",
|
||||
" \"top_p\": 0.95,\n",
|
||||
" \"structural_tag\": json.dumps(\n",
|
||||
" {\n",
|
||||
" \"type\": \"structural_tag\",\n",
|
||||
" \"structures\": [\n",
|
||||
" {\n",
|
||||
" \"begin\": \"<function=get_current_weather>\",\n",
|
||||
" \"schema\": schema_get_current_weather,\n",
|
||||
" \"end\": \"</function>\",\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"begin\": \"<function=get_current_date>\",\n",
|
||||
" \"schema\": schema_get_current_date,\n",
|
||||
" \"end\": \"</function>\",\n",
|
||||
" },\n",
|
||||
" ],\n",
|
||||
" \"triggers\": [\"<function=\"],\n",
|
||||
" }\n",
|
||||
" ),\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Send POST request to the API endpoint\n",
|
||||
"outputs = llm.generate(prompts, sampling_params)\n",
|
||||
"for prompt, output in zip(prompts, outputs):\n",
|
||||
" print_highlight(\"===============================\")\n",
|
||||
" print_highlight(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Support for XGrammar latest structural tag format\n",
|
||||
"# https://xgrammar.mlc.ai/docs/tutorials/structural_tag.html\n",
|
||||
"\n",
|
||||
"sampling_params = {\n",
|
||||
" \"temperature\": 0.8,\n",
|
||||
" \"top_p\": 0.95,\n",
|
||||
" \"structural_tag\": json.dumps(\n",
|
||||
" {\n",
|
||||
" \"type\": \"structural_tag\",\n",
|
||||
" \"format\": {\n",
|
||||
" \"type\": \"triggered_tags\",\n",
|
||||
" \"triggers\": [\"<function=\"],\n",
|
||||
" \"tags\": [\n",
|
||||
" {\n",
|
||||
" \"begin\": \"<function=get_current_weather>\",\n",
|
||||
" \"content\": {\n",
|
||||
" \"type\": \"json_schema\",\n",
|
||||
" \"json_schema\": schema_get_current_weather,\n",
|
||||
" },\n",
|
||||
" \"end\": \"</function>\",\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"begin\": \"<function=get_current_date>\",\n",
|
||||
" \"content\": {\n",
|
||||
" \"type\": \"json_schema\",\n",
|
||||
" \"json_schema\": schema_get_current_date,\n",
|
||||
" },\n",
|
||||
" \"end\": \"</function>\",\n",
|
||||
" },\n",
|
||||
" ],\n",
|
||||
" \"at_least_one\": False,\n",
|
||||
" \"stop_after_first\": False,\n",
|
||||
" },\n",
|
||||
" }\n",
|
||||
" ),\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Send POST request to the API endpoint\n",
|
||||
"outputs = llm.generate(prompts, sampling_params)\n",
|
||||
"for prompt, output in zip(prompts, outputs):\n",
|
||||
" print_highlight(\"===============================\")\n",
|
||||
" print_highlight(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"llm.shutdown()"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,803 @@
|
||||
---
|
||||
title: "Structured Outputs"
|
||||
metatags:
|
||||
description: "SGLang structured outputs: JSON schema, regex, EBNF constraints. XGrammar, Outlines, Llguidance backends for guaranteed output format."
|
||||
---
|
||||
You can specify a JSON schema, [regular expression](https://en.wikipedia.org/wiki/Regular_expression) or [EBNF](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) to constrain the model output. The model output will be guaranteed to follow the given constraints. Only one constraint parameter (`json_schema`, `regex`, or `ebnf`) can be specified for a request.
|
||||
|
||||
SGLang supports three grammar backends:
|
||||
|
||||
- [XGrammar](https://github.com/mlc-ai/xgrammar)(default): Supports JSON schema, regular expression, and EBNF constraints.
|
||||
- [Outlines](https://github.com/dottxt-ai/outlines): Supports JSON schema and regular expression constraints.
|
||||
- [Llguidance](https://github.com/guidance-ai/llguidance): Supports JSON schema, regular expression, and EBNF constraints.
|
||||
|
||||
We suggest using XGrammar for its better performance and utility. XGrammar currently uses the [GGML BNF format](https://github.com/ggerganov/llama.cpp/blob/master/grammars/README). For more details, see [XGrammar technical overview](https://blog.mlc.ai/2024/11/22/achieving-efficient-flexible-portable-structured-generation-with-xgrammar).
|
||||
|
||||
To use Outlines, simply add `--grammar-backend outlines` when launching the server.
|
||||
To use llguidance, add `--grammar-backend llguidance` when launching the server.
|
||||
If no backend is specified, XGrammar will be used as the default.
|
||||
|
||||
For better output quality, **It's advisable to explicitly include instructions in the prompt to guide the model to generate the desired format.** For example, you can specify, 'Please generate the output in the following JSON format: ...'.
|
||||
|
||||
|
||||
|
||||
## OpenAI Compatible API
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
import os
|
||||
|
||||
from sglang.test.doc_patch import launch_server_cmd
|
||||
from sglang.utils import wait_for_server, print_highlight, terminate_process
|
||||
|
||||
os.environ["TOKENIZERS_PARALLELISM"] = "false"
|
||||
|
||||
|
||||
server_process, port = launch_server_cmd(
|
||||
"python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --host 0.0.0.0 --log-level warning"
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")
|
||||
```
|
||||
|
||||
### JSON
|
||||
|
||||
you can directly define a JSON schema or use [Pydantic](https://docs.pydantic.dev/latest/) to define and validate the response.
|
||||
|
||||
|
||||
**Using Pydantic**
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
# Define the schema using Pydantic
|
||||
class CapitalInfo(BaseModel):
|
||||
name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city")
|
||||
population: int = Field(..., description="Population of the capital city")
|
||||
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Please generate the information of the capital of France in the JSON format.",
|
||||
},
|
||||
],
|
||||
temperature=0,
|
||||
max_tokens=128,
|
||||
response_format={
|
||||
"type": "json_schema",
|
||||
"json_schema": {
|
||||
"name": "foo",
|
||||
# convert the pydantic model to json schema
|
||||
"schema": CapitalInfo.model_json_schema(),
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
response_content = response.choices[0].message.content
|
||||
# validate the JSON response by the pydantic model
|
||||
capital_info = CapitalInfo.model_validate_json(response_content)
|
||||
print_highlight(f"Validated response: {capital_info.model_dump_json()}")
|
||||
```
|
||||
|
||||
**JSON Schema Directly**
|
||||
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import json
|
||||
|
||||
json_schema = json.dumps(
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": {"type": "string", "pattern": "^[\\w]+$"},
|
||||
"population": {"type": "integer"},
|
||||
},
|
||||
"required": ["name", "population"],
|
||||
}
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Give me the information of the capital of France in the JSON format.",
|
||||
},
|
||||
],
|
||||
temperature=0,
|
||||
max_tokens=128,
|
||||
response_format={
|
||||
"type": "json_schema",
|
||||
"json_schema": {"name": "foo", "schema": json.loads(json_schema)},
|
||||
},
|
||||
)
|
||||
|
||||
print_highlight(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
### EBNF
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
ebnf_grammar = """
|
||||
root ::= city | description
|
||||
city ::= "London" | "Paris" | "Berlin" | "Rome"
|
||||
description ::= city " is " status
|
||||
status ::= "the capital of " country
|
||||
country ::= "England" | "France" | "Germany" | "Italy"
|
||||
"""
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful geography bot."},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Give me the information of the capital of France.",
|
||||
},
|
||||
],
|
||||
temperature=0,
|
||||
max_tokens=32,
|
||||
extra_body={"ebnf": ebnf_grammar},
|
||||
)
|
||||
|
||||
print_highlight(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Regular expression
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
response = client.chat.completions.create(
|
||||
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
|
||||
messages=[
|
||||
{"role": "user", "content": "What is the capital of France?"},
|
||||
],
|
||||
temperature=0,
|
||||
max_tokens=128,
|
||||
extra_body={"regex": "(Paris|London)"},
|
||||
)
|
||||
|
||||
print_highlight(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Structural Tag
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
tool_get_current_weather = {
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_current_weather",
|
||||
"description": "Get the current weather in a given location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"city": {
|
||||
"type": "string",
|
||||
"description": "The city to find the weather for, e.g. 'San Francisco'",
|
||||
},
|
||||
"state": {
|
||||
"type": "string",
|
||||
"description": "the two-letter abbreviation for the state that the city is"
|
||||
" in, e.g. 'CA' which would mean 'California'",
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"description": "The unit to fetch the temperature in",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
},
|
||||
},
|
||||
"required": ["city", "state", "unit"],
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
tool_get_current_date = {
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_current_date",
|
||||
"description": "Get the current date and time for a given timezone",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"timezone": {
|
||||
"type": "string",
|
||||
"description": "The timezone to fetch the current date and time for, e.g. 'America/New_York'",
|
||||
}
|
||||
},
|
||||
"required": ["timezone"],
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
schema_get_current_weather = tool_get_current_weather["function"]["parameters"]
|
||||
schema_get_current_date = tool_get_current_date["function"]["parameters"]
|
||||
|
||||
|
||||
def get_messages():
|
||||
return [
|
||||
{
|
||||
"role": "system",
|
||||
"content": f"""
|
||||
# Tool Instructions
|
||||
- Always execute python code in messages that you share.
|
||||
- When looking for real time information use relevant functions if available else fallback to brave_search
|
||||
You have access to the following functions:
|
||||
Use the function 'get_current_weather' to: Get the current weather in a given location
|
||||
{tool_get_current_weather["function"]}
|
||||
Use the function 'get_current_date' to: Get the current date and time for a given timezone
|
||||
{tool_get_current_date["function"]}
|
||||
If a you choose to call a function ONLY reply in the following format:
|
||||
<{{start_tag}}={{function_name}}>{{parameters}}{{end_tag}}
|
||||
where
|
||||
start_tag => `<function`
|
||||
parameters => a JSON dict with the function argument name as key and function argument value as value.
|
||||
end_tag => `</function>`
|
||||
Here is an example,
|
||||
<function=example_function_name>{{"example_name": "example_value"}}</function>
|
||||
Reminder:
|
||||
- Function calls MUST follow the specified format
|
||||
- Required parameters MUST be specified
|
||||
- Only call one function at a time
|
||||
- Put the entire function call reply on one line
|
||||
- Always add your sources when using search results to answer the user query
|
||||
You are a helpful assistant.""",
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "You are in New York. Please get the current date and time, and the weather.",
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
messages = get_messages()
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
|
||||
messages=messages,
|
||||
response_format={
|
||||
"type": "structural_tag",
|
||||
"structures": [
|
||||
{
|
||||
"begin": "<function=get_current_weather>",
|
||||
"schema": schema_get_current_weather,
|
||||
"end": "</function>",
|
||||
},
|
||||
{
|
||||
"begin": "<function=get_current_date>",
|
||||
"schema": schema_get_current_date,
|
||||
"end": "</function>",
|
||||
},
|
||||
],
|
||||
"triggers": ["<function="],
|
||||
},
|
||||
)
|
||||
|
||||
print_highlight(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
# Support for XGrammar latest structural tag format
|
||||
# https://xgrammar.mlc.ai/docs/tutorials/structural_tag.html
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
|
||||
messages=messages,
|
||||
response_format={
|
||||
"type": "structural_tag",
|
||||
"format": {
|
||||
"type": "triggered_tags",
|
||||
"triggers": ["<function="],
|
||||
"tags": [
|
||||
{
|
||||
"begin": "<function=get_current_weather>",
|
||||
"content": {
|
||||
"type": "json_schema",
|
||||
"json_schema": schema_get_current_weather,
|
||||
},
|
||||
"end": "</function>",
|
||||
},
|
||||
{
|
||||
"begin": "<function=get_current_date>",
|
||||
"content": {
|
||||
"type": "json_schema",
|
||||
"json_schema": schema_get_current_date,
|
||||
},
|
||||
"end": "</function>",
|
||||
},
|
||||
],
|
||||
"at_least_one": False,
|
||||
"stop_after_first": False,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
print_highlight(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
## Native API and SGLang Runtime (SRT)
|
||||
|
||||
|
||||
### JSON
|
||||
|
||||
|
||||
**Using Pydantic**
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
import json
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct")
|
||||
|
||||
|
||||
# Define the schema using Pydantic
|
||||
class CapitalInfo(BaseModel):
|
||||
name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city")
|
||||
population: int = Field(..., description="Population of the capital city")
|
||||
|
||||
|
||||
# Make API request
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Here is the information of the capital of France in the JSON format.\n",
|
||||
}
|
||||
]
|
||||
text = tokenizer.apply_chat_template(
|
||||
messages, tokenize=False, add_generation_prompt=True, return_dict=False
|
||||
)
|
||||
response = requests.post(
|
||||
f"http://localhost:{port}/generate",
|
||||
json={
|
||||
"text": text,
|
||||
"sampling_params": {
|
||||
"temperature": 0,
|
||||
"max_new_tokens": 64,
|
||||
"json_schema": json.dumps(CapitalInfo.model_json_schema()),
|
||||
},
|
||||
},
|
||||
)
|
||||
print_highlight(response.json())
|
||||
|
||||
|
||||
response_data = json.loads(response.json()["text"])
|
||||
# validate the response by the pydantic model
|
||||
capital_info = CapitalInfo.model_validate(response_data)
|
||||
print_highlight(f"Validated response: {capital_info.model_dump_json()}")
|
||||
```
|
||||
|
||||
**JSON Schema Directly**
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
json_schema = json.dumps(
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": {"type": "string", "pattern": "^[\\w]+$"},
|
||||
"population": {"type": "integer"},
|
||||
},
|
||||
"required": ["name", "population"],
|
||||
}
|
||||
)
|
||||
|
||||
# JSON
|
||||
response = requests.post(
|
||||
f"http://localhost:{port}/generate",
|
||||
json={
|
||||
"text": text,
|
||||
"sampling_params": {
|
||||
"temperature": 0,
|
||||
"max_new_tokens": 64,
|
||||
"json_schema": json_schema,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
print_highlight(response.json())
|
||||
```
|
||||
|
||||
### EBNF
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Give me the information of the capital of France.",
|
||||
}
|
||||
]
|
||||
text = tokenizer.apply_chat_template(
|
||||
messages, tokenize=False, add_generation_prompt=True, return_dict=False
|
||||
)
|
||||
response = requests.post(
|
||||
f"http://localhost:{port}/generate",
|
||||
json={
|
||||
"text": text,
|
||||
"sampling_params": {
|
||||
"max_new_tokens": 128,
|
||||
"temperature": 0,
|
||||
"n": 3,
|
||||
"ebnf": (
|
||||
"root ::= city | description\n"
|
||||
'city ::= "London" | "Paris" | "Berlin" | "Rome"\n'
|
||||
'description ::= city " is " status\n'
|
||||
'status ::= "the capital of " country\n'
|
||||
'country ::= "England" | "France" | "Germany" | "Italy"'
|
||||
),
|
||||
},
|
||||
"stream": False,
|
||||
"return_logprob": False,
|
||||
},
|
||||
)
|
||||
|
||||
print_highlight(response.json())
|
||||
```
|
||||
|
||||
### Regular expression
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Paris is the capital of",
|
||||
}
|
||||
]
|
||||
text = tokenizer.apply_chat_template(
|
||||
messages, tokenize=False, add_generation_prompt=True, return_dict=False
|
||||
)
|
||||
response = requests.post(
|
||||
f"http://localhost:{port}/generate",
|
||||
json={
|
||||
"text": text,
|
||||
"sampling_params": {
|
||||
"temperature": 0,
|
||||
"max_new_tokens": 64,
|
||||
"regex": "(France|England)",
|
||||
},
|
||||
},
|
||||
)
|
||||
print_highlight(response.json())
|
||||
```
|
||||
|
||||
### Structural Tag
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
# generate an answer
|
||||
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct")
|
||||
|
||||
text = tokenizer.apply_chat_template(
|
||||
messages, tokenize=False, add_generation_prompt=True, return_dict=False
|
||||
)
|
||||
payload = {
|
||||
"text": text,
|
||||
"sampling_params": {
|
||||
"structural_tag": json.dumps(
|
||||
{
|
||||
"type": "structural_tag",
|
||||
"structures": [
|
||||
{
|
||||
"begin": "<function=get_current_weather>",
|
||||
"schema": schema_get_current_weather,
|
||||
"end": "</function>",
|
||||
},
|
||||
{
|
||||
"begin": "<function=get_current_date>",
|
||||
"schema": schema_get_current_date,
|
||||
"end": "</function>",
|
||||
},
|
||||
],
|
||||
"triggers": ["<function="],
|
||||
}
|
||||
)
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
# Send POST request to the API endpoint
|
||||
response = requests.post(f"http://localhost:{port}/generate", json=payload)
|
||||
print_highlight(response.json())
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
# Support for XGrammar latest structural tag format
|
||||
# https://xgrammar.mlc.ai/docs/tutorials/structural_tag.html
|
||||
|
||||
payload = {
|
||||
"text": text,
|
||||
"sampling_params": {
|
||||
"structural_tag": json.dumps(
|
||||
{
|
||||
"type": "structural_tag",
|
||||
"format": {
|
||||
"type": "triggered_tags",
|
||||
"triggers": ["<function="],
|
||||
"tags": [
|
||||
{
|
||||
"begin": "<function=get_current_weather>",
|
||||
"content": {
|
||||
"type": "json_schema",
|
||||
"json_schema": schema_get_current_weather,
|
||||
},
|
||||
"end": "</function>",
|
||||
},
|
||||
{
|
||||
"begin": "<function=get_current_date>",
|
||||
"content": {
|
||||
"type": "json_schema",
|
||||
"json_schema": schema_get_current_date,
|
||||
},
|
||||
"end": "</function>",
|
||||
},
|
||||
],
|
||||
"at_least_one": False,
|
||||
"stop_after_first": False,
|
||||
},
|
||||
}
|
||||
)
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
# Send POST request to the API endpoint
|
||||
response = requests.post(f"http://localhost:{port}/generate", json=payload)
|
||||
print_highlight(response.json())
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
terminate_process(server_process)
|
||||
```
|
||||
|
||||
## Offline Engine API
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import sglang as sgl
|
||||
|
||||
llm = sgl.Engine(
|
||||
model_path="meta-llama/Meta-Llama-3.1-8B-Instruct", grammar_backend="xgrammar"
|
||||
)
|
||||
```
|
||||
|
||||
### JSON
|
||||
|
||||
|
||||
**Using Pydantic**
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import json
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
prompts = [
|
||||
"Give me the information of the capital of China in the JSON format.",
|
||||
"Give me the information of the capital of France in the JSON format.",
|
||||
"Give me the information of the capital of Ireland in the JSON format.",
|
||||
]
|
||||
|
||||
|
||||
# Define the schema using Pydantic
|
||||
class CapitalInfo(BaseModel):
|
||||
name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city")
|
||||
population: int = Field(..., description="Population of the capital city")
|
||||
|
||||
|
||||
sampling_params = {
|
||||
"temperature": 0.1,
|
||||
"top_p": 0.95,
|
||||
"json_schema": json.dumps(CapitalInfo.model_json_schema()),
|
||||
}
|
||||
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
for prompt, output in zip(prompts, outputs):
|
||||
print_highlight("===============================")
|
||||
print_highlight(f"Prompt: {prompt}") # validate the output by the pydantic model
|
||||
capital_info = CapitalInfo.model_validate_json(output["text"])
|
||||
print_highlight(f"Validated output: {capital_info.model_dump_json()}")
|
||||
```
|
||||
|
||||
**JSON Schema Directly**
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
prompts = [
|
||||
"Give me the information of the capital of China in the JSON format.",
|
||||
"Give me the information of the capital of France in the JSON format.",
|
||||
"Give me the information of the capital of Ireland in the JSON format.",
|
||||
]
|
||||
|
||||
json_schema = json.dumps(
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": {"type": "string", "pattern": "^[\\w]+$"},
|
||||
"population": {"type": "integer"},
|
||||
},
|
||||
"required": ["name", "population"],
|
||||
}
|
||||
)
|
||||
|
||||
sampling_params = {"temperature": 0.1, "top_p": 0.95, "json_schema": json_schema}
|
||||
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
for prompt, output in zip(prompts, outputs):
|
||||
print_highlight("===============================")
|
||||
print_highlight(f"Prompt: {prompt}\nGenerated text: {output['text']}")
|
||||
```
|
||||
|
||||
### EBNF
|
||||
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
prompts = [
|
||||
"Give me the information of the capital of France.",
|
||||
"Give me the information of the capital of Germany.",
|
||||
"Give me the information of the capital of Italy.",
|
||||
]
|
||||
|
||||
sampling_params = {
|
||||
"temperature": 0.8,
|
||||
"top_p": 0.95,
|
||||
"ebnf": (
|
||||
"root ::= city | description\n"
|
||||
'city ::= "London" | "Paris" | "Berlin" | "Rome"\n'
|
||||
'description ::= city " is " status\n'
|
||||
'status ::= "the capital of " country\n'
|
||||
'country ::= "England" | "France" | "Germany" | "Italy"'
|
||||
),
|
||||
}
|
||||
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
for prompt, output in zip(prompts, outputs):
|
||||
print_highlight("===============================")
|
||||
print_highlight(f"Prompt: {prompt}\nGenerated text: {output['text']}")
|
||||
```
|
||||
|
||||
### Regular expression
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
prompts = [
|
||||
"Please provide information about London as a major global city:",
|
||||
"Please provide information about Paris as a major global city:",
|
||||
]
|
||||
|
||||
sampling_params = {"temperature": 0.8, "top_p": 0.95, "regex": "(France|England)"}
|
||||
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
for prompt, output in zip(prompts, outputs):
|
||||
print_highlight("===============================")
|
||||
print_highlight(f"Prompt: {prompt}\nGenerated text: {output['text']}")
|
||||
```
|
||||
|
||||
### Structural Tag
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
text = tokenizer.apply_chat_template(
|
||||
messages, tokenize=False, add_generation_prompt=True, return_dict=False
|
||||
)
|
||||
prompts = [text]
|
||||
|
||||
|
||||
sampling_params = {
|
||||
"temperature": 0.8,
|
||||
"top_p": 0.95,
|
||||
"structural_tag": json.dumps(
|
||||
{
|
||||
"type": "structural_tag",
|
||||
"structures": [
|
||||
{
|
||||
"begin": "<function=get_current_weather>",
|
||||
"schema": schema_get_current_weather,
|
||||
"end": "</function>",
|
||||
},
|
||||
{
|
||||
"begin": "<function=get_current_date>",
|
||||
"schema": schema_get_current_date,
|
||||
"end": "</function>",
|
||||
},
|
||||
],
|
||||
"triggers": ["<function="],
|
||||
}
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
# Send POST request to the API endpoint
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
for prompt, output in zip(prompts, outputs):
|
||||
print_highlight("===============================")
|
||||
print_highlight(f"Prompt: {prompt}\nGenerated text: {output['text']}")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
# Support for XGrammar latest structural tag format
|
||||
# https://xgrammar.mlc.ai/docs/tutorials/structural_tag.html
|
||||
|
||||
sampling_params = {
|
||||
"temperature": 0.8,
|
||||
"top_p": 0.95,
|
||||
"structural_tag": json.dumps(
|
||||
{
|
||||
"type": "structural_tag",
|
||||
"format": {
|
||||
"type": "triggered_tags",
|
||||
"triggers": ["<function="],
|
||||
"tags": [
|
||||
{
|
||||
"begin": "<function=get_current_weather>",
|
||||
"content": {
|
||||
"type": "json_schema",
|
||||
"json_schema": schema_get_current_weather,
|
||||
},
|
||||
"end": "</function>",
|
||||
},
|
||||
{
|
||||
"begin": "<function=get_current_date>",
|
||||
"content": {
|
||||
"type": "json_schema",
|
||||
"json_schema": schema_get_current_date,
|
||||
},
|
||||
"end": "</function>",
|
||||
},
|
||||
],
|
||||
"at_least_one": False,
|
||||
"stop_after_first": False,
|
||||
},
|
||||
}
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
# Send POST request to the API endpoint
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
for prompt, output in zip(prompts, outputs):
|
||||
print_highlight("===============================")
|
||||
print_highlight(f"Prompt: {prompt}\nGenerated text: {output['text']}")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
llm.shutdown()
|
||||
```
|
||||
@@ -0,0 +1,841 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Structured Outputs For Reasoning Models\n",
|
||||
"\n",
|
||||
"When working with reasoning models that use special tokens like `<think>...</think>` to denote reasoning sections, you might want to allow free-form text within these sections while still enforcing grammar constraints on the rest of the output.\n",
|
||||
"\n",
|
||||
"SGLang provides a feature to disable grammar restrictions within reasoning sections. This is particularly useful for models that need to perform complex reasoning steps before providing a structured output.\n",
|
||||
"\n",
|
||||
"To enable this feature, use the `--reasoning-parser` flag which decide the think_end_token, such as `</think>`, when launching the server. You can also specify the reasoning parser using the `--reasoning-parser` flag.\n",
|
||||
"\n",
|
||||
"## Supported Models\n",
|
||||
"\n",
|
||||
"Currently, SGLang supports the following reasoning models:\n",
|
||||
"- [DeepSeek R1 series](https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d): The reasoning content is wrapped with `<think>` and `</think>` tags.\n",
|
||||
"- [QwQ](https://huggingface.co/Qwen/QwQ-32B): The reasoning content is wrapped with `<think>` and `</think>` tags.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Usage\n",
|
||||
"\n",
|
||||
"## OpenAI Compatible API"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Specify the `--grammar-backend`, `--reasoning-parser` option."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import openai\n",
|
||||
"import os\n",
|
||||
"\n",
|
||||
"from sglang.test.doc_patch import launch_server_cmd\n",
|
||||
"from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
|
||||
"\n",
|
||||
"os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"server_process, port = launch_server_cmd(\n",
|
||||
" \"python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --host 0.0.0.0 --reasoning-parser deepseek-r1 --log-level warning\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"wait_for_server(f\"http://localhost:{port}\")\n",
|
||||
"client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### JSON\n",
|
||||
"\n",
|
||||
"you can directly define a JSON schema or use [Pydantic](https://docs.pydantic.dev/latest/) to define and validate the response."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Using Pydantic**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from pydantic import BaseModel, Field\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Define the schema using Pydantic\n",
|
||||
"class CapitalInfo(BaseModel):\n",
|
||||
" name: str = Field(..., pattern=r\"^\\w+$\", description=\"Name of the capital city\")\n",
|
||||
" population: int = Field(..., description=\"Population of the capital city\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"deepseek-ai/DeepSeek-R1-Distill-Qwen-7B\",\n",
|
||||
" messages=[\n",
|
||||
" {\n",
|
||||
" \"role\": \"assistant\",\n",
|
||||
" \"content\": \"Give me the information and population of the capital of France in the JSON format.\",\n",
|
||||
" },\n",
|
||||
" ],\n",
|
||||
" temperature=0,\n",
|
||||
" max_tokens=2048,\n",
|
||||
" response_format={\n",
|
||||
" \"type\": \"json_schema\",\n",
|
||||
" \"json_schema\": {\n",
|
||||
" \"name\": \"foo\",\n",
|
||||
" # convert the pydantic model to json schema\n",
|
||||
" \"schema\": CapitalInfo.model_json_schema(),\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(\n",
|
||||
" f\"reasoing_content: {response.choices[0].message.reasoning_content}\\n\\ncontent: {response.choices[0].message.content}\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**JSON Schema Directly**\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"\n",
|
||||
"json_schema = json.dumps(\n",
|
||||
" {\n",
|
||||
" \"type\": \"object\",\n",
|
||||
" \"properties\": {\n",
|
||||
" \"name\": {\"type\": \"string\", \"pattern\": \"^[\\\\w]+$\"},\n",
|
||||
" \"population\": {\"type\": \"integer\"},\n",
|
||||
" },\n",
|
||||
" \"required\": [\"name\", \"population\"],\n",
|
||||
" }\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"deepseek-ai/DeepSeek-R1-Distill-Qwen-7B\",\n",
|
||||
" messages=[\n",
|
||||
" {\n",
|
||||
" \"role\": \"assistant\",\n",
|
||||
" \"content\": \"Give me the information and population of the capital of France in the JSON format.\",\n",
|
||||
" },\n",
|
||||
" ],\n",
|
||||
" temperature=0,\n",
|
||||
" max_tokens=2048,\n",
|
||||
" response_format={\n",
|
||||
" \"type\": \"json_schema\",\n",
|
||||
" \"json_schema\": {\"name\": \"foo\", \"schema\": json.loads(json_schema)},\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(\n",
|
||||
" f\"reasoing_content: {response.choices[0].message.reasoning_content}\\n\\ncontent: {response.choices[0].message.content}\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### EBNF"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"ebnf_grammar = \"\"\"\n",
|
||||
"root ::= city | description\n",
|
||||
"city ::= \"London\" | \"Paris\" | \"Berlin\" | \"Rome\"\n",
|
||||
"description ::= city \" is \" status\n",
|
||||
"status ::= \"the capital of \" country\n",
|
||||
"country ::= \"England\" | \"France\" | \"Germany\" | \"Italy\"\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"deepseek-ai/DeepSeek-R1-Distill-Qwen-7B\",\n",
|
||||
" messages=[\n",
|
||||
" {\"role\": \"system\", \"content\": \"You are a helpful geography bot.\"},\n",
|
||||
" {\n",
|
||||
" \"role\": \"assistant\",\n",
|
||||
" \"content\": \"Give me the information and population of the capital of France in the JSON format.\",\n",
|
||||
" },\n",
|
||||
" ],\n",
|
||||
" temperature=0,\n",
|
||||
" max_tokens=2048,\n",
|
||||
" extra_body={\"ebnf\": ebnf_grammar},\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(\n",
|
||||
" f\"reasoing_content: {response.choices[0].message.reasoning_content}\\n\\ncontent: {response.choices[0].message.content}\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Regular expression"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"deepseek-ai/DeepSeek-R1-Distill-Qwen-7B\",\n",
|
||||
" messages=[\n",
|
||||
" {\"role\": \"assistant\", \"content\": \"What is the capital of France?\"},\n",
|
||||
" ],\n",
|
||||
" temperature=0,\n",
|
||||
" max_tokens=2048,\n",
|
||||
" extra_body={\"regex\": \"(Paris|London)\"},\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(\n",
|
||||
" f\"reasoing_content: {response.choices[0].message.reasoning_content}\\n\\ncontent: {response.choices[0].message.content}\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Structural Tag"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"tool_get_current_weather = {\n",
|
||||
" \"type\": \"function\",\n",
|
||||
" \"function\": {\n",
|
||||
" \"name\": \"get_current_weather\",\n",
|
||||
" \"description\": \"Get the current weather in a given location\",\n",
|
||||
" \"parameters\": {\n",
|
||||
" \"type\": \"object\",\n",
|
||||
" \"properties\": {\n",
|
||||
" \"city\": {\n",
|
||||
" \"type\": \"string\",\n",
|
||||
" \"description\": \"The city to find the weather for, e.g. 'San Francisco'\",\n",
|
||||
" },\n",
|
||||
" \"state\": {\n",
|
||||
" \"type\": \"string\",\n",
|
||||
" \"description\": \"the two-letter abbreviation for the state that the city is\"\n",
|
||||
" \" in, e.g. 'CA' which would mean 'California'\",\n",
|
||||
" },\n",
|
||||
" \"unit\": {\n",
|
||||
" \"type\": \"string\",\n",
|
||||
" \"description\": \"The unit to fetch the temperature in\",\n",
|
||||
" \"enum\": [\"celsius\", \"fahrenheit\"],\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
" \"required\": [\"city\", \"state\", \"unit\"],\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"tool_get_current_date = {\n",
|
||||
" \"type\": \"function\",\n",
|
||||
" \"function\": {\n",
|
||||
" \"name\": \"get_current_date\",\n",
|
||||
" \"description\": \"Get the current date and time for a given timezone\",\n",
|
||||
" \"parameters\": {\n",
|
||||
" \"type\": \"object\",\n",
|
||||
" \"properties\": {\n",
|
||||
" \"timezone\": {\n",
|
||||
" \"type\": \"string\",\n",
|
||||
" \"description\": \"The timezone to fetch the current date and time for, e.g. 'America/New_York'\",\n",
|
||||
" }\n",
|
||||
" },\n",
|
||||
" \"required\": [\"timezone\"],\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"schema_get_current_weather = tool_get_current_weather[\"function\"][\"parameters\"]\n",
|
||||
"schema_get_current_date = tool_get_current_date[\"function\"][\"parameters\"]\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def get_messages():\n",
|
||||
" return [\n",
|
||||
" {\n",
|
||||
" \"role\": \"system\",\n",
|
||||
" \"content\": f\"\"\"\n",
|
||||
"# Tool Instructions\n",
|
||||
"- Always execute python code in messages that you share.\n",
|
||||
"- When looking for real time information use relevant functions if available else fallback to brave_search\n",
|
||||
"You have access to the following functions:\n",
|
||||
"Use the function 'get_current_weather' to: Get the current weather in a given location\n",
|
||||
"{tool_get_current_weather[\"function\"]}\n",
|
||||
"Use the function 'get_current_date' to: Get the current date and time for a given timezone\n",
|
||||
"{tool_get_current_date[\"function\"]}\n",
|
||||
"If a you choose to call a function ONLY reply in the following format:\n",
|
||||
"<{{start_tag}}={{function_name}}>{{parameters}}{{end_tag}}\n",
|
||||
"where\n",
|
||||
"start_tag => `<function`\n",
|
||||
"parameters => a JSON dict with the function argument name as key and function argument value as value.\n",
|
||||
"end_tag => `</function>`\n",
|
||||
"Here is an example,\n",
|
||||
"<function=example_function_name>{{\"example_name\": \"example_value\"}}</function>\n",
|
||||
"Reminder:\n",
|
||||
"- Function calls MUST follow the specified format\n",
|
||||
"- Required parameters MUST be specified\n",
|
||||
"- Only call one function at a time\n",
|
||||
"- Put the entire function call reply on one line\n",
|
||||
"- Always add your sources when using search results to answer the user query\n",
|
||||
"You are a helpful assistant.\"\"\",\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"role\": \"assistant\",\n",
|
||||
" \"content\": \"You are in New York. Please get the current date and time, and the weather.\",\n",
|
||||
" },\n",
|
||||
" ]\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"messages = get_messages()\n",
|
||||
"\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"deepseek-ai/DeepSeek-R1-Distill-Qwen-7B\",\n",
|
||||
" messages=messages,\n",
|
||||
" response_format={\n",
|
||||
" \"type\": \"structural_tag\",\n",
|
||||
" \"max_new_tokens\": 2048,\n",
|
||||
" \"structures\": [\n",
|
||||
" {\n",
|
||||
" \"begin\": \"<function=get_current_weather>\",\n",
|
||||
" \"schema\": schema_get_current_weather,\n",
|
||||
" \"end\": \"</function>\",\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"begin\": \"<function=get_current_date>\",\n",
|
||||
" \"schema\": schema_get_current_date,\n",
|
||||
" \"end\": \"</function>\",\n",
|
||||
" },\n",
|
||||
" ],\n",
|
||||
" \"triggers\": [\"<function=\"],\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(\n",
|
||||
" f\"reasoing_content: {response.choices[0].message.reasoning_content}\\n\\ncontent: {response.choices[0].message.content}\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Native API and SGLang Runtime (SRT)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> Note: For native API, as a work-around, you need to set `require_reasoning` argument to `True` to ensure the model will think before generating the structured output. It's not required for chat-completion API."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### JSON"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Using Pydantic**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import requests\n",
|
||||
"from pydantic import BaseModel, Field\n",
|
||||
"from transformers import AutoTokenizer\n",
|
||||
"\n",
|
||||
"tokenizer = AutoTokenizer.from_pretrained(\"deepseek-ai/DeepSeek-R1-Distill-Qwen-7B\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Define the schema using Pydantic\n",
|
||||
"class CapitalInfo(BaseModel):\n",
|
||||
" name: str = Field(..., pattern=r\"^\\w+$\", description=\"Name of the capital city\")\n",
|
||||
" population: int = Field(..., description=\"Population of the capital city\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"messages = [\n",
|
||||
" {\n",
|
||||
" \"role\": \"assistant\",\n",
|
||||
" \"content\": \"Give me the information and population of the capital of France in the JSON format.\",\n",
|
||||
" },\n",
|
||||
"]\n",
|
||||
"text = tokenizer.apply_chat_template(\n",
|
||||
" messages, tokenize=False, add_generation_prompt=True, return_dict=False\n",
|
||||
")\n",
|
||||
"# Make API request\n",
|
||||
"response = requests.post(\n",
|
||||
" f\"http://localhost:{port}/generate\",\n",
|
||||
" json={\n",
|
||||
" \"text\": text,\n",
|
||||
" \"require_reasoning\": True,\n",
|
||||
" \"sampling_params\": {\n",
|
||||
" \"temperature\": 0,\n",
|
||||
" \"max_new_tokens\": 2048,\n",
|
||||
" \"json_schema\": json.dumps(CapitalInfo.model_json_schema()),\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"print(response.json())\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"reasoing_content = response.json()[\"text\"].split(\"</think>\")[0]\n",
|
||||
"content = response.json()[\"text\"].split(\"</think>\")[1]\n",
|
||||
"print_highlight(f\"reasoing_content: {reasoing_content}\\n\\ncontent: {content}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**JSON Schema Directly**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"json_schema = json.dumps(\n",
|
||||
" {\n",
|
||||
" \"type\": \"object\",\n",
|
||||
" \"properties\": {\n",
|
||||
" \"name\": {\"type\": \"string\", \"pattern\": \"^[\\\\w]+$\"},\n",
|
||||
" \"population\": {\"type\": \"integer\"},\n",
|
||||
" },\n",
|
||||
" \"required\": [\"name\", \"population\"],\n",
|
||||
" }\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# JSON\n",
|
||||
"text = tokenizer.apply_chat_template(\n",
|
||||
" messages, tokenize=False, add_generation_prompt=True, return_dict=False\n",
|
||||
")\n",
|
||||
"response = requests.post(\n",
|
||||
" f\"http://localhost:{port}/generate\",\n",
|
||||
" json={\n",
|
||||
" \"text\": text,\n",
|
||||
" \"require_reasoning\": True,\n",
|
||||
" \"sampling_params\": {\n",
|
||||
" \"temperature\": 0,\n",
|
||||
" \"max_new_tokens\": 2048,\n",
|
||||
" \"json_schema\": json_schema,\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(response.json())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### EBNF"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"response = requests.post(\n",
|
||||
" f\"http://localhost:{port}/generate\",\n",
|
||||
" json={\n",
|
||||
" \"text\": \"Give me the information of the capital of France.\",\n",
|
||||
" \"require_reasoning\": True,\n",
|
||||
" \"sampling_params\": {\n",
|
||||
" \"max_new_tokens\": 2048,\n",
|
||||
" \"temperature\": 0,\n",
|
||||
" \"n\": 3,\n",
|
||||
" \"ebnf\": (\n",
|
||||
" \"root ::= city | description\\n\"\n",
|
||||
" 'city ::= \"London\" | \"Paris\" | \"Berlin\" | \"Rome\"\\n'\n",
|
||||
" 'description ::= city \" is \" status\\n'\n",
|
||||
" 'status ::= \"the capital of \" country\\n'\n",
|
||||
" 'country ::= \"England\" | \"France\" | \"Germany\" | \"Italy\"'\n",
|
||||
" ),\n",
|
||||
" },\n",
|
||||
" \"stream\": False,\n",
|
||||
" \"return_logprob\": False,\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(response.json())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Regular expression"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"response = requests.post(\n",
|
||||
" f\"http://localhost:{port}/generate\",\n",
|
||||
" json={\n",
|
||||
" \"text\": \"Paris is the capital of\",\n",
|
||||
" \"require_reasoning\": True,\n",
|
||||
" \"sampling_params\": {\n",
|
||||
" \"temperature\": 0,\n",
|
||||
" \"max_new_tokens\": 2048,\n",
|
||||
" \"regex\": \"(France|England)\",\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"print(response.json())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Structural Tag"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"text = tokenizer.apply_chat_template(\n",
|
||||
" messages, tokenize=False, add_generation_prompt=True, return_dict=False\n",
|
||||
")\n",
|
||||
"payload = {\n",
|
||||
" \"text\": text,\n",
|
||||
" \"require_reasoning\": True,\n",
|
||||
" \"sampling_params\": {\n",
|
||||
" \"max_new_tokens\": 2048,\n",
|
||||
" \"structural_tag\": json.dumps(\n",
|
||||
" {\n",
|
||||
" \"type\": \"structural_tag\",\n",
|
||||
" \"structures\": [\n",
|
||||
" {\n",
|
||||
" \"begin\": \"<function=get_current_weather>\",\n",
|
||||
" \"schema\": schema_get_current_weather,\n",
|
||||
" \"end\": \"</function>\",\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"begin\": \"<function=get_current_date>\",\n",
|
||||
" \"schema\": schema_get_current_date,\n",
|
||||
" \"end\": \"</function>\",\n",
|
||||
" },\n",
|
||||
" ],\n",
|
||||
" \"triggers\": [\"<function=\"],\n",
|
||||
" }\n",
|
||||
" ),\n",
|
||||
" },\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Send POST request to the API endpoint\n",
|
||||
"response = requests.post(f\"http://localhost:{port}/generate\", json=payload)\n",
|
||||
"print_highlight(response.json())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(server_process)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Offline Engine API"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sglang as sgl\n",
|
||||
"\n",
|
||||
"llm = sgl.Engine(\n",
|
||||
" model_path=\"deepseek-ai/DeepSeek-R1-Distill-Qwen-7B\",\n",
|
||||
" reasoning_parser=\"deepseek-r1\",\n",
|
||||
" grammar_backend=\"xgrammar\",\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### JSON"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Using Pydantic**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"from pydantic import BaseModel, Field\n",
|
||||
"\n",
|
||||
"prompts = [\n",
|
||||
" \"Give me the information of the capital of China in the JSON format.\",\n",
|
||||
" \"Give me the information of the capital of France in the JSON format.\",\n",
|
||||
" \"Give me the information of the capital of Ireland in the JSON format.\",\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Define the schema using Pydantic\n",
|
||||
"class CapitalInfo(BaseModel):\n",
|
||||
" name: str = Field(..., pattern=r\"^\\w+$\", description=\"Name of the capital city\")\n",
|
||||
" population: int = Field(..., description=\"Population of the capital city\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"sampling_params = {\n",
|
||||
" \"temperature\": 0,\n",
|
||||
" \"top_p\": 0.95,\n",
|
||||
" \"max_new_tokens\": 2048,\n",
|
||||
" \"json_schema\": json.dumps(CapitalInfo.model_json_schema()),\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"outputs = llm.generate(prompts, sampling_params)\n",
|
||||
"for prompt, output in zip(prompts, outputs):\n",
|
||||
" print(\"===============================\")\n",
|
||||
" print(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**JSON Schema Directly**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"prompts = [\n",
|
||||
" \"Give me the information of the capital of China in the JSON format.\",\n",
|
||||
" \"Give me the information of the capital of France in the JSON format.\",\n",
|
||||
" \"Give me the information of the capital of Ireland in the JSON format.\",\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"json_schema = json.dumps(\n",
|
||||
" {\n",
|
||||
" \"type\": \"object\",\n",
|
||||
" \"properties\": {\n",
|
||||
" \"name\": {\"type\": \"string\", \"pattern\": \"^[\\\\w]+$\"},\n",
|
||||
" \"population\": {\"type\": \"integer\"},\n",
|
||||
" },\n",
|
||||
" \"required\": [\"name\", \"population\"],\n",
|
||||
" }\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"sampling_params = {\"temperature\": 0, \"max_new_tokens\": 2048, \"json_schema\": json_schema}\n",
|
||||
"\n",
|
||||
"outputs = llm.generate(prompts, sampling_params)\n",
|
||||
"for prompt, output in zip(prompts, outputs):\n",
|
||||
" print(\"===============================\")\n",
|
||||
" print(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### EBNF\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"prompts = [\n",
|
||||
" \"Give me the information of the capital of France.\",\n",
|
||||
" \"Give me the information of the capital of Germany.\",\n",
|
||||
" \"Give me the information of the capital of Italy.\",\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"sampling_params = {\n",
|
||||
" \"temperature\": 0.8,\n",
|
||||
" \"top_p\": 0.95,\n",
|
||||
" \"ebnf\": (\n",
|
||||
" \"root ::= city | description\\n\"\n",
|
||||
" 'city ::= \"London\" | \"Paris\" | \"Berlin\" | \"Rome\"\\n'\n",
|
||||
" 'description ::= city \" is \" status\\n'\n",
|
||||
" 'status ::= \"the capital of \" country\\n'\n",
|
||||
" 'country ::= \"England\" | \"France\" | \"Germany\" | \"Italy\"'\n",
|
||||
" ),\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"outputs = llm.generate(prompts, sampling_params)\n",
|
||||
"for prompt, output in zip(prompts, outputs):\n",
|
||||
" print(\"===============================\")\n",
|
||||
" print(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Regular expression"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"prompts = [\n",
|
||||
" \"Please provide information about London as a major global city:\",\n",
|
||||
" \"Please provide information about Paris as a major global city:\",\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"sampling_params = {\"temperature\": 0.8, \"top_p\": 0.95, \"regex\": \"(France|England)\"}\n",
|
||||
"\n",
|
||||
"outputs = llm.generate(prompts, sampling_params)\n",
|
||||
"for prompt, output in zip(prompts, outputs):\n",
|
||||
" print(\"===============================\")\n",
|
||||
" print(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"text = tokenizer.apply_chat_template(\n",
|
||||
" messages, tokenize=False, add_generation_prompt=True, return_dict=False\n",
|
||||
")\n",
|
||||
"prompts = [text]\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"sampling_params = {\n",
|
||||
" \"temperature\": 0.8,\n",
|
||||
" \"top_p\": 0.95,\n",
|
||||
" \"max_new_tokens\": 2048,\n",
|
||||
" \"structural_tag\": json.dumps(\n",
|
||||
" {\n",
|
||||
" \"type\": \"structural_tag\",\n",
|
||||
" \"structures\": [\n",
|
||||
" {\n",
|
||||
" \"begin\": \"<function=get_current_weather>\",\n",
|
||||
" \"schema\": schema_get_current_weather,\n",
|
||||
" \"end\": \"</function>\",\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"begin\": \"<function=get_current_date>\",\n",
|
||||
" \"schema\": schema_get_current_date,\n",
|
||||
" \"end\": \"</function>\",\n",
|
||||
" },\n",
|
||||
" ],\n",
|
||||
" \"triggers\": [\"<function=\"],\n",
|
||||
" }\n",
|
||||
" ),\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Send POST request to the API endpoint\n",
|
||||
"outputs = llm.generate(prompts, sampling_params)\n",
|
||||
"for prompt, output in zip(prompts, outputs):\n",
|
||||
" print(\"===============================\")\n",
|
||||
" print(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"llm.shutdown()"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,663 @@
|
||||
---
|
||||
title: "Structured Outputs For Reasoning Models"
|
||||
metatags:
|
||||
description: "SGLang structured outputs for reasoning models: free-form thinking with constrained final output for DeepSeek R1, QwQ models."
|
||||
---
|
||||
When working with reasoning models that use special tokens like `<think>...</think>` to denote reasoning sections, you might want to allow free-form text within these sections while still enforcing grammar constraints on the rest of the output.
|
||||
|
||||
SGLang provides a feature to disable grammar restrictions within reasoning sections. This is particularly useful for models that need to perform complex reasoning steps before providing a structured output.
|
||||
|
||||
To enable this feature, use the `--reasoning-parser` flag which decide the think_end_token, such as `</think>`, when launching the server. You can also specify the reasoning parser using the `--reasoning-parser` flag.
|
||||
|
||||
## Supported Models
|
||||
|
||||
Currently, SGLang supports the following reasoning models:
|
||||
- [DeepSeek R1 series](https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d): The reasoning content is wrapped with `<think>` and `</think>` tags.
|
||||
- [QwQ](https://huggingface.co/Qwen/QwQ-32B): The reasoning content is wrapped with `<think>` and `</think>` tags.
|
||||
|
||||
|
||||
## Usage
|
||||
|
||||
## OpenAI Compatible API
|
||||
|
||||
|
||||
Specify the `--grammar-backend`, `--reasoning-parser` option.
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
import os
|
||||
|
||||
from sglang.test.doc_patch import launch_server_cmd
|
||||
from sglang.utils import wait_for_server, print_highlight, terminate_process
|
||||
|
||||
os.environ["TOKENIZERS_PARALLELISM"] = "false"
|
||||
|
||||
|
||||
server_process, port = launch_server_cmd(
|
||||
"python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --host 0.0.0.0 --reasoning-parser deepseek-r1 --log-level warning"
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")
|
||||
```
|
||||
|
||||
### JSON
|
||||
|
||||
you can directly define a JSON schema or use [Pydantic](https://docs.pydantic.dev/latest/) to define and validate the response.
|
||||
|
||||
|
||||
**Using Pydantic**
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
# Define the schema using Pydantic
|
||||
class CapitalInfo(BaseModel):
|
||||
name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city")
|
||||
population: int = Field(..., description="Population of the capital city")
|
||||
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
|
||||
messages=[
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Give me the information and population of the capital of France in the JSON format.",
|
||||
},
|
||||
],
|
||||
temperature=0,
|
||||
max_tokens=2048,
|
||||
response_format={
|
||||
"type": "json_schema",
|
||||
"json_schema": {
|
||||
"name": "foo",
|
||||
# convert the pydantic model to json schema
|
||||
"schema": CapitalInfo.model_json_schema(),
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
print_highlight(
|
||||
f"reasoing_content: {response.choices[0].message.reasoning_content}\n\ncontent: {response.choices[0].message.content}"
|
||||
)
|
||||
```
|
||||
|
||||
**JSON Schema Directly**
|
||||
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import json
|
||||
|
||||
json_schema = json.dumps(
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": {"type": "string", "pattern": "^[\\w]+$"},
|
||||
"population": {"type": "integer"},
|
||||
},
|
||||
"required": ["name", "population"],
|
||||
}
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
|
||||
messages=[
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Give me the information and population of the capital of France in the JSON format.",
|
||||
},
|
||||
],
|
||||
temperature=0,
|
||||
max_tokens=2048,
|
||||
response_format={
|
||||
"type": "json_schema",
|
||||
"json_schema": {"name": "foo", "schema": json.loads(json_schema)},
|
||||
},
|
||||
)
|
||||
|
||||
print_highlight(
|
||||
f"reasoing_content: {response.choices[0].message.reasoning_content}\n\ncontent: {response.choices[0].message.content}"
|
||||
)
|
||||
```
|
||||
|
||||
### EBNF
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
ebnf_grammar = """
|
||||
root ::= city | description
|
||||
city ::= "London" | "Paris" | "Berlin" | "Rome"
|
||||
description ::= city " is " status
|
||||
status ::= "the capital of " country
|
||||
country ::= "England" | "France" | "Germany" | "Italy"
|
||||
"""
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful geography bot."},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Give me the information and population of the capital of France in the JSON format.",
|
||||
},
|
||||
],
|
||||
temperature=0,
|
||||
max_tokens=2048,
|
||||
extra_body={"ebnf": ebnf_grammar},
|
||||
)
|
||||
|
||||
print_highlight(
|
||||
f"reasoing_content: {response.choices[0].message.reasoning_content}\n\ncontent: {response.choices[0].message.content}"
|
||||
)
|
||||
```
|
||||
|
||||
### Regular expression
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
|
||||
messages=[
|
||||
{"role": "assistant", "content": "What is the capital of France?"},
|
||||
],
|
||||
temperature=0,
|
||||
max_tokens=2048,
|
||||
extra_body={"regex": "(Paris|London)"},
|
||||
)
|
||||
|
||||
print_highlight(
|
||||
f"reasoing_content: {response.choices[0].message.reasoning_content}\n\ncontent: {response.choices[0].message.content}"
|
||||
)
|
||||
```
|
||||
|
||||
### Structural Tag
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
tool_get_current_weather = {
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_current_weather",
|
||||
"description": "Get the current weather in a given location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"city": {
|
||||
"type": "string",
|
||||
"description": "The city to find the weather for, e.g. 'San Francisco'",
|
||||
},
|
||||
"state": {
|
||||
"type": "string",
|
||||
"description": "the two-letter abbreviation for the state that the city is"
|
||||
" in, e.g. 'CA' which would mean 'California'",
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"description": "The unit to fetch the temperature in",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
},
|
||||
},
|
||||
"required": ["city", "state", "unit"],
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
tool_get_current_date = {
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_current_date",
|
||||
"description": "Get the current date and time for a given timezone",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"timezone": {
|
||||
"type": "string",
|
||||
"description": "The timezone to fetch the current date and time for, e.g. 'America/New_York'",
|
||||
}
|
||||
},
|
||||
"required": ["timezone"],
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
schema_get_current_weather = tool_get_current_weather["function"]["parameters"]
|
||||
schema_get_current_date = tool_get_current_date["function"]["parameters"]
|
||||
|
||||
|
||||
def get_messages():
|
||||
return [
|
||||
{
|
||||
"role": "system",
|
||||
"content": f"""
|
||||
# Tool Instructions
|
||||
- Always execute python code in messages that you share.
|
||||
- When looking for real time information use relevant functions if available else fallback to brave_search
|
||||
You have access to the following functions:
|
||||
Use the function 'get_current_weather' to: Get the current weather in a given location
|
||||
{tool_get_current_weather["function"]}
|
||||
Use the function 'get_current_date' to: Get the current date and time for a given timezone
|
||||
{tool_get_current_date["function"]}
|
||||
If a you choose to call a function ONLY reply in the following format:
|
||||
<{{start_tag}}={{function_name}}>{{parameters}}{{end_tag}}
|
||||
where
|
||||
start_tag => `<function`
|
||||
parameters => a JSON dict with the function argument name as key and function argument value as value.
|
||||
end_tag => `</function>`
|
||||
Here is an example,
|
||||
<function=example_function_name>{{"example_name": "example_value"}}</function>
|
||||
Reminder:
|
||||
- Function calls MUST follow the specified format
|
||||
- Required parameters MUST be specified
|
||||
- Only call one function at a time
|
||||
- Put the entire function call reply on one line
|
||||
- Always add your sources when using search results to answer the user query
|
||||
You are a helpful assistant.""",
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "You are in New York. Please get the current date and time, and the weather.",
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
messages = get_messages()
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
|
||||
messages=messages,
|
||||
response_format={
|
||||
"type": "structural_tag",
|
||||
"max_new_tokens": 2048,
|
||||
"structures": [
|
||||
{
|
||||
"begin": "<function=get_current_weather>",
|
||||
"schema": schema_get_current_weather,
|
||||
"end": "</function>",
|
||||
},
|
||||
{
|
||||
"begin": "<function=get_current_date>",
|
||||
"schema": schema_get_current_date,
|
||||
"end": "</function>",
|
||||
},
|
||||
],
|
||||
"triggers": ["<function="],
|
||||
},
|
||||
)
|
||||
|
||||
print_highlight(
|
||||
f"reasoing_content: {response.choices[0].message.reasoning_content}\n\ncontent: {response.choices[0].message.content}"
|
||||
)
|
||||
```
|
||||
|
||||
## Native API and SGLang Runtime (SRT)
|
||||
|
||||
|
||||
> Note: For native API, as a work-around, you need to set `require_reasoning` argument to `True` to ensure the model will think before generating the structured output. It's not required for chat-completion API.
|
||||
|
||||
|
||||
### JSON
|
||||
|
||||
|
||||
**Using Pydantic**
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
from pydantic import BaseModel, Field
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-7B")
|
||||
|
||||
|
||||
# Define the schema using Pydantic
|
||||
class CapitalInfo(BaseModel):
|
||||
name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city")
|
||||
population: int = Field(..., description="Population of the capital city")
|
||||
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Give me the information and population of the capital of France in the JSON format.",
|
||||
},
|
||||
]
|
||||
text = tokenizer.apply_chat_template(
|
||||
messages, tokenize=False, add_generation_prompt=True, return_dict=False
|
||||
)
|
||||
# Make API request
|
||||
response = requests.post(
|
||||
f"http://localhost:{port}/generate",
|
||||
json={
|
||||
"text": text,
|
||||
"require_reasoning": True,
|
||||
"sampling_params": {
|
||||
"temperature": 0,
|
||||
"max_new_tokens": 2048,
|
||||
"json_schema": json.dumps(CapitalInfo.model_json_schema()),
|
||||
},
|
||||
},
|
||||
)
|
||||
print(response.json())
|
||||
|
||||
|
||||
reasoing_content = response.json()["text"].split("</think>")[0]
|
||||
content = response.json()["text"].split("</think>")[1]
|
||||
print_highlight(f"reasoing_content: {reasoing_content}\n\ncontent: {content}")
|
||||
```
|
||||
|
||||
**JSON Schema Directly**
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
json_schema = json.dumps(
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": {"type": "string", "pattern": "^[\\w]+$"},
|
||||
"population": {"type": "integer"},
|
||||
},
|
||||
"required": ["name", "population"],
|
||||
}
|
||||
)
|
||||
|
||||
# JSON
|
||||
text = tokenizer.apply_chat_template(
|
||||
messages, tokenize=False, add_generation_prompt=True, return_dict=False
|
||||
)
|
||||
response = requests.post(
|
||||
f"http://localhost:{port}/generate",
|
||||
json={
|
||||
"text": text,
|
||||
"require_reasoning": True,
|
||||
"sampling_params": {
|
||||
"temperature": 0,
|
||||
"max_new_tokens": 2048,
|
||||
"json_schema": json_schema,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
print_highlight(response.json())
|
||||
```
|
||||
|
||||
### EBNF
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
response = requests.post(
|
||||
f"http://localhost:{port}/generate",
|
||||
json={
|
||||
"text": "Give me the information of the capital of France.",
|
||||
"require_reasoning": True,
|
||||
"sampling_params": {
|
||||
"max_new_tokens": 2048,
|
||||
"temperature": 0,
|
||||
"n": 3,
|
||||
"ebnf": (
|
||||
"root ::= city | description\n"
|
||||
'city ::= "London" | "Paris" | "Berlin" | "Rome"\n'
|
||||
'description ::= city " is " status\n'
|
||||
'status ::= "the capital of " country\n'
|
||||
'country ::= "England" | "France" | "Germany" | "Italy"'
|
||||
),
|
||||
},
|
||||
"stream": False,
|
||||
"return_logprob": False,
|
||||
},
|
||||
)
|
||||
|
||||
print(response.json())
|
||||
```
|
||||
|
||||
### Regular expression
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
response = requests.post(
|
||||
f"http://localhost:{port}/generate",
|
||||
json={
|
||||
"text": "Paris is the capital of",
|
||||
"require_reasoning": True,
|
||||
"sampling_params": {
|
||||
"temperature": 0,
|
||||
"max_new_tokens": 2048,
|
||||
"regex": "(France|England)",
|
||||
},
|
||||
},
|
||||
)
|
||||
print(response.json())
|
||||
```
|
||||
|
||||
### Structural Tag
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
text = tokenizer.apply_chat_template(
|
||||
messages, tokenize=False, add_generation_prompt=True, return_dict=False
|
||||
)
|
||||
payload = {
|
||||
"text": text,
|
||||
"require_reasoning": True,
|
||||
"sampling_params": {
|
||||
"max_new_tokens": 2048,
|
||||
"structural_tag": json.dumps(
|
||||
{
|
||||
"type": "structural_tag",
|
||||
"structures": [
|
||||
{
|
||||
"begin": "<function=get_current_weather>",
|
||||
"schema": schema_get_current_weather,
|
||||
"end": "</function>",
|
||||
},
|
||||
{
|
||||
"begin": "<function=get_current_date>",
|
||||
"schema": schema_get_current_date,
|
||||
"end": "</function>",
|
||||
},
|
||||
],
|
||||
"triggers": ["<function="],
|
||||
}
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
# Send POST request to the API endpoint
|
||||
response = requests.post(f"http://localhost:{port}/generate", json=payload)
|
||||
print_highlight(response.json())
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
terminate_process(server_process)
|
||||
```
|
||||
|
||||
## Offline Engine API
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import sglang as sgl
|
||||
|
||||
llm = sgl.Engine(
|
||||
model_path="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
|
||||
reasoning_parser="deepseek-r1",
|
||||
grammar_backend="xgrammar",
|
||||
)
|
||||
```
|
||||
|
||||
### JSON
|
||||
|
||||
|
||||
**Using Pydantic**
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import json
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
prompts = [
|
||||
"Give me the information of the capital of China in the JSON format.",
|
||||
"Give me the information of the capital of France in the JSON format.",
|
||||
"Give me the information of the capital of Ireland in the JSON format.",
|
||||
]
|
||||
|
||||
|
||||
# Define the schema using Pydantic
|
||||
class CapitalInfo(BaseModel):
|
||||
name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city")
|
||||
population: int = Field(..., description="Population of the capital city")
|
||||
|
||||
|
||||
sampling_params = {
|
||||
"temperature": 0,
|
||||
"top_p": 0.95,
|
||||
"max_new_tokens": 2048,
|
||||
"json_schema": json.dumps(CapitalInfo.model_json_schema()),
|
||||
}
|
||||
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
for prompt, output in zip(prompts, outputs):
|
||||
print("===============================")
|
||||
print(f"Prompt: {prompt}\nGenerated text: {output['text']}")
|
||||
```
|
||||
|
||||
**JSON Schema Directly**
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
prompts = [
|
||||
"Give me the information of the capital of China in the JSON format.",
|
||||
"Give me the information of the capital of France in the JSON format.",
|
||||
"Give me the information of the capital of Ireland in the JSON format.",
|
||||
]
|
||||
|
||||
json_schema = json.dumps(
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": {"type": "string", "pattern": "^[\\w]+$"},
|
||||
"population": {"type": "integer"},
|
||||
},
|
||||
"required": ["name", "population"],
|
||||
}
|
||||
)
|
||||
|
||||
sampling_params = {"temperature": 0, "max_new_tokens": 2048, "json_schema": json_schema}
|
||||
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
for prompt, output in zip(prompts, outputs):
|
||||
print("===============================")
|
||||
print(f"Prompt: {prompt}\nGenerated text: {output['text']}")
|
||||
```
|
||||
|
||||
### EBNF
|
||||
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
prompts = [
|
||||
"Give me the information of the capital of France.",
|
||||
"Give me the information of the capital of Germany.",
|
||||
"Give me the information of the capital of Italy.",
|
||||
]
|
||||
|
||||
sampling_params = {
|
||||
"temperature": 0.8,
|
||||
"top_p": 0.95,
|
||||
"ebnf": (
|
||||
"root ::= city | description\n"
|
||||
'city ::= "London" | "Paris" | "Berlin" | "Rome"\n'
|
||||
'description ::= city " is " status\n'
|
||||
'status ::= "the capital of " country\n'
|
||||
'country ::= "England" | "France" | "Germany" | "Italy"'
|
||||
),
|
||||
}
|
||||
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
for prompt, output in zip(prompts, outputs):
|
||||
print("===============================")
|
||||
print(f"Prompt: {prompt}\nGenerated text: {output['text']}")
|
||||
```
|
||||
|
||||
### Regular expression
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
prompts = [
|
||||
"Please provide information about London as a major global city:",
|
||||
"Please provide information about Paris as a major global city:",
|
||||
]
|
||||
|
||||
sampling_params = {"temperature": 0.8, "top_p": 0.95, "regex": "(France|England)"}
|
||||
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
for prompt, output in zip(prompts, outputs):
|
||||
print("===============================")
|
||||
print(f"Prompt: {prompt}\nGenerated text: {output['text']}")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
text = tokenizer.apply_chat_template(
|
||||
messages, tokenize=False, add_generation_prompt=True, return_dict=False
|
||||
)
|
||||
prompts = [text]
|
||||
|
||||
|
||||
sampling_params = {
|
||||
"temperature": 0.8,
|
||||
"top_p": 0.95,
|
||||
"max_new_tokens": 2048,
|
||||
"structural_tag": json.dumps(
|
||||
{
|
||||
"type": "structural_tag",
|
||||
"structures": [
|
||||
{
|
||||
"begin": "<function=get_current_weather>",
|
||||
"schema": schema_get_current_weather,
|
||||
"end": "</function>",
|
||||
},
|
||||
{
|
||||
"begin": "<function=get_current_date>",
|
||||
"schema": schema_get_current_date,
|
||||
"end": "</function>",
|
||||
},
|
||||
],
|
||||
"triggers": ["<function="],
|
||||
}
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
# Send POST request to the API endpoint
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
for prompt, output in zip(prompts, outputs):
|
||||
print("===============================")
|
||||
print(f"Prompt: {prompt}\nGenerated text: {output['text']}")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
llm.shutdown()
|
||||
```
|
||||
@@ -0,0 +1,854 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Tool Parser\n",
|
||||
"\n",
|
||||
"This guide demonstrates how to use SGLang’s [Function calling](https://platform.openai.com/docs/guides/function-calling) functionality."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Currently supported parsers:\n",
|
||||
"\n",
|
||||
"| Parser | Supported Models | Notes |\n",
|
||||
"|---|---|---|\n",
|
||||
"| `deepseekv3` | DeepSeek-v3 (e.g., `deepseek-ai/DeepSeek-V3-0324`) | Recommend adding `--chat-template ./examples/chat_template/tool_chat_template_deepseekv3.jinja` to launch command. |\n",
|
||||
"| `deepseekv31` | DeepSeek-V3.1 and DeepSeek-V3.2-Exp (e.g. `deepseek-ai/DeepSeek-V3.1`, `deepseek-ai/DeepSeek-V3.2-Exp`) | Recommend adding `--chat-template ./examples/chat_template/tool_chat_template_deepseekv31.jinja` (Or ..deepseekv32.jinja for DeepSeek-V3.2) to launch command. |\n",
|
||||
"| `deepseekv32` | DeepSeek-V3.2 (`deepseek-ai/DeepSeek-V3.2`) | |\n",
|
||||
"| `glm` | GLM series (e.g. `zai-org/GLM-4.6`) | |\n",
|
||||
"| `gpt-oss` | GPT-OSS (e.g., `openai/gpt-oss-120b`, `openai/gpt-oss-20b`, `lmsys/gpt-oss-120b-bf16`, `lmsys/gpt-oss-20b-bf16`) | The gpt-oss tool parser filters out analysis channel events and only preserves normal text. This can cause the content to be empty when explanations are in the analysis channel. To work around this, complete the tool round by returning tool results as `role=\"tool\"` messages, which enables the model to generate the final content. |\n",
|
||||
"| `kimi_k2` | `moonshotai/Kimi-K2-Instruct` | |\n",
|
||||
"| `llama3` | Llama 3.1 / 3.2 / 3.3 (e.g. `meta-llama/Llama-3.1-8B-Instruct`, `meta-llama/Llama-3.2-1B-Instruct`, `meta-llama/Llama-3.3-70B-Instruct`) | |\n",
|
||||
"| `llama4` | Llama 4 (e.g. `meta-llama/Llama-4-Scout-17B-16E-Instruct`) | |\n",
|
||||
"| `mistral` | Mistral (e.g. `mistralai/Mistral-7B-Instruct-v0.3`, `mistralai/Mistral-Nemo-Instruct-2407`, `mistralai/Mistral-7B-v0.3`) | |\n",
|
||||
"| `pythonic` | Llama-3.2 / Llama-3.3 / Llama-4 | Model outputs function calls as Python code. Requires `--tool-call-parser pythonic` and is recommended to use with a specific chat template. |\n",
|
||||
"| `qwen` | Qwen series (e.g. `Qwen/Qwen3-Next-80B-A3B-Instruct`, `Qwen/Qwen3-VL-30B-A3B-Thinking`) except Qwen3-Coder| |\n",
|
||||
"| `qwen3_coder` | Qwen3-Coder (e.g. `Qwen/Qwen3-Coder-30B-A3B-Instruct`) | |\n",
|
||||
"| `step3` | Step-3 | |\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## OpenAI Compatible API"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Launching the Server"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"from sglang.test.doc_patch import launch_server_cmd\n",
|
||||
"from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
|
||||
"from openai import OpenAI\n",
|
||||
"\n",
|
||||
"server_process, port = launch_server_cmd(\n",
|
||||
" \"python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --tool-call-parser qwen25 --host 0.0.0.0 --log-level warning\" # qwen25\n",
|
||||
")\n",
|
||||
"wait_for_server(f\"http://localhost:{port}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Note that `--tool-call-parser` defines the parser used to interpret responses."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Define Tools for Function Call\n",
|
||||
"Below is a Python snippet that shows how to define a tool as a dictionary. The dictionary includes a tool name, a description, and property defined Parameters."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Define tools\n",
|
||||
"tools = [\n",
|
||||
" {\n",
|
||||
" \"type\": \"function\",\n",
|
||||
" \"function\": {\n",
|
||||
" \"name\": \"get_current_weather\",\n",
|
||||
" \"description\": \"Get the current weather in a given location\",\n",
|
||||
" \"parameters\": {\n",
|
||||
" \"type\": \"object\",\n",
|
||||
" \"properties\": {\n",
|
||||
" \"city\": {\n",
|
||||
" \"type\": \"string\",\n",
|
||||
" \"description\": \"The city to find the weather for, e.g. 'San Francisco'\",\n",
|
||||
" },\n",
|
||||
" \"state\": {\n",
|
||||
" \"type\": \"string\",\n",
|
||||
" \"description\": \"the two-letter abbreviation for the state that the city is\"\n",
|
||||
" \" in, e.g. 'CA' which would mean 'California'\",\n",
|
||||
" },\n",
|
||||
" \"unit\": {\n",
|
||||
" \"type\": \"string\",\n",
|
||||
" \"description\": \"The unit to fetch the temperature in\",\n",
|
||||
" \"enum\": [\"celsius\", \"fahrenheit\"],\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
" \"required\": [\"city\", \"state\", \"unit\"],\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
" }\n",
|
||||
"]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Define Messages"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def get_messages():\n",
|
||||
" return [\n",
|
||||
" {\n",
|
||||
" \"role\": \"user\",\n",
|
||||
" \"content\": \"What's the weather like in Boston today? Output a reasoning before act, then use the tools to help you.\",\n",
|
||||
" }\n",
|
||||
" ]\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"messages = get_messages()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Initialize the Client"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Initialize OpenAI-like client\n",
|
||||
"client = OpenAI(api_key=\"None\", base_url=f\"http://0.0.0.0:{port}/v1\")\n",
|
||||
"model_name = client.models.list().data[0].id"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Non-Streaming Request"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Non-streaming mode test\n",
|
||||
"response_non_stream = client.chat.completions.create(\n",
|
||||
" model=model_name,\n",
|
||||
" messages=messages,\n",
|
||||
" temperature=0,\n",
|
||||
" top_p=0.95,\n",
|
||||
" max_tokens=1024,\n",
|
||||
" stream=False, # Non-streaming\n",
|
||||
" tools=tools,\n",
|
||||
")\n",
|
||||
"print_highlight(\"Non-stream response:\")\n",
|
||||
"print_highlight(response_non_stream)\n",
|
||||
"print_highlight(\"==== content ====\")\n",
|
||||
"print_highlight(response_non_stream.choices[0].message.content)\n",
|
||||
"print_highlight(\"==== tool_calls ====\")\n",
|
||||
"print_highlight(response_non_stream.choices[0].message.tool_calls)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Handle Tools\n",
|
||||
"When the engine determines it should call a particular tool, it will return arguments or partial arguments through the response. You can parse these arguments and later invoke the tool accordingly."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"name_non_stream = response_non_stream.choices[0].message.tool_calls[0].function.name\n",
|
||||
"arguments_non_stream = (\n",
|
||||
" response_non_stream.choices[0].message.tool_calls[0].function.arguments\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(f\"Final streamed function call name: {name_non_stream}\")\n",
|
||||
"print_highlight(f\"Final streamed function call arguments: {arguments_non_stream}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Streaming Request"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Streaming mode test\n",
|
||||
"print_highlight(\"Streaming response:\")\n",
|
||||
"response_stream = client.chat.completions.create(\n",
|
||||
" model=model_name,\n",
|
||||
" messages=messages,\n",
|
||||
" temperature=0,\n",
|
||||
" top_p=0.95,\n",
|
||||
" max_tokens=1024,\n",
|
||||
" stream=True, # Enable streaming\n",
|
||||
" tools=tools,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"texts = \"\"\n",
|
||||
"tool_calls = []\n",
|
||||
"name = \"\"\n",
|
||||
"arguments = \"\"\n",
|
||||
"for chunk in response_stream:\n",
|
||||
" if chunk.choices[0].delta.content:\n",
|
||||
" texts += chunk.choices[0].delta.content\n",
|
||||
" if chunk.choices[0].delta.tool_calls:\n",
|
||||
" tool_calls.append(chunk.choices[0].delta.tool_calls[0])\n",
|
||||
"print_highlight(\"==== Text ====\")\n",
|
||||
"print_highlight(texts)\n",
|
||||
"\n",
|
||||
"print_highlight(\"==== Tool Call ====\")\n",
|
||||
"for tool_call in tool_calls:\n",
|
||||
" print_highlight(tool_call)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Handle Tools\n",
|
||||
"When the engine determines it should call a particular tool, it will return arguments or partial arguments through the response. You can parse these arguments and later invoke the tool accordingly."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Parse and combine function call arguments\n",
|
||||
"arguments = []\n",
|
||||
"for tool_call in tool_calls:\n",
|
||||
" if tool_call.function.name:\n",
|
||||
" print_highlight(f\"Streamed function call name: {tool_call.function.name}\")\n",
|
||||
"\n",
|
||||
" if tool_call.function.arguments:\n",
|
||||
" arguments.append(tool_call.function.arguments)\n",
|
||||
"\n",
|
||||
"# Combine all fragments into a single JSON string\n",
|
||||
"full_arguments = \"\".join(arguments)\n",
|
||||
"print_highlight(f\"streamed function call arguments: {full_arguments}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Define a Tool Function"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# This is a demonstration, define real function according to your usage.\n",
|
||||
"def get_current_weather(city: str, state: str, unit: \"str\"):\n",
|
||||
" return (\n",
|
||||
" f\"The weather in {city}, {state} is 85 degrees {unit}. It is \"\n",
|
||||
" \"partly cloudly, with highs in the 90's.\"\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"available_tools = {\"get_current_weather\": get_current_weather}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n",
|
||||
"### Execute the Tool"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"messages.append(response_non_stream.choices[0].message)\n",
|
||||
"\n",
|
||||
"# Call the corresponding tool function\n",
|
||||
"tool_call = messages[-1].tool_calls[0]\n",
|
||||
"tool_name = tool_call.function.name\n",
|
||||
"tool_to_call = available_tools[tool_name]\n",
|
||||
"result = tool_to_call(**(json.loads(tool_call.function.arguments)))\n",
|
||||
"print_highlight(f\"Function call result: {result}\")\n",
|
||||
"# messages.append({\"role\": \"tool\", \"content\": result, \"name\": tool_name})\n",
|
||||
"messages.append(\n",
|
||||
" {\n",
|
||||
" \"role\": \"tool\",\n",
|
||||
" \"tool_call_id\": tool_call.id,\n",
|
||||
" \"content\": str(result),\n",
|
||||
" \"name\": tool_name,\n",
|
||||
" }\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(f\"Updated message history: {messages}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Send Results Back to Model"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"final_response = client.chat.completions.create(\n",
|
||||
" model=model_name,\n",
|
||||
" messages=messages,\n",
|
||||
" temperature=0,\n",
|
||||
" top_p=0.95,\n",
|
||||
" stream=False,\n",
|
||||
" tools=tools,\n",
|
||||
")\n",
|
||||
"print_highlight(\"Non-stream response:\")\n",
|
||||
"print_highlight(final_response)\n",
|
||||
"\n",
|
||||
"print_highlight(\"==== Text ====\")\n",
|
||||
"print_highlight(final_response.choices[0].message.content)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Native API and SGLang Runtime (SRT)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from transformers import AutoTokenizer\n",
|
||||
"import requests\n",
|
||||
"\n",
|
||||
"# generate an answer\n",
|
||||
"tokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen2.5-7B-Instruct\")\n",
|
||||
"\n",
|
||||
"messages = get_messages()\n",
|
||||
"\n",
|
||||
"input = tokenizer.apply_chat_template(\n",
|
||||
" messages, tokenize=False, add_generation_prompt=True, tools=tools, return_dict=False\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"gen_url = f\"http://localhost:{port}/generate\"\n",
|
||||
"gen_data = {\n",
|
||||
" \"text\": input,\n",
|
||||
" \"sampling_params\": {\n",
|
||||
" \"skip_special_tokens\": False,\n",
|
||||
" \"max_new_tokens\": 1024,\n",
|
||||
" \"temperature\": 0,\n",
|
||||
" \"top_p\": 0.95,\n",
|
||||
" },\n",
|
||||
"}\n",
|
||||
"gen_response = requests.post(gen_url, json=gen_data).json()[\"text\"]\n",
|
||||
"print_highlight(\"==== Response ====\")\n",
|
||||
"print_highlight(gen_response)\n",
|
||||
"\n",
|
||||
"# parse the response\n",
|
||||
"parse_url = f\"http://localhost:{port}/parse_function_call\"\n",
|
||||
"\n",
|
||||
"function_call_input = {\n",
|
||||
" \"text\": gen_response,\n",
|
||||
" \"tool_call_parser\": \"qwen25\",\n",
|
||||
" \"tools\": tools,\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"function_call_response = requests.post(parse_url, json=function_call_input)\n",
|
||||
"function_call_response_json = function_call_response.json()\n",
|
||||
"\n",
|
||||
"print_highlight(\"==== Text ====\")\n",
|
||||
"print(function_call_response_json[\"normal_text\"])\n",
|
||||
"print_highlight(\"==== Calls ====\")\n",
|
||||
"print(\"function name: \", function_call_response_json[\"calls\"][0][\"name\"])\n",
|
||||
"print(\"function arguments: \", function_call_response_json[\"calls\"][0][\"parameters\"])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(server_process)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Offline Engine API"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sglang as sgl\n",
|
||||
"from sglang.srt.function_call.function_call_parser import FunctionCallParser\n",
|
||||
"from sglang.srt.managers.io_struct import Tool, Function\n",
|
||||
"\n",
|
||||
"llm = sgl.Engine(model_path=\"Qwen/Qwen2.5-7B-Instruct\")\n",
|
||||
"tokenizer = llm.tokenizer_manager.tokenizer\n",
|
||||
"input_ids = tokenizer.apply_chat_template(\n",
|
||||
" messages, tokenize=True, add_generation_prompt=True, tools=tools, return_dict=False\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# Note that for gpt-oss tool parser, adding \"no_stop_trim\": True\n",
|
||||
"# to make sure the tool call token <call> is not trimmed.\n",
|
||||
"\n",
|
||||
"sampling_params = {\n",
|
||||
" \"max_new_tokens\": 1024,\n",
|
||||
" \"temperature\": 0,\n",
|
||||
" \"top_p\": 0.95,\n",
|
||||
" \"skip_special_tokens\": False,\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"# 1) Offline generation\n",
|
||||
"result = llm.generate(input_ids=input_ids, sampling_params=sampling_params)\n",
|
||||
"generated_text = result[\"text\"] # Assume there is only one prompt\n",
|
||||
"\n",
|
||||
"print_highlight(\"=== Offline Engine Output Text ===\")\n",
|
||||
"print_highlight(generated_text)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# 2) Parse using FunctionCallParser\n",
|
||||
"def convert_dict_to_tool(tool_dict: dict) -> Tool:\n",
|
||||
" function_dict = tool_dict.get(\"function\", {})\n",
|
||||
" return Tool(\n",
|
||||
" type=tool_dict.get(\"type\", \"function\"),\n",
|
||||
" function=Function(\n",
|
||||
" name=function_dict.get(\"name\"),\n",
|
||||
" description=function_dict.get(\"description\"),\n",
|
||||
" parameters=function_dict.get(\"parameters\"),\n",
|
||||
" ),\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"tools = [convert_dict_to_tool(raw_tool) for raw_tool in tools]\n",
|
||||
"\n",
|
||||
"parser = FunctionCallParser(tools=tools, tool_call_parser=\"qwen25\")\n",
|
||||
"normal_text, calls = parser.parse_non_stream(generated_text)\n",
|
||||
"\n",
|
||||
"print_highlight(\"=== Parsing Result ===\")\n",
|
||||
"print(\"Normal text portion:\", normal_text)\n",
|
||||
"print_highlight(\"Function call portion:\")\n",
|
||||
"for call in calls:\n",
|
||||
" # call: ToolCallItem\n",
|
||||
" print_highlight(f\" - tool name: {call.name}\")\n",
|
||||
" print_highlight(f\" parameters: {call.parameters}\")\n",
|
||||
"\n",
|
||||
"# 3) If needed, perform additional logic on the parsed functions, such as automatically calling the corresponding function to obtain a return value, etc."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"llm.shutdown()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Tool Choice Mode\n",
|
||||
"\n",
|
||||
"SGLang supports OpenAI's `tool_choice` parameter to control when and which tools the model should call. This feature is implemented using EBNF (Extended Backus-Naur Form) grammar to ensure reliable tool calling behavior.\n",
|
||||
"\n",
|
||||
"### Supported Tool Choice Options\n",
|
||||
"\n",
|
||||
"- **`tool_choice=\"required\"`**: Forces the model to call at least one tool\n",
|
||||
"- **`tool_choice={\"type\": \"function\", \"function\": {\"name\": \"specific_function\"}}`**: Forces the model to call a specific function\n",
|
||||
"\n",
|
||||
"### Backend Compatibility\n",
|
||||
"\n",
|
||||
"Tool choice is fully supported with the **Xgrammar backend**, which is the default grammar backend (`--grammar-backend xgrammar`). However, it may not be fully supported with other backends such as `outlines`.\n",
|
||||
"\n",
|
||||
"### Example: Required Tool Choice"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from openai import OpenAI\n",
|
||||
"from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
|
||||
"from sglang.test.doc_patch import launch_server_cmd\n",
|
||||
"\n",
|
||||
"# Start a new server session for tool choice examples\n",
|
||||
"server_process_tool_choice, port_tool_choice = launch_server_cmd(\n",
|
||||
" \"python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --tool-call-parser qwen25 --host 0.0.0.0 --log-level warning\"\n",
|
||||
")\n",
|
||||
"wait_for_server(f\"http://localhost:{port_tool_choice}\")\n",
|
||||
"\n",
|
||||
"# Initialize client for tool choice examples\n",
|
||||
"client_tool_choice = OpenAI(\n",
|
||||
" api_key=\"None\", base_url=f\"http://0.0.0.0:{port_tool_choice}/v1\"\n",
|
||||
")\n",
|
||||
"model_name_tool_choice = client_tool_choice.models.list().data[0].id\n",
|
||||
"\n",
|
||||
"# Example with tool_choice=\"required\" - forces the model to call a tool\n",
|
||||
"messages_required = [\n",
|
||||
" {\"role\": \"user\", \"content\": \"Hello, what is the capital of France?\"}\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"# Define tools\n",
|
||||
"tools = [\n",
|
||||
" {\n",
|
||||
" \"type\": \"function\",\n",
|
||||
" \"function\": {\n",
|
||||
" \"name\": \"get_current_weather\",\n",
|
||||
" \"description\": \"Get the current weather in a given location\",\n",
|
||||
" \"parameters\": {\n",
|
||||
" \"type\": \"object\",\n",
|
||||
" \"properties\": {\n",
|
||||
" \"city\": {\n",
|
||||
" \"type\": \"string\",\n",
|
||||
" \"description\": \"The city to find the weather for, e.g. 'San Francisco'\",\n",
|
||||
" },\n",
|
||||
" \"unit\": {\n",
|
||||
" \"type\": \"string\",\n",
|
||||
" \"description\": \"The unit to fetch the temperature in\",\n",
|
||||
" \"enum\": [\"celsius\", \"fahrenheit\"],\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
" \"required\": [\"city\", \"unit\"],\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
" }\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"response_required = client_tool_choice.chat.completions.create(\n",
|
||||
" model=model_name_tool_choice,\n",
|
||||
" messages=messages_required,\n",
|
||||
" temperature=0,\n",
|
||||
" max_tokens=1024,\n",
|
||||
" tools=tools,\n",
|
||||
" tool_choice=\"required\", # Force the model to call a tool\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(\"Response with tool_choice='required':\")\n",
|
||||
"print(\"Content:\", response_required.choices[0].message.content)\n",
|
||||
"print(\"Tool calls:\", response_required.choices[0].message.tool_calls)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Example: Specific Function Choice\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Example with specific function choice - forces the model to call a specific function\n",
|
||||
"messages_specific = [\n",
|
||||
" {\"role\": \"user\", \"content\": \"What are the most attactive places in France?\"}\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"response_specific = client_tool_choice.chat.completions.create(\n",
|
||||
" model=model_name_tool_choice,\n",
|
||||
" messages=messages_specific,\n",
|
||||
" temperature=0,\n",
|
||||
" max_tokens=1024,\n",
|
||||
" tools=tools,\n",
|
||||
" tool_choice={\n",
|
||||
" \"type\": \"function\",\n",
|
||||
" \"function\": {\"name\": \"get_current_weather\"},\n",
|
||||
" }, # Force the model to call the specific get_current_weather function\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(\"Response with specific function choice:\")\n",
|
||||
"print(\"Content:\", response_specific.choices[0].message.content)\n",
|
||||
"print(\"Tool calls:\", response_specific.choices[0].message.tool_calls)\n",
|
||||
"\n",
|
||||
"if response_specific.choices[0].message.tool_calls:\n",
|
||||
" tool_call = response_specific.choices[0].message.tool_calls[0]\n",
|
||||
" print_highlight(f\"Called function: {tool_call.function.name}\")\n",
|
||||
" print_highlight(f\"Arguments: {tool_call.function.arguments}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(server_process_tool_choice)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Pythonic Tool Call Format (Llama-3.2 / Llama-3.3 / Llama-4)\n",
|
||||
"\n",
|
||||
"Some Llama models (such as Llama-3.2-1B, Llama-3.2-3B, Llama-3.3-70B, and Llama-4) support a \"pythonic\" tool call format, where the model outputs function calls as Python code, e.g.:\n",
|
||||
"\n",
|
||||
"```python\n",
|
||||
"[get_current_weather(city=\"San Francisco\", state=\"CA\", unit=\"celsius\")]\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"- The output is a Python list of function calls, with arguments as Python literals (not JSON).\n",
|
||||
"- Multiple tool calls can be returned in the same list:\n",
|
||||
"```python\n",
|
||||
"[get_current_weather(city=\"San Francisco\", state=\"CA\", unit=\"celsius\"),\n",
|
||||
" get_current_weather(city=\"New York\", state=\"NY\", unit=\"fahrenheit\")]\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"For more information, refer to Meta’s documentation on [Zero shot function calling](https://github.com/meta-llama/llama-models/blob/main/models/llama4/prompt_format.md#zero-shot-function-calling---system-message).\n",
|
||||
"\n",
|
||||
"Note that this feature is still under development on Blackwell.\n",
|
||||
"\n",
|
||||
"### How to enable\n",
|
||||
"- Launch the server with `--tool-call-parser pythonic`\n",
|
||||
"- You may also specify --chat-template with the improved template for the model (e.g., `--chat-template=examples/chat_template/tool_chat_template_llama4_pythonic.jinja`).\n",
|
||||
"This is recommended because the model expects a special prompt format to reliably produce valid pythonic tool call outputs. The template ensures that the prompt structure (e.g., special tokens, message boundaries like `<|eom|>`, and function call delimiters) matches what the model was trained or fine-tuned on. If you do not use the correct chat template, tool calling may fail or produce inconsistent results.\n",
|
||||
"\n",
|
||||
"#### Forcing Pythonic Tool Call Output Without a Chat Template\n",
|
||||
"If you don't want to specify a chat template, you must give the model extremely explicit instructions in your messages to enforce pythonic output. For example, for `Llama-3.2-1B-Instruct`, you need:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import openai\n",
|
||||
"\n",
|
||||
"server_process, port = launch_server_cmd(\n",
|
||||
" \" python3 -m sglang.launch_server --model-path meta-llama/Llama-3.2-1B-Instruct --tool-call-parser pythonic --tp 1 --log-level warning\" # llama-3.2-1b-instruct\n",
|
||||
")\n",
|
||||
"wait_for_server(f\"http://localhost:{port}\")\n",
|
||||
"\n",
|
||||
"tools = [\n",
|
||||
" {\n",
|
||||
" \"type\": \"function\",\n",
|
||||
" \"function\": {\n",
|
||||
" \"name\": \"get_weather\",\n",
|
||||
" \"description\": \"Get the current weather for a given location.\",\n",
|
||||
" \"parameters\": {\n",
|
||||
" \"type\": \"object\",\n",
|
||||
" \"properties\": {\n",
|
||||
" \"location\": {\n",
|
||||
" \"type\": \"string\",\n",
|
||||
" \"description\": \"The name of the city or location.\",\n",
|
||||
" }\n",
|
||||
" },\n",
|
||||
" \"required\": [\"location\"],\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"type\": \"function\",\n",
|
||||
" \"function\": {\n",
|
||||
" \"name\": \"get_tourist_attractions\",\n",
|
||||
" \"description\": \"Get a list of top tourist attractions for a given city.\",\n",
|
||||
" \"parameters\": {\n",
|
||||
" \"type\": \"object\",\n",
|
||||
" \"properties\": {\n",
|
||||
" \"city\": {\n",
|
||||
" \"type\": \"string\",\n",
|
||||
" \"description\": \"The name of the city to find attractions for.\",\n",
|
||||
" }\n",
|
||||
" },\n",
|
||||
" \"required\": [\"city\"],\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def get_messages():\n",
|
||||
" return [\n",
|
||||
" {\n",
|
||||
" \"role\": \"system\",\n",
|
||||
" \"content\": (\n",
|
||||
" \"You are a travel assistant. \"\n",
|
||||
" \"When asked to call functions, ALWAYS respond ONLY with a python list of function calls, \"\n",
|
||||
" \"using this format: [func_name1(param1=value1, param2=value2), func_name2(param=value)]. \"\n",
|
||||
" \"Do NOT use JSON, do NOT use variables, do NOT use any other format. \"\n",
|
||||
" \"Here is an example:\\n\"\n",
|
||||
" '[get_weather(location=\"Paris\"), get_tourist_attractions(city=\"Paris\")]'\n",
|
||||
" ),\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"role\": \"user\",\n",
|
||||
" \"content\": (\n",
|
||||
" \"I'm planning a trip to Tokyo next week. What's the weather like and what are some top tourist attractions? \"\n",
|
||||
" \"Propose parallel tool calls at once, using the python list of function calls format as shown above.\"\n",
|
||||
" ),\n",
|
||||
" },\n",
|
||||
" ]\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"messages = get_messages()\n",
|
||||
"\n",
|
||||
"client = openai.Client(base_url=f\"http://localhost:{port}/v1\", api_key=\"xxxxxx\")\n",
|
||||
"model_name = client.models.list().data[0].id\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"response_non_stream = client.chat.completions.create(\n",
|
||||
" model=model_name,\n",
|
||||
" messages=messages,\n",
|
||||
" temperature=0,\n",
|
||||
" top_p=0.9,\n",
|
||||
" stream=False, # Non-streaming\n",
|
||||
" tools=tools,\n",
|
||||
")\n",
|
||||
"print_highlight(\"Non-stream response:\")\n",
|
||||
"print_highlight(response_non_stream)\n",
|
||||
"\n",
|
||||
"response_stream = client.chat.completions.create(\n",
|
||||
" model=model_name,\n",
|
||||
" messages=messages,\n",
|
||||
" temperature=0,\n",
|
||||
" top_p=0.9,\n",
|
||||
" stream=True,\n",
|
||||
" tools=tools,\n",
|
||||
")\n",
|
||||
"texts = \"\"\n",
|
||||
"tool_calls = []\n",
|
||||
"name = \"\"\n",
|
||||
"arguments = \"\"\n",
|
||||
"\n",
|
||||
"for chunk in response_stream:\n",
|
||||
" if chunk.choices[0].delta.content:\n",
|
||||
" texts += chunk.choices[0].delta.content\n",
|
||||
" if chunk.choices[0].delta.tool_calls:\n",
|
||||
" tool_calls.append(chunk.choices[0].delta.tool_calls[0])\n",
|
||||
"\n",
|
||||
"print_highlight(\"Streaming Response:\")\n",
|
||||
"print_highlight(\"==== Text ====\")\n",
|
||||
"print_highlight(texts)\n",
|
||||
"\n",
|
||||
"print_highlight(\"==== Tool Call ====\")\n",
|
||||
"for tool_call in tool_calls:\n",
|
||||
" print_highlight(tool_call)\n",
|
||||
"\n",
|
||||
"terminate_process(server_process)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Note:** \n",
|
||||
"> The model may still default to JSON if it was heavily finetuned on that format. Prompt engineering (including examples) is the only way to increase the chance of pythonic output if you are not using a chat template."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## How to support a new model?\n",
|
||||
"1. Update the TOOLS_TAG_LIST in sglang/srt/function_call_parser.py with the model’s tool tags. Currently supported tags include:\n",
|
||||
"```\n",
|
||||
"\tTOOLS_TAG_LIST = [\n",
|
||||
"\t “<|plugin|>“,\n",
|
||||
"\t “<function=“,\n",
|
||||
"\t “<tool_call>“,\n",
|
||||
"\t “<|python_tag|>“,\n",
|
||||
"\t “[TOOL_CALLS]”\n",
|
||||
"\t]\n",
|
||||
"```\n",
|
||||
"2. Create a new detector class in sglang/srt/function_call_parser.py that inherits from BaseFormatDetector. The detector should handle the model’s specific function call format. For example:\n",
|
||||
"```\n",
|
||||
" class NewModelDetector(BaseFormatDetector):\n",
|
||||
"```\n",
|
||||
"3. Add the new detector to the MultiFormatParser class that manages all the format detectors."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
@@ -0,0 +1,740 @@
|
||||
---
|
||||
title: "Tool Parser"
|
||||
metatags:
|
||||
description: "SGLang function calling: tool parsers for DeepSeek, Llama, Qwen, Mistral, GLM, Kimi K2. OpenAI-compatible tool use API."
|
||||
---
|
||||
This guide demonstrates how to use SGLang’s [Function calling](https://platform.openai.com/docs/guides/function-calling) functionality.
|
||||
|
||||
|
||||
## Currently supported parsers:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "34%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parser</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Supported Models</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Notes</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`deepseekv3`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek-v3 (e.g., `deepseek-ai/DeepSeek-V3-0324`)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Recommend adding `--chat-template ./examples/chat_template/tool_chat_template_deepseekv3.jinja` to launch command.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`deepseekv31`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek-V3.1 and DeepSeek-V3.2-Exp (e.g. `deepseek-ai/DeepSeek-V3.1`, `deepseek-ai/DeepSeek-V3.2-Exp`)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Recommend adding `--chat-template ./examples/chat_template/tool_chat_template_deepseekv31.jinja` (Or ..deepseekv32.jinja for DeepSeek-V3.2) to launch command.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`deepseekv32`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek-V3.2 (`deepseek-ai/DeepSeek-V3.2`)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`glm`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GLM series (e.g. `zai-org/GLM-4.6`)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`gpt-oss`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GPT-OSS (e.g., `openai/gpt-oss-120b`, `openai/gpt-oss-20b`, `lmsys/gpt-oss-120b-bf16`, `lmsys/gpt-oss-20b-bf16`)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The gpt-oss tool parser filters out analysis channel events and only preserves normal text. This can cause the content to be empty when explanations are in the analysis channel. To work around this, complete the tool round by returning tool results as `role="tool"` messages, which enables the model to generate the final content.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`kimi_k2`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`moonshotai/Kimi-K2-Instruct`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`llama3`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama 3.1 / 3.2 / 3.3 (e.g. `meta-llama/Llama-3.1-8B-Instruct`, `meta-llama/Llama-3.2-1B-Instruct`, `meta-llama/Llama-3.3-70B-Instruct`)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`llama4`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama 4 (e.g. `meta-llama/Llama-4-Scout-17B-16E-Instruct`)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`mistral`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Mistral (e.g. `mistralai/Mistral-7B-Instruct-v0.3`, `mistralai/Mistral-Nemo-Instruct-2407`, `mistralai/Mistral-7B-v0.3`)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`pythonic`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama-3.2 / Llama-3.3 / Llama-4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Model outputs function calls as Python code. Requires `--tool-call-parser pythonic` and is recommended to use with a specific chat template.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`qwen`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen series (e.g. `Qwen/Qwen3-Next-80B-A3B-Instruct`, `Qwen/Qwen3-VL-30B-A3B-Thinking`) except Qwen3-Coder</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`qwen3_coder`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3-Coder (e.g. `Qwen/Qwen3-Coder-30B-A3B-Instruct`)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`step3`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Step-3</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
|
||||
## OpenAI Compatible API
|
||||
|
||||
|
||||
### Launching the Server
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import json
|
||||
from sglang.test.doc_patch import launch_server_cmd
|
||||
from sglang.utils import wait_for_server, print_highlight, terminate_process
|
||||
from openai import OpenAI
|
||||
|
||||
server_process, port = launch_server_cmd(
|
||||
"python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --tool-call-parser qwen25 --host 0.0.0.0 --log-level warning" # qwen25
|
||||
)
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
```
|
||||
|
||||
Note that `--tool-call-parser` defines the parser used to interpret responses.
|
||||
|
||||
|
||||
### Define Tools for Function Call
|
||||
Below is a Python snippet that shows how to define a tool as a dictionary. The dictionary includes a tool name, a description, and property defined Parameters.
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
# Define tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_current_weather",
|
||||
"description": "Get the current weather in a given location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"city": {
|
||||
"type": "string",
|
||||
"description": "The city to find the weather for, e.g. 'San Francisco'",
|
||||
},
|
||||
"state": {
|
||||
"type": "string",
|
||||
"description": "the two-letter abbreviation for the state that the city is"
|
||||
" in, e.g. 'CA' which would mean 'California'",
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"description": "The unit to fetch the temperature in",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
},
|
||||
},
|
||||
"required": ["city", "state", "unit"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
### Define Messages
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
def get_messages():
|
||||
return [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "What's the weather like in Boston today? Output a reasoning before act, then use the tools to help you.",
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
messages = get_messages()
|
||||
```
|
||||
|
||||
### Initialize the Client
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
# Initialize OpenAI-like client
|
||||
client = OpenAI(api_key="None", base_url=f"http://0.0.0.0:{port}/v1")
|
||||
model_name = client.models.list().data[0].id
|
||||
```
|
||||
|
||||
### Non-Streaming Request
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
# Non-streaming mode test
|
||||
response_non_stream = client.chat.completions.create(
|
||||
model=model_name,
|
||||
messages=messages,
|
||||
temperature=0,
|
||||
top_p=0.95,
|
||||
max_tokens=1024,
|
||||
stream=False, # Non-streaming
|
||||
tools=tools,
|
||||
)
|
||||
print_highlight("Non-stream response:")
|
||||
print_highlight(response_non_stream)
|
||||
print_highlight("==== content ====")
|
||||
print_highlight(response_non_stream.choices[0].message.content)
|
||||
print_highlight("==== tool_calls ====")
|
||||
print_highlight(response_non_stream.choices[0].message.tool_calls)
|
||||
```
|
||||
|
||||
#### Handle Tools
|
||||
When the engine determines it should call a particular tool, it will return arguments or partial arguments through the response. You can parse these arguments and later invoke the tool accordingly.
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
name_non_stream = response_non_stream.choices[0].message.tool_calls[0].function.name
|
||||
arguments_non_stream = (
|
||||
response_non_stream.choices[0].message.tool_calls[0].function.arguments
|
||||
)
|
||||
|
||||
print_highlight(f"Final streamed function call name: {name_non_stream}")
|
||||
print_highlight(f"Final streamed function call arguments: {arguments_non_stream}")
|
||||
```
|
||||
|
||||
### Streaming Request
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
# Streaming mode test
|
||||
print_highlight("Streaming response:")
|
||||
response_stream = client.chat.completions.create(
|
||||
model=model_name,
|
||||
messages=messages,
|
||||
temperature=0,
|
||||
top_p=0.95,
|
||||
max_tokens=1024,
|
||||
stream=True, # Enable streaming
|
||||
tools=tools,
|
||||
)
|
||||
|
||||
texts = ""
|
||||
tool_calls = []
|
||||
name = ""
|
||||
arguments = ""
|
||||
for chunk in response_stream:
|
||||
if chunk.choices[0].delta.content:
|
||||
texts += chunk.choices[0].delta.content
|
||||
if chunk.choices[0].delta.tool_calls:
|
||||
tool_calls.append(chunk.choices[0].delta.tool_calls[0])
|
||||
print_highlight("==== Text ====")
|
||||
print_highlight(texts)
|
||||
|
||||
print_highlight("==== Tool Call ====")
|
||||
for tool_call in tool_calls:
|
||||
print_highlight(tool_call)
|
||||
```
|
||||
|
||||
#### Handle Tools
|
||||
When the engine determines it should call a particular tool, it will return arguments or partial arguments through the response. You can parse these arguments and later invoke the tool accordingly.
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
# Parse and combine function call arguments
|
||||
arguments = []
|
||||
for tool_call in tool_calls:
|
||||
if tool_call.function.name:
|
||||
print_highlight(f"Streamed function call name: {tool_call.function.name}")
|
||||
|
||||
if tool_call.function.arguments:
|
||||
arguments.append(tool_call.function.arguments)
|
||||
|
||||
# Combine all fragments into a single JSON string
|
||||
full_arguments = "".join(arguments)
|
||||
print_highlight(f"streamed function call arguments: {full_arguments}")
|
||||
```
|
||||
|
||||
### Define a Tool Function
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
# This is a demonstration, define real function according to your usage.
|
||||
def get_current_weather(city: str, state: str, unit: "str"):
|
||||
return (
|
||||
f"The weather in {city}, {state} is 85 degrees {unit}. It is "
|
||||
"partly cloudly, with highs in the 90's."
|
||||
)
|
||||
|
||||
|
||||
available_tools = {"get_current_weather": get_current_weather}
|
||||
```
|
||||
|
||||
|
||||
### Execute the Tool
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
messages.append(response_non_stream.choices[0].message)
|
||||
|
||||
# Call the corresponding tool function
|
||||
tool_call = messages[-1].tool_calls[0]
|
||||
tool_name = tool_call.function.name
|
||||
tool_to_call = available_tools[tool_name]
|
||||
result = tool_to_call(**(json.loads(tool_call.function.arguments)))
|
||||
print_highlight(f"Function call result: {result}")
|
||||
# messages.append({"role": "tool", "content": result, "name": tool_name})
|
||||
messages.append(
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": tool_call.id,
|
||||
"content": str(result),
|
||||
"name": tool_name,
|
||||
}
|
||||
)
|
||||
|
||||
print_highlight(f"Updated message history: {messages}")
|
||||
```
|
||||
|
||||
### Send Results Back to Model
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
final_response = client.chat.completions.create(
|
||||
model=model_name,
|
||||
messages=messages,
|
||||
temperature=0,
|
||||
top_p=0.95,
|
||||
stream=False,
|
||||
tools=tools,
|
||||
)
|
||||
print_highlight("Non-stream response:")
|
||||
print_highlight(final_response)
|
||||
|
||||
print_highlight("==== Text ====")
|
||||
print_highlight(final_response.choices[0].message.content)
|
||||
```
|
||||
|
||||
## Native API and SGLang Runtime (SRT)
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
from transformers import AutoTokenizer
|
||||
import requests
|
||||
|
||||
# generate an answer
|
||||
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
|
||||
|
||||
messages = get_messages()
|
||||
|
||||
input = tokenizer.apply_chat_template(
|
||||
messages, tokenize=False, add_generation_prompt=True, tools=tools, return_dict=False
|
||||
)
|
||||
|
||||
gen_url = f"http://localhost:{port}/generate"
|
||||
gen_data = {
|
||||
"text": input,
|
||||
"sampling_params": {
|
||||
"skip_special_tokens": False,
|
||||
"max_new_tokens": 1024,
|
||||
"temperature": 0,
|
||||
"top_p": 0.95,
|
||||
},
|
||||
}
|
||||
gen_response = requests.post(gen_url, json=gen_data).json()["text"]
|
||||
print_highlight("==== Response ====")
|
||||
print_highlight(gen_response)
|
||||
|
||||
# parse the response
|
||||
parse_url = f"http://localhost:{port}/parse_function_call"
|
||||
|
||||
function_call_input = {
|
||||
"text": gen_response,
|
||||
"tool_call_parser": "qwen25",
|
||||
"tools": tools,
|
||||
}
|
||||
|
||||
function_call_response = requests.post(parse_url, json=function_call_input)
|
||||
function_call_response_json = function_call_response.json()
|
||||
|
||||
print_highlight("==== Text ====")
|
||||
print(function_call_response_json["normal_text"])
|
||||
print_highlight("==== Calls ====")
|
||||
print("function name: ", function_call_response_json["calls"][0]["name"])
|
||||
print("function arguments: ", function_call_response_json["calls"][0]["parameters"])
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
terminate_process(server_process)
|
||||
```
|
||||
|
||||
## Offline Engine API
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import sglang as sgl
|
||||
from sglang.srt.function_call.function_call_parser import FunctionCallParser
|
||||
from sglang.srt.managers.io_struct import Tool, Function
|
||||
|
||||
llm = sgl.Engine(model_path="Qwen/Qwen2.5-7B-Instruct")
|
||||
tokenizer = llm.tokenizer_manager.tokenizer
|
||||
input_ids = tokenizer.apply_chat_template(
|
||||
messages, tokenize=True, add_generation_prompt=True, tools=tools, return_dict=False
|
||||
)
|
||||
|
||||
# Note that for gpt-oss tool parser, adding "no_stop_trim": True
|
||||
# to make sure the tool call token <call> is not trimmed.
|
||||
|
||||
sampling_params = {
|
||||
"max_new_tokens": 1024,
|
||||
"temperature": 0,
|
||||
"top_p": 0.95,
|
||||
"skip_special_tokens": False,
|
||||
}
|
||||
|
||||
# 1) Offline generation
|
||||
result = llm.generate(input_ids=input_ids, sampling_params=sampling_params)
|
||||
generated_text = result["text"] # Assume there is only one prompt
|
||||
|
||||
print_highlight("=== Offline Engine Output Text ===")
|
||||
print_highlight(generated_text)
|
||||
|
||||
|
||||
# 2) Parse using FunctionCallParser
|
||||
def convert_dict_to_tool(tool_dict: dict) -> Tool:
|
||||
function_dict = tool_dict.get("function", {})
|
||||
return Tool(
|
||||
type=tool_dict.get("type", "function"),
|
||||
function=Function(
|
||||
name=function_dict.get("name"),
|
||||
description=function_dict.get("description"),
|
||||
parameters=function_dict.get("parameters"),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
tools = [convert_dict_to_tool(raw_tool) for raw_tool in tools]
|
||||
|
||||
parser = FunctionCallParser(tools=tools, tool_call_parser="qwen25")
|
||||
normal_text, calls = parser.parse_non_stream(generated_text)
|
||||
|
||||
print_highlight("=== Parsing Result ===")
|
||||
print("Normal text portion:", normal_text)
|
||||
print_highlight("Function call portion:")
|
||||
for call in calls:
|
||||
# call: ToolCallItem
|
||||
print_highlight(f" - tool name: {call.name}")
|
||||
print_highlight(f" parameters: {call.parameters}")
|
||||
|
||||
# 3) If needed, perform additional logic on the parsed functions, such as automatically calling the corresponding function to obtain a return value, etc.
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
llm.shutdown()
|
||||
```
|
||||
|
||||
## Tool Choice Mode
|
||||
|
||||
SGLang supports OpenAI's `tool_choice` parameter to control when and which tools the model should call. This feature is implemented using EBNF (Extended Backus-Naur Form) grammar to ensure reliable tool calling behavior.
|
||||
|
||||
### Supported Tool Choice Options
|
||||
|
||||
- **`tool_choice="required"`**: Forces the model to call at least one tool
|
||||
- **`tool_choice={"type": "function", "function": {"name": "specific_function"}}`**: Forces the model to call a specific function
|
||||
|
||||
### Backend Compatibility
|
||||
|
||||
Tool choice is fully supported with the **Xgrammar backend**, which is the default grammar backend (`--grammar-backend xgrammar`). However, it may not be fully supported with other backends such as `outlines`.
|
||||
|
||||
### Example: Required Tool Choice
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
from sglang.utils import wait_for_server, print_highlight, terminate_process
|
||||
from sglang.test.doc_patch import launch_server_cmd
|
||||
|
||||
# Start a new server session for tool choice examples
|
||||
server_process_tool_choice, port_tool_choice = launch_server_cmd(
|
||||
"python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --tool-call-parser qwen25 --host 0.0.0.0 --log-level warning"
|
||||
)
|
||||
wait_for_server(f"http://localhost:{port_tool_choice}")
|
||||
|
||||
# Initialize client for tool choice examples
|
||||
client_tool_choice = OpenAI(
|
||||
api_key="None", base_url=f"http://0.0.0.0:{port_tool_choice}/v1"
|
||||
)
|
||||
model_name_tool_choice = client_tool_choice.models.list().data[0].id
|
||||
|
||||
# Example with tool_choice="required" - forces the model to call a tool
|
||||
messages_required = [
|
||||
{"role": "user", "content": "Hello, what is the capital of France?"}
|
||||
]
|
||||
|
||||
# Define tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_current_weather",
|
||||
"description": "Get the current weather in a given location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"city": {
|
||||
"type": "string",
|
||||
"description": "The city to find the weather for, e.g. 'San Francisco'",
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"description": "The unit to fetch the temperature in",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
},
|
||||
},
|
||||
"required": ["city", "unit"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response_required = client_tool_choice.chat.completions.create(
|
||||
model=model_name_tool_choice,
|
||||
messages=messages_required,
|
||||
temperature=0,
|
||||
max_tokens=1024,
|
||||
tools=tools,
|
||||
tool_choice="required", # Force the model to call a tool
|
||||
)
|
||||
|
||||
print_highlight("Response with tool_choice='required':")
|
||||
print("Content:", response_required.choices[0].message.content)
|
||||
print("Tool calls:", response_required.choices[0].message.tool_calls)
|
||||
```
|
||||
|
||||
### Example: Specific Function Choice
|
||||
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
# Example with specific function choice - forces the model to call a specific function
|
||||
messages_specific = [
|
||||
{"role": "user", "content": "What are the most attactive places in France?"}
|
||||
]
|
||||
|
||||
response_specific = client_tool_choice.chat.completions.create(
|
||||
model=model_name_tool_choice,
|
||||
messages=messages_specific,
|
||||
temperature=0,
|
||||
max_tokens=1024,
|
||||
tools=tools,
|
||||
tool_choice={
|
||||
"type": "function",
|
||||
"function": {"name": "get_current_weather"},
|
||||
}, # Force the model to call the specific get_current_weather function
|
||||
)
|
||||
|
||||
print_highlight("Response with specific function choice:")
|
||||
print("Content:", response_specific.choices[0].message.content)
|
||||
print("Tool calls:", response_specific.choices[0].message.tool_calls)
|
||||
|
||||
if response_specific.choices[0].message.tool_calls:
|
||||
tool_call = response_specific.choices[0].message.tool_calls[0]
|
||||
print_highlight(f"Called function: {tool_call.function.name}")
|
||||
print_highlight(f"Arguments: {tool_call.function.arguments}")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
terminate_process(server_process_tool_choice)
|
||||
```
|
||||
|
||||
## Pythonic Tool Call Format (Llama-3.2 / Llama-3.3 / Llama-4)
|
||||
|
||||
Some Llama models (such as Llama-3.2-1B, Llama-3.2-3B, Llama-3.3-70B, and Llama-4) support a "pythonic" tool call format, where the model outputs function calls as Python code, e.g.:
|
||||
|
||||
```python Example
|
||||
[get_current_weather(city="San Francisco", state="CA", unit="celsius")]
|
||||
```
|
||||
|
||||
- The output is a Python list of function calls, with arguments as Python literals (not JSON).
|
||||
- Multiple tool calls can be returned in the same list:
|
||||
```python Example
|
||||
[get_current_weather(city="San Francisco", state="CA", unit="celsius"),
|
||||
get_current_weather(city="New York", state="NY", unit="fahrenheit")]
|
||||
```
|
||||
|
||||
For more information, refer to Meta’s documentation on [Zero shot function calling](https://github.com/meta-llama/llama-models/blob/main/models/llama4/prompt_format.md#zero-shot-function-calling---system-message).
|
||||
|
||||
Note that this feature is still under development on Blackwell.
|
||||
|
||||
### How to enable
|
||||
- Launch the server with `--tool-call-parser pythonic`
|
||||
- You may also specify --chat-template with the improved template for the model (e.g., `--chat-template=examples/chat_template/tool_chat_template_llama4_pythonic.jinja`).
|
||||
This is recommended because the model expects a special prompt format to reliably produce valid pythonic tool call outputs. The template ensures that the prompt structure (e.g., special tokens, message boundaries like `<|eom|>`, and function call delimiters) matches what the model was trained or fine-tuned on. If you do not use the correct chat template, tool calling may fail or produce inconsistent results.
|
||||
|
||||
#### Forcing Pythonic Tool Call Output Without a Chat Template
|
||||
If you don't want to specify a chat template, you must give the model extremely explicit instructions in your messages to enforce pythonic output. For example, for `Llama-3.2-1B-Instruct`, you need:
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
|
||||
server_process, port = launch_server_cmd(
|
||||
" python3 -m sglang.launch_server --model-path meta-llama/Llama-3.2-1B-Instruct --tool-call-parser pythonic --tp 1 --log-level warning" # llama-3.2-1b-instruct
|
||||
)
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a given location.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The name of the city or location.",
|
||||
}
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_tourist_attractions",
|
||||
"description": "Get a list of top tourist attractions for a given city.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"city": {
|
||||
"type": "string",
|
||||
"description": "The name of the city to find attractions for.",
|
||||
}
|
||||
},
|
||||
"required": ["city"],
|
||||
},
|
||||
},
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def get_messages():
|
||||
return [
|
||||
{
|
||||
"role": "system",
|
||||
"content": (
|
||||
"You are a travel assistant. "
|
||||
"When asked to call functions, ALWAYS respond ONLY with a python list of function calls, "
|
||||
"using this format: [func_name1(param1=value1, param2=value2), func_name2(param=value)]. "
|
||||
"Do NOT use JSON, do NOT use variables, do NOT use any other format. "
|
||||
"Here is an example:\n"
|
||||
'[get_weather(location="Paris"), get_tourist_attractions(city="Paris")]'
|
||||
),
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": (
|
||||
"I'm planning a trip to Tokyo next week. What's the weather like and what are some top tourist attractions? "
|
||||
"Propose parallel tool calls at once, using the python list of function calls format as shown above."
|
||||
),
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
messages = get_messages()
|
||||
|
||||
client = openai.Client(base_url=f"http://localhost:{port}/v1", api_key="xxxxxx")
|
||||
model_name = client.models.list().data[0].id
|
||||
|
||||
|
||||
response_non_stream = client.chat.completions.create(
|
||||
model=model_name,
|
||||
messages=messages,
|
||||
temperature=0,
|
||||
top_p=0.9,
|
||||
stream=False, # Non-streaming
|
||||
tools=tools,
|
||||
)
|
||||
print_highlight("Non-stream response:")
|
||||
print_highlight(response_non_stream)
|
||||
|
||||
response_stream = client.chat.completions.create(
|
||||
model=model_name,
|
||||
messages=messages,
|
||||
temperature=0,
|
||||
top_p=0.9,
|
||||
stream=True,
|
||||
tools=tools,
|
||||
)
|
||||
texts = ""
|
||||
tool_calls = []
|
||||
name = ""
|
||||
arguments = ""
|
||||
|
||||
for chunk in response_stream:
|
||||
if chunk.choices[0].delta.content:
|
||||
texts += chunk.choices[0].delta.content
|
||||
if chunk.choices[0].delta.tool_calls:
|
||||
tool_calls.append(chunk.choices[0].delta.tool_calls[0])
|
||||
|
||||
print_highlight("Streaming Response:")
|
||||
print_highlight("==== Text ====")
|
||||
print_highlight(texts)
|
||||
|
||||
print_highlight("==== Tool Call ====")
|
||||
for tool_call in tool_calls:
|
||||
print_highlight(tool_call)
|
||||
|
||||
terminate_process(server_process)
|
||||
```
|
||||
|
||||
> **Note:**
|
||||
> The model may still default to JSON if it was heavily finetuned on that format. Prompt engineering (including examples) is the only way to increase the chance of pythonic output if you are not using a chat template.
|
||||
|
||||
|
||||
## How to support a new model?
|
||||
1. Update the TOOLS_TAG_LIST in sglang/srt/function_call_parser.py with the model’s tool tags. Currently supported tags include:
|
||||
```text Output
|
||||
TOOLS_TAG_LIST = [
|
||||
“<|plugin|>“,
|
||||
“<function=“,
|
||||
“<tool_call>“,
|
||||
“<|python_tag|>“,
|
||||
“[TOOL_CALLS]”
|
||||
]
|
||||
```
|
||||
2. Create a new detector class in sglang/srt/function_call_parser.py that inherits from BaseFormatDetector. The detector should handle the model’s specific function call format. For example:
|
||||
```text Output
|
||||
class NewModelDetector(BaseFormatDetector):
|
||||
```
|
||||
3. Add the new detector to the MultiFormatParser class that manages all the format detectors.
|
||||
@@ -0,0 +1,388 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Query VLM with Offline Engine\n",
|
||||
"\n",
|
||||
"This tutorial demonstrates how to use SGLang's **offline Engine API** to query VLMs. We will demonstrate usage with Qwen2.5-VL and Llama 4. This section demonstrates three different calling approaches:\n",
|
||||
"\n",
|
||||
"1. **Basic Call**: Directly pass images and text.\n",
|
||||
"2. **Processor Output**: Use HuggingFace processor for data preprocessing.\n",
|
||||
"3. **Precomputed Embeddings**: Pre-calculate image features to improve inference efficiency."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Understanding the Three Input Formats\n",
|
||||
"\n",
|
||||
"SGLang supports three ways to pass visual data, each optimized for different scenarios:\n",
|
||||
"\n",
|
||||
"### 1. **Raw Images** - Simplest approach\n",
|
||||
"- Pass PIL Images, file paths, URLs, or base64 strings directly\n",
|
||||
"- SGLang handles all preprocessing automatically\n",
|
||||
"- Best for: Quick prototyping, simple applications\n",
|
||||
"\n",
|
||||
"### 2. **Processor Output** - For custom preprocessing\n",
|
||||
"- Pre-process images with HuggingFace processor\n",
|
||||
"- Pass the complete processor output dict with `format: \"processor_output\"`\n",
|
||||
"- Best for: Custom image transformations, integration with existing pipelines\n",
|
||||
"- Requirement: Must use `input_ids` instead of text prompt\n",
|
||||
"\n",
|
||||
"### 3. **Precomputed Embeddings** - For maximum performance\n",
|
||||
"- Pre-calculate visual embeddings using the vision encoder\n",
|
||||
"- Pass embeddings with `format: \"precomputed_embedding\"`\n",
|
||||
"- Best for: Repeated queries on same images, caching, high-throughput serving\n",
|
||||
"- Performance gain: Avoids redundant vision encoder computation (30-50% speedup)\n",
|
||||
"\n",
|
||||
"**Key Rule**: Within a single request, use only one format for all images. Don't mix formats.\n",
|
||||
"\n",
|
||||
"The examples below demonstrate all three approaches with both Qwen2.5-VL and Llama 4 models."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Querying Qwen2.5-VL Model"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "3",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import nest_asyncio\n",
|
||||
"\n",
|
||||
"nest_asyncio.apply()\n",
|
||||
"\n",
|
||||
"model_path = \"Qwen/Qwen2.5-VL-3B-Instruct\"\n",
|
||||
"chat_template = \"qwen2-vl\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "4",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from io import BytesIO\n",
|
||||
"import requests\n",
|
||||
"from PIL import Image\n",
|
||||
"\n",
|
||||
"from sglang.srt.parser.conversation import chat_templates\n",
|
||||
"\n",
|
||||
"image = Image.open(\n",
|
||||
" BytesIO(\n",
|
||||
" requests.get(\n",
|
||||
" \"https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true\"\n",
|
||||
" ).content\n",
|
||||
" )\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"conv = chat_templates[chat_template].copy()\n",
|
||||
"conv.append_message(conv.roles[0], f\"What's shown here: {conv.image_token}?\")\n",
|
||||
"conv.append_message(conv.roles[1], \"\")\n",
|
||||
"conv.image_data = [image]\n",
|
||||
"\n",
|
||||
"print(\"Generated prompt text:\")\n",
|
||||
"print(conv.get_prompt())\n",
|
||||
"print(f\"\\nImage size: {image.size}\")\n",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Basic Offline Engine API Call"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "6",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sglang import Engine\n",
|
||||
"\n",
|
||||
"llm = Engine(model_path=model_path, chat_template=chat_template, log_level=\"warning\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "7",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"out = llm.generate(prompt=conv.get_prompt(), image_data=[image])\n",
|
||||
"print(\"Model response:\")\n",
|
||||
"print(out[\"text\"])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Call with Processor Output\n",
|
||||
"\n",
|
||||
"Using a HuggingFace processor to preprocess text and images, and passing the `processor_output` directly into `Engine.generate`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "9",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from transformers import AutoProcessor\n",
|
||||
"\n",
|
||||
"processor = AutoProcessor.from_pretrained(model_path, use_fast=True)\n",
|
||||
"processor_output = processor(\n",
|
||||
" images=[image], text=conv.get_prompt(), return_tensors=\"pt\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"out = llm.generate(\n",
|
||||
" input_ids=processor_output[\"input_ids\"][0].detach().cpu().tolist(),\n",
|
||||
" image_data=[dict(processor_output, format=\"processor_output\")],\n",
|
||||
")\n",
|
||||
"print(\"Response using processor output:\")\n",
|
||||
"print(out[\"text\"])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "10",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Call with Precomputed Embeddings\n",
|
||||
"\n",
|
||||
"You can pre-calculate image features to avoid repeated visual encoding processes."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "11",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from transformers import AutoProcessor\n",
|
||||
"from transformers import Qwen2_5_VLForConditionalGeneration\n",
|
||||
"\n",
|
||||
"processor = AutoProcessor.from_pretrained(model_path, use_fast=True)\n",
|
||||
"vision = (\n",
|
||||
" Qwen2_5_VLForConditionalGeneration.from_pretrained(model_path).eval().visual.cuda()\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "12",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"processor_output = processor(\n",
|
||||
" images=[image], text=conv.get_prompt(), return_tensors=\"pt\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"input_ids = processor_output[\"input_ids\"][0].detach().cpu().tolist()\n",
|
||||
"\n",
|
||||
"precomputed_embeddings = vision(\n",
|
||||
" processor_output[\"pixel_values\"].cuda(), processor_output[\"image_grid_thw\"].cuda()\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"multi_modal_item = dict(\n",
|
||||
" processor_output,\n",
|
||||
" format=\"precomputed_embedding\",\n",
|
||||
" feature=precomputed_embeddings,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"out = llm.generate(input_ids=input_ids, image_data=[multi_modal_item])\n",
|
||||
"print(\"Response using precomputed embeddings:\")\n",
|
||||
"print(out[\"text\"])\n",
|
||||
"\n",
|
||||
"llm.shutdown()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "13",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Querying Llama 4 Vision Model\n",
|
||||
"\n",
|
||||
"```python\n",
|
||||
"model_path = \"meta-llama/Llama-4-Scout-17B-16E-Instruct\"\n",
|
||||
"chat_template = \"llama-4\"\n",
|
||||
"\n",
|
||||
"from io import BytesIO\n",
|
||||
"import requests\n",
|
||||
"from PIL import Image\n",
|
||||
"\n",
|
||||
"from sglang.srt.parser.conversation import chat_templates\n",
|
||||
"\n",
|
||||
"# Download the same example image\n",
|
||||
"image = Image.open(\n",
|
||||
" BytesIO(\n",
|
||||
" requests.get(\n",
|
||||
" \"https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true\"\n",
|
||||
" ).content\n",
|
||||
" )\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"conv = chat_templates[chat_template].copy()\n",
|
||||
"conv.append_message(conv.roles[0], f\"What's shown here: {conv.image_token}?\")\n",
|
||||
"conv.append_message(conv.roles[1], \"\")\n",
|
||||
"conv.image_data = [image]\n",
|
||||
"\n",
|
||||
"print(\"Llama 4 generated prompt text:\")\n",
|
||||
"print(conv.get_prompt())\n",
|
||||
"print(f\"Image size: {image.size}\")\n",
|
||||
"\n",
|
||||
"image\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "14",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Llama 4 Basic Call\n",
|
||||
"\n",
|
||||
"Llama 4 requires more computational resources, so it's configured with multi-GPU parallelism (tp_size=4) and larger context length.\n",
|
||||
"\n",
|
||||
"```python\n",
|
||||
"llm = Engine(\n",
|
||||
" model_path=model_path,\n",
|
||||
" enable_multimodal=True,\n",
|
||||
" attention_backend=\"fa3\",\n",
|
||||
" tp_size=4,\n",
|
||||
" context_length=65536,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"out = llm.generate(prompt=conv.get_prompt(), image_data=[image])\n",
|
||||
"print(\"Llama 4 response:\")\n",
|
||||
"print(out[\"text\"])\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "15",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Call with Processor Output\n",
|
||||
"\n",
|
||||
"Using HuggingFace processor to preprocess data can reduce computational overhead during inference.\n",
|
||||
"\n",
|
||||
"```python\n",
|
||||
"from transformers import AutoProcessor\n",
|
||||
"\n",
|
||||
"processor = AutoProcessor.from_pretrained(model_path, use_fast=True)\n",
|
||||
"processor_output = processor(\n",
|
||||
" images=[image], text=conv.get_prompt(), return_tensors=\"pt\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"out = llm.generate(\n",
|
||||
" input_ids=processor_output[\"input_ids\"][0].detach().cpu().tolist(),\n",
|
||||
" image_data=[dict(processor_output, format=\"processor_output\")],\n",
|
||||
")\n",
|
||||
"print(\"Response using processor output:\")\n",
|
||||
"print(out)\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "16",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Call with Precomputed Embeddings\n",
|
||||
"\n",
|
||||
"```python\n",
|
||||
"from transformers import AutoProcessor\n",
|
||||
"from transformers import Llama4ForConditionalGeneration\n",
|
||||
"\n",
|
||||
"processor = AutoProcessor.from_pretrained(model_path, use_fast=True)\n",
|
||||
"model = Llama4ForConditionalGeneration.from_pretrained(\n",
|
||||
" model_path, torch_dtype=\"auto\"\n",
|
||||
").eval()\n",
|
||||
"\n",
|
||||
"vision = model.vision_model.cuda()\n",
|
||||
"multi_modal_projector = model.multi_modal_projector.cuda()\n",
|
||||
"\n",
|
||||
"print(f'Image pixel values shape: {processor_output[\"pixel_values\"].shape}')\n",
|
||||
"input_ids = processor_output[\"input_ids\"][0].detach().cpu().tolist()\n",
|
||||
"\n",
|
||||
"# Process image through vision encoder\n",
|
||||
"image_outputs = vision(\n",
|
||||
" processor_output[\"pixel_values\"].to(\"cuda\"), \n",
|
||||
" aspect_ratio_ids=processor_output[\"aspect_ratio_ids\"].to(\"cuda\"),\n",
|
||||
" aspect_ratio_mask=processor_output[\"aspect_ratio_mask\"].to(\"cuda\"),\n",
|
||||
" output_hidden_states=False\n",
|
||||
")\n",
|
||||
"image_features = image_outputs.last_hidden_state\n",
|
||||
"\n",
|
||||
"# Flatten image features and pass through multimodal projector\n",
|
||||
"vision_flat = image_features.view(-1, image_features.size(-1))\n",
|
||||
"precomputed_embeddings = multi_modal_projector(vision_flat)\n",
|
||||
"\n",
|
||||
"# Build precomputed embedding data item\n",
|
||||
"mm_item = dict(\n",
|
||||
" processor_output, \n",
|
||||
" format=\"precomputed_embedding\", \n",
|
||||
" feature=precomputed_embeddings\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# Use precomputed embeddings for efficient inference\n",
|
||||
"out = llm.generate(input_ids=input_ids, image_data=[mm_item])\n",
|
||||
"print(\"Llama 4 precomputed embedding response:\")\n",
|
||||
"print(out[\"text\"])\n",
|
||||
"```"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"jupytext": {
|
||||
"cell_metadata_filter": "-all",
|
||||
"custom_cell_magics": "kql",
|
||||
"encoding": "# -*- coding: utf-8 -*-",
|
||||
"text_representation": {
|
||||
"extension": ".py",
|
||||
"format_name": "light",
|
||||
"format_version": "1.5",
|
||||
"jupytext_version": "1.16.1"
|
||||
}
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,275 @@
|
||||
---
|
||||
title: "Query VLM with Offline Engine"
|
||||
metatags:
|
||||
description: "SGLang VLM offline engine: raw images, processor output, precomputed embeddings. Qwen2.5-VL and Llama 4 examples."
|
||||
---
|
||||
This tutorial demonstrates how to use SGLang's **offline Engine API** to query VLMs. We will demonstrate usage with Qwen2.5-VL and Llama 4. This section demonstrates three different calling approaches:
|
||||
|
||||
1. **Basic Call**: Directly pass images and text.
|
||||
2. **Processor Output**: Use HuggingFace processor for data preprocessing.
|
||||
3. **Precomputed Embeddings**: Pre-calculate image features to improve inference efficiency.
|
||||
|
||||
|
||||
## Understanding the Three Input Formats
|
||||
|
||||
SGLang supports three ways to pass visual data, each optimized for different scenarios:
|
||||
|
||||
### 1. **Raw Images** - Simplest approach
|
||||
- Pass PIL Images, file paths, URLs, or base64 strings directly
|
||||
- SGLang handles all preprocessing automatically
|
||||
- Best for: Quick prototyping, simple applications
|
||||
|
||||
### 2. **Processor Output** - For custom preprocessing
|
||||
- Pre-process images with HuggingFace processor
|
||||
- Pass the complete processor output dict with `format: "processor_output"`
|
||||
- Best for: Custom image transformations, integration with existing pipelines
|
||||
- Requirement: Must use `input_ids` instead of text prompt
|
||||
|
||||
### 3. **Precomputed Embeddings** - For maximum performance
|
||||
- Pre-calculate visual embeddings using the vision encoder
|
||||
- Pass embeddings with `format: "precomputed_embedding"`
|
||||
- Best for: Repeated queries on same images, caching, high-throughput serving
|
||||
- Performance gain: Avoids redundant vision encoder computation (30-50% speedup)
|
||||
|
||||
**Key Rule**: Within a single request, use only one format for all images. Don't mix formats.
|
||||
|
||||
The examples below demonstrate all three approaches with both Qwen2.5-VL and Llama 4 models.
|
||||
|
||||
|
||||
## Querying Qwen2.5-VL Model
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import nest_asyncio
|
||||
|
||||
nest_asyncio.apply()
|
||||
|
||||
model_path = "Qwen/Qwen2.5-VL-3B-Instruct"
|
||||
chat_template = "qwen2-vl"
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
from io import BytesIO
|
||||
import requests
|
||||
from PIL import Image
|
||||
|
||||
from sglang.srt.parser.conversation import chat_templates
|
||||
|
||||
image = Image.open(
|
||||
BytesIO(
|
||||
requests.get(
|
||||
"https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true"
|
||||
).content
|
||||
)
|
||||
)
|
||||
|
||||
conv = chat_templates[chat_template].copy()
|
||||
conv.append_message(conv.roles[0], f"What's shown here: {conv.image_token}?")
|
||||
conv.append_message(conv.roles[1], "")
|
||||
conv.image_data = [image]
|
||||
|
||||
print("Generated prompt text:")
|
||||
print(conv.get_prompt())
|
||||
print(f"\nImage size: {image.size}")
|
||||
image
|
||||
```
|
||||
|
||||
### Basic Offline Engine API Call
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
from sglang import Engine
|
||||
|
||||
|
||||
llm = Engine(model_path=model_path, chat_template=chat_template, log_level="warning")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
out = llm.generate(prompt=conv.get_prompt(), image_data=[image])
|
||||
print("Model response:")
|
||||
print(out["text"])
|
||||
```
|
||||
|
||||
### Call with Processor Output
|
||||
|
||||
Using a HuggingFace processor to preprocess text and images, and passing the `processor_output` directly into `Engine.generate`.
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
from transformers import AutoProcessor
|
||||
|
||||
processor = AutoProcessor.from_pretrained(model_path, use_fast=True)
|
||||
processor_output = processor(
|
||||
images=[image], text=conv.get_prompt(), return_tensors="pt"
|
||||
)
|
||||
|
||||
out = llm.generate(
|
||||
input_ids=processor_output["input_ids"][0].detach().cpu().tolist(),
|
||||
image_data=[dict(processor_output, format="processor_output")],
|
||||
)
|
||||
print("Response using processor output:")
|
||||
print(out["text"])
|
||||
```
|
||||
|
||||
### Call with Precomputed Embeddings
|
||||
|
||||
You can pre-calculate image features to avoid repeated visual encoding processes.
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
from transformers import AutoProcessor
|
||||
from transformers import Qwen2_5_VLForConditionalGeneration
|
||||
|
||||
processor = AutoProcessor.from_pretrained(model_path, use_fast=True)
|
||||
vision = (
|
||||
Qwen2_5_VLForConditionalGeneration.from_pretrained(model_path).eval().visual.cuda()
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
processor_output = processor(
|
||||
images=[image], text=conv.get_prompt(), return_tensors="pt"
|
||||
)
|
||||
|
||||
input_ids = processor_output["input_ids"][0].detach().cpu().tolist()
|
||||
|
||||
precomputed_embeddings = vision(
|
||||
processor_output["pixel_values"].cuda(), processor_output["image_grid_thw"].cuda()
|
||||
)
|
||||
|
||||
multi_modal_item = dict(
|
||||
processor_output,
|
||||
format="precomputed_embedding",
|
||||
feature=precomputed_embeddings,
|
||||
)
|
||||
|
||||
out = llm.generate(input_ids=input_ids, image_data=[multi_modal_item])
|
||||
print("Response using precomputed embeddings:")
|
||||
print(out["text"])
|
||||
|
||||
llm.shutdown()
|
||||
```
|
||||
|
||||
## Querying Llama 4 Vision Model
|
||||
|
||||
```python Example
|
||||
model_path = "meta-llama/Llama-4-Scout-17B-16E-Instruct"
|
||||
chat_template = "llama-4"
|
||||
|
||||
from io import BytesIO
|
||||
import requests
|
||||
from PIL import Image
|
||||
|
||||
from sglang.srt.parser.conversation import chat_templates
|
||||
|
||||
# Download the same example image
|
||||
image = Image.open(
|
||||
BytesIO(
|
||||
requests.get(
|
||||
"https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true"
|
||||
).content
|
||||
)
|
||||
)
|
||||
|
||||
conv = chat_templates[chat_template].copy()
|
||||
conv.append_message(conv.roles[0], f"What's shown here: {conv.image_token}?")
|
||||
conv.append_message(conv.roles[1], "")
|
||||
conv.image_data = [image]
|
||||
|
||||
print("Llama 4 generated prompt text:")
|
||||
print(conv.get_prompt())
|
||||
print(f"Image size: {image.size}")
|
||||
|
||||
image
|
||||
```
|
||||
|
||||
|
||||
### Llama 4 Basic Call
|
||||
|
||||
Llama 4 requires more computational resources, so it's configured with multi-GPU parallelism (tp_size=4) and larger context length.
|
||||
|
||||
```python Example
|
||||
llm = Engine(
|
||||
model_path=model_path,
|
||||
enable_multimodal=True,
|
||||
attention_backend="fa3",
|
||||
tp_size=4,
|
||||
context_length=65536,
|
||||
)
|
||||
|
||||
out = llm.generate(prompt=conv.get_prompt(), image_data=[image])
|
||||
print("Llama 4 response:")
|
||||
print(out["text"])
|
||||
```
|
||||
|
||||
|
||||
### Call with Processor Output
|
||||
|
||||
Using HuggingFace processor to preprocess data can reduce computational overhead during inference.
|
||||
|
||||
```python Example
|
||||
from transformers import AutoProcessor
|
||||
|
||||
processor = AutoProcessor.from_pretrained(model_path, use_fast=True)
|
||||
processor_output = processor(
|
||||
images=[image], text=conv.get_prompt(), return_tensors="pt"
|
||||
)
|
||||
|
||||
out = llm.generate(
|
||||
input_ids=processor_output["input_ids"][0].detach().cpu().tolist(),
|
||||
image_data=[dict(processor_output, format="processor_output")],
|
||||
)
|
||||
print("Response using processor output:")
|
||||
print(out)
|
||||
```
|
||||
|
||||
|
||||
### Call with Precomputed Embeddings
|
||||
|
||||
```python Example
|
||||
from transformers import AutoProcessor
|
||||
from transformers import Llama4ForConditionalGeneration
|
||||
|
||||
processor = AutoProcessor.from_pretrained(model_path, use_fast=True)
|
||||
model = Llama4ForConditionalGeneration.from_pretrained(
|
||||
model_path, torch_dtype="auto"
|
||||
).eval()
|
||||
|
||||
vision = model.vision_model.cuda()
|
||||
multi_modal_projector = model.multi_modal_projector.cuda()
|
||||
|
||||
print(f'Image pixel values shape: {processor_output["pixel_values"].shape}')
|
||||
input_ids = processor_output["input_ids"][0].detach().cpu().tolist()
|
||||
|
||||
# Process image through vision encoder
|
||||
image_outputs = vision(
|
||||
processor_output["pixel_values"].to("cuda"),
|
||||
aspect_ratio_ids=processor_output["aspect_ratio_ids"].to("cuda"),
|
||||
aspect_ratio_mask=processor_output["aspect_ratio_mask"].to("cuda"),
|
||||
output_hidden_states=False
|
||||
)
|
||||
image_features = image_outputs.last_hidden_state
|
||||
|
||||
# Flatten image features and pass through multimodal projector
|
||||
vision_flat = image_features.view(-1, image_features.size(-1))
|
||||
precomputed_embeddings = multi_modal_projector(vision_flat)
|
||||
|
||||
# Build precomputed embedding data item
|
||||
mm_item = dict(
|
||||
processor_output,
|
||||
format="precomputed_embedding",
|
||||
feature=precomputed_embeddings
|
||||
)
|
||||
|
||||
# Use precomputed embeddings for efficient inference
|
||||
out = llm.generate(input_ids=input_ids, image_data=[mm_item])
|
||||
print("Llama 4 precomputed embedding response:")
|
||||
print(out["text"])
|
||||
```
|
||||
@@ -0,0 +1,337 @@
|
||||
---
|
||||
title: "DeepSeek V3/V3.1/R1 Usage"
|
||||
metatags:
|
||||
description: "Deploy DeepSeek V3/R1 with SGLang: MLA optimization, FP8 quantization, multi-node TP, DP attention, MTP speculative decoding. Supports H200, B200, MI300X, A100."
|
||||
---
|
||||
SGLang provides many optimizations specifically designed for the DeepSeek models, making it the inference engine recommended by the official [DeepSeek team](https://github.com/deepseek-ai/DeepSeek-V3/tree/main?tab=readme-ov-file#62-inference-with-sglang-recommended) from Day 0.
|
||||
|
||||
This document outlines current optimizations for DeepSeek.
|
||||
For an overview of the implemented features see the completed [Roadmap](https://github.com/sgl-project/sglang/issues/2591).
|
||||
|
||||
## Launch DeepSeek V3.1/V3/R1 with SGLang
|
||||
|
||||
To run DeepSeek V3.1/V3/R1 models, the recommended settings are as follows:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "50%"}} />
|
||||
<col style={{width: "50%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Weight Type</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Configuration</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}} rowSpan={5}>**Full precision [FP8](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528)** *(recommended)*</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8 x H200</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8 x B200</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8 x MI300X</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>2 x 8 x H100/800/20</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Xeon 6980P CPU</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}} rowSpan={4}>**Full precision ([BF16](https://huggingface.co/unsloth/DeepSeek-R1-0528-BF16))** (upcast from original FP8)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>2 x 8 x H200</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>2 x 8 x MI300X</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>4 x 8 x H100/800/20</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>4 x 8 x A100/A800</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}} rowSpan={4}>**Quantized weights ([INT8](https://huggingface.co/meituan/DeepSeek-R1-Channel-INT8))**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16 x A100/800</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>32 x L40S</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Xeon 6980P CPU</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>4 x Atlas 800I A3</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Quantized weights ([W4A8](https://huggingface.co/novita/Deepseek-R1-0528-W4AFP8))**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8 x H20/100, 4 x H200</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}} rowSpan={2}>**Quantized weights ([AWQ](https://huggingface.co/QuixiAI/DeepSeek-R1-0528-AWQ))**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8 x H100/800/20</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8 x A100/A800</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Quantized weights ([MXFP4](https://huggingface.co/amd/DeepSeek-R1-MXFP4-Preview))**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8, 4 x MI355X/350X</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Quantized weights ([NVFP4](https://huggingface.co/nvidia/DeepSeek-R1-0528-NVFP4-v2))**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8, 4 x B200</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
<Callout icon="key" color="#FFC107" iconType="regular">
|
||||
The official DeepSeek V3 is already in FP8 format, so you should not run it with any quantization arguments like `--quantization fp8`.
|
||||
</Callout>
|
||||
|
||||
Detailed commands for reference:
|
||||
|
||||
- [8 x H200](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#using-docker-recommended)
|
||||
- [4 x B200, 8 x B200](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-one-b200-node)
|
||||
- [8 x MI300X](../hardware-platforms/amd-gpus#running-deepseek-v3)
|
||||
- [2 x 8 x H200](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-two-h208-nodes)
|
||||
- [4 x 8 x A100](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-four-a1008-nodes)
|
||||
- [8 x A100 (AWQ)](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-8-a100a800-with-awq-quantization)
|
||||
- [16 x A100 (INT8)](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-16-a100a800-with-int8-quantization)
|
||||
- [32 x L40S (INT8)](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-32-l40s-with-int8-quantization)
|
||||
- [Xeon 6980P CPU](../hardware-platforms/cpu-server#example-running-deepseek-v31-terminus)
|
||||
- [4 x Atlas 800I A3 (int8)](../hardware-platforms/ascend-npus/DeepSeek-Examples#running-deepseek-with-pd-disaggregation-on-4-x-atlas-800i-a3)
|
||||
|
||||
### Download Weights
|
||||
If you encounter errors when starting the server, ensure the weights have finished downloading. It's recommended to download them beforehand or restart multiple times until all weights are downloaded. Please refer to [DeepSeek V3](https://huggingface.co/deepseek-ai/DeepSeek-V3-Base#61-inference-with-deepseek-infer-demo-example-only) official guide to download the weights.
|
||||
|
||||
### Launch with one node of 8 x H200
|
||||
Please refer to [the example](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#installation--launch).
|
||||
|
||||
### Running examples on Multi-Node
|
||||
|
||||
- [Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP](https://lmsys.org/blog/2025-06-16-gb200-part-1/) ([Part I](https://lmsys.org/blog/2025-06-16-gb200-part-1/), [Part II](https://lmsys.org/blog/2025-09-25-gb200-part-2/)) - Comprehensive guide on GB200 optimizations.
|
||||
|
||||
- [Deploying DeepSeek with PD Disaggregation and Large-Scale Expert Parallelism on 96 H100 GPUs](https://lmsys.org/blog/2025-05-05-deepseek-pd-ep/) - Guide on PD disaggregation and large-scale EP.
|
||||
|
||||
- [Serving with two H20*8 nodes](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-two-h208-nodes).
|
||||
|
||||
- [Best Practices for Serving DeepSeek-R1 on H20](https://lmsys.org/blog/2025-09-26-sglang-ant-group/) - Comprehensive guide on H20 optimizations, deployment and performance.
|
||||
|
||||
- [Serving with two H200*8 nodes and docker](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-two-h2008-nodes-and-docker).
|
||||
|
||||
- [Serving with four A100*8 nodes](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-four-a1008-nodes).
|
||||
|
||||
## Optimizations
|
||||
|
||||
### Multi-head Latent Attention (MLA) Throughput Optimizations
|
||||
|
||||
**Description**: [MLA](https://arxiv.org/pdf/2405.04434) is an innovative attention mechanism introduced by the DeepSeek team, aimed at improving inference efficiency. SGLang has implemented specific optimizations for this, including:
|
||||
|
||||
- **Weight Absorption**: By applying the associative law of matrix multiplication to reorder computation steps, this method balances computation and memory access and improves efficiency in the decoding phase.
|
||||
|
||||
- **MLA Attention Backends**: Currently SGLang supports different optimized MLA attention backends, including [FlashAttention3](https://github.com/Dao-AILab/flash-attention), [Flashinfer](https://docs.flashinfer.ai/api/attention.html#flashinfer-mla), [FlashMLA](https://github.com/deepseek-ai/FlashMLA), [CutlassMLA](https://github.com/sgl-project/sglang/pull/5390), **TRTLLM MLA** (optimized for Blackwell architecture), and [Triton](https://github.com/triton-lang/triton) backends. The default FA3 provides good performance across wide workloads.
|
||||
|
||||
- **FP8 Quantization**: W8A8 FP8 and KV Cache FP8 quantization enables efficient FP8 inference. Additionally, we have implemented Batched Matrix Multiplication (BMM) operator to facilitate FP8 inference in MLA with weight absorption.
|
||||
|
||||
- **CUDA Graph & Torch.compile**: Both MLA and Mixture of Experts (MoE) are compatible with CUDA Graph and Torch.compile, which reduces latency and accelerates decoding speed for small batch sizes.
|
||||
|
||||
- **Chunked Prefix Cache**: Chunked prefix cache optimization can increase throughput by cutting prefix cache into chunks, processing them with multi-head attention and merging their states. Its improvement can be significant when doing chunked prefill on long sequences. Currently this optimization is only available for FlashAttention3 backend.
|
||||
|
||||
Overall, with these optimizations, we have achieved up to **7x** acceleration in output throughput compared to the previous version.
|
||||
|
||||
<Frame>
|
||||
<img src="https://lmsys.org/images/blog/sglang_v0_3/deepseek_mla.svg" alt="Multi-head Latent Attention for DeepSeek Series Models"/>
|
||||
</Frame>
|
||||
|
||||
**Usage**: MLA optimization is enabled by default.
|
||||
|
||||
**Reference**: Check [Blog](https://lmsys.org/blog/2024-09-04-sglang-v0-3/#deepseek-multi-head-latent-attention-mla-throughput-optimizations) and [Slides](https://github.com/sgl-project/sgl-learning-materials/blob/main/slides/lmsys_1st_meetup_deepseek_mla.pdf) for more details.
|
||||
|
||||
### Data Parallelism Attention
|
||||
|
||||
**Description**: This optimization involves data parallelism (DP) for the MLA attention mechanism of DeepSeek Series Models, which allows for a significant reduction in the KV cache size, enabling larger batch sizes. Each DP worker independently handles different types of batches (prefill, decode, idle), which are then synchronized before and after processing through the Mixture-of-Experts (MoE) layer. If you do not use DP attention, KV cache will be duplicated among all TP ranks.
|
||||
|
||||
<Frame>
|
||||
<img src="https://lmsys.org/images/blog/sglang_v0_4/dp_attention.svg" alt="Data Parallelism Attention for DeepSeek Series Models"/>
|
||||
</Frame>
|
||||
|
||||
With data parallelism attention enabled, we have achieved up to **1.9x** decoding throughput improvement compared to the previous version.
|
||||
|
||||
<Frame>
|
||||
<img src="https://lmsys.org/images/blog/sglang_v0_4/deepseek_coder_v2.svg" alt="Data Parallelism Attention Performance Comparison"/>
|
||||
</Frame>
|
||||
|
||||
**Usage**:
|
||||
- Append `--enable-dp-attention --tp 8 --dp 8` to the server arguments when using 8 H200 GPUs. This optimization improves peak throughput in high batch size scenarios where the server is limited by KV cache capacity.
|
||||
- DP and TP attention can be flexibly combined. For example, to deploy DeepSeek-V3/R1 on 2 nodes with 8 H100 GPUs each, you can specify `--enable-dp-attention --tp 16 --dp 2`. This configuration runs attention with 2 DP groups, each containing 8 TP GPUs.
|
||||
|
||||
<Warning>
|
||||
Data parallelism attention is not recommended for low-latency, small-batch use cases. It is optimized for high-throughput scenarios with large batch sizes.
|
||||
</Warning>
|
||||
|
||||
**Reference**: Check [Blog](https://lmsys.org/blog/2024-12-04-sglang-v0-4/#data-parallelism-attention-for-deepseek-models).
|
||||
|
||||
### Multi-Node Tensor Parallelism
|
||||
|
||||
**Description**: For users with limited memory on a single node, SGLang supports serving DeepSeek Series Models, including DeepSeek V3, across multiple nodes using tensor parallelism. This approach partitions the model parameters across multiple GPUs or nodes to handle models that are too large for one node's memory.
|
||||
|
||||
**Usage**: Check [here](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-2-h208) for usage examples.
|
||||
|
||||
### Block-wise FP8
|
||||
|
||||
**Description**: SGLang implements block-wise FP8 quantization with two key optimizations:
|
||||
|
||||
- **Activation**: E4M3 format using per-token-per-128-channel sub-vector scales with online casting.
|
||||
|
||||
- **Weight**: Per-128x128-block quantization for better numerical stability.
|
||||
|
||||
- **DeepGEMM**: The [DeepGEMM](https://github.com/deepseek-ai/DeepGEMM) kernel library optimized for FP8 matrix multiplications.
|
||||
|
||||
**Usage**: The activation and weight optimization above are turned on by default for DeepSeek V3 models. DeepGEMM is enabled by default on NVIDIA Hopper/Blackwell GPUs and disabled by default on other devices. DeepGEMM can also be manually turned off by setting the environment variable `SGLANG_ENABLE_JIT_DEEPGEMM=0`.
|
||||
|
||||
<Tip>
|
||||
Before serving the DeepSeek model, precompile the DeepGEMM kernels to improve first-run performance. The precompilation process typically takes around 10 minutes to complete.
|
||||
</Tip>
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-V3 --tp 8 --trust-remote-code
|
||||
```
|
||||
|
||||
### Multi-token Prediction
|
||||
**Description**: SGLang implements DeepSeek V3 Multi-Token Prediction (MTP) based on [EAGLE speculative decoding](../advanced_features/speculative_decoding#EAGLE-Decoding). With this optimization, the decoding speed can be improved by **1.8x** for batch size 1 and **1.5x** for batch size 32 respectively on H200 TP8 setting.
|
||||
|
||||
**Usage**:
|
||||
Add `--speculative-algorithm EAGLE`. Other flags, like `--speculative-num-steps`, `--speculative-eagle-topk` and `--speculative-num-draft-tokens` are optional. For example:
|
||||
```text Output
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3-0324 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--trust-remote-code \
|
||||
--tp 8
|
||||
```
|
||||
- The default configuration for DeepSeek models is `--speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The best configuration for `--speculative-num-steps`, `--speculative-eagle-topk` and `--speculative-num-draft-tokens` can be searched with [bench_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py) script for given batch size. The minimum configuration is `--speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2`, which can achieve speedup for larger batch sizes.
|
||||
- Most MLA attention backends fully support MTP usage. See [MLA Backends](../advanced_features/attention_backend.md#mla-backends) for details.
|
||||
|
||||
<Note>
|
||||
To enable DeepSeek MTP for large batch sizes (>48), you need to adjust some parameters (Reference [this discussion](https://github.com/sgl-project/sglang/issues/4543#issuecomment-2737413756)):
|
||||
- Adjust `--max-running-requests` to a larger number. The default value is `48` for MTP. For larger batch sizes, you should increase this value beyond the default value.
|
||||
- Set `--cuda-graph-bs`. It's a list of batch sizes for cuda graph capture. The [default captured batch sizes for speculative decoding](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/server_args.py#L888-L895) is 48. You can customize this by including more batch sizes.
|
||||
</Note>
|
||||
|
||||
<Tip>
|
||||
To enable the experimental overlap scheduler for EAGLE speculative decoding, set the environment variable `SGLANG_ENABLE_SPEC_V2=1`. This can improve performance by enabling overlap scheduling between draft and verification stages.
|
||||
</Tip>
|
||||
|
||||
|
||||
### Reasoning Content for DeepSeek R1 & V3.1
|
||||
|
||||
See [Reasoning Parser](../advanced_features/separate_reasoning) and [Thinking Parameter for DeepSeek V3.1](./openai_api_completions#Example:-DeepSeek-V3-Models).
|
||||
|
||||
|
||||
### Function calling for DeepSeek Models
|
||||
|
||||
Add arguments `--tool-call-parser deepseekv3` and `--chat-template ./examples/chat_template/tool_chat_template_deepseekv3.jinja`(recommended) to enable this feature. For example (running on 1 * H20 node):
|
||||
|
||||
```text Output
|
||||
python3 -m sglang.launch_server \
|
||||
--model deepseek-ai/DeepSeek-V3-0324 \
|
||||
--tp 8 \
|
||||
--port 30000 \
|
||||
--host 0.0.0.0 \
|
||||
--mem-fraction-static 0.9 \
|
||||
--tool-call-parser deepseekv3 \
|
||||
--chat-template ./examples/chat_template/tool_chat_template_deepseekv3.jinja
|
||||
```
|
||||
|
||||
Sample Request:
|
||||
|
||||
```text Output
|
||||
curl "http://127.0.0.1:30000/v1/chat/completions" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"temperature": 0, "max_tokens": 100, "model": "deepseek-ai/DeepSeek-V3-0324", "tools": [{"type": "function", "function": {"name": "query_weather", "description": "Get weather of an city, the user should supply a city first", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "The city, e.g. Beijing"}}, "required": ["city"]}}}], "messages": [{"role": "user", "content": "Hows the weather like in Qingdao today"}]}'
|
||||
```
|
||||
|
||||
Expected Response
|
||||
|
||||
```text Output
|
||||
{"id":"6501ef8e2d874006bf555bc80cddc7c5","object":"chat.completion","created":1745993638,"model":"deepseek-ai/DeepSeek-V3-0324","choices":[{"index":0,"message":{"role":"assistant","content":null,"reasoning_content":null,"tool_calls":[{"id":"0","index":null,"type":"function","function":{"name":"query_weather","arguments":"{\"city\": \"Qingdao\"}"}}]},"logprobs":null,"finish_reason":"tool_calls","matched_stop":null}],"usage":{"prompt_tokens":116,"total_tokens":138,"completion_tokens":22,"prompt_tokens_details":null}}
|
||||
|
||||
```
|
||||
Sample Streaming Request:
|
||||
```text Output
|
||||
curl "http://127.0.0.1:30000/v1/chat/completions" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"temperature": 0, "max_tokens": 100, "model": "deepseek-ai/DeepSeek-V3-0324","stream":true,"tools": [{"type": "function", "function": {"name": "query_weather", "description": "Get weather of an city, the user should supply a city first", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "The city, e.g. Beijing"}}, "required": ["city"]}}}], "messages": [{"role": "user", "content": "Hows the weather like in Qingdao today"}]}'
|
||||
```
|
||||
Expected Streamed Chunks (simplified for clarity):
|
||||
```text Output
|
||||
data: {"choices":[{"delta":{"tool_calls":[{"function":{"arguments":"{\""}}]}}]}
|
||||
data: {"choices":[{"delta":{"tool_calls":[{"function":{"arguments":"city"}}]}}]}
|
||||
data: {"choices":[{"delta":{"tool_calls":[{"function":{"arguments":"\":\""}}]}}]}
|
||||
data: {"choices":[{"delta":{"tool_calls":[{"function":{"arguments":"Q"}}]}}]}
|
||||
data: {"choices":[{"delta":{"tool_calls":[{"function":{"arguments":"ing"}}]}}]}
|
||||
data: {"choices":[{"delta":{"tool_calls":[{"function":{"arguments":"dao"}}]}}]}
|
||||
data: {"choices":[{"delta":{"tool_calls":[{"function":{"arguments":"\"}"}}]}}]}
|
||||
data: {"choices":[{"delta":{"tool_calls":null}}], "finish_reason": "tool_calls"}
|
||||
data: [DONE]
|
||||
```
|
||||
The client needs to concatenate all arguments fragments to reconstruct the complete tool call:
|
||||
```text Output
|
||||
{"city": "Qingdao"}
|
||||
```
|
||||
<Callout icon="key" color="#FFC107" iconType="regular">
|
||||
1. Use a lower `"temperature"` value for better results.
|
||||
2. To receive more consistent tool call results, it is recommended to use `--chat-template examples/chat_template/tool_chat_template_deepseekv3.jinja`. It provides an improved unified prompt.
|
||||
</Callout>
|
||||
|
||||
|
||||
### Thinking Budget for DeepSeek R1
|
||||
|
||||
In SGLang, we can implement thinking budget with `CustomLogitProcessor`.
|
||||
|
||||
Launch a server with `--enable-custom-logit-processor` flag on.
|
||||
|
||||
```text Output
|
||||
python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-R1 --tp 8 --port 30000 --host 0.0.0.0 --mem-fraction-static 0.9 --disable-cuda-graph --reasoning-parser deepseek-r1 --enable-custom-logit-processor
|
||||
```
|
||||
|
||||
Sample Request:
|
||||
|
||||
<CodeGroup>
|
||||
```python Sample Request
|
||||
import openai
|
||||
from rich.pretty import pprint
|
||||
from sglang.srt.sampling.custom_logit_processor import DeepSeekR1ThinkingBudgetLogitProcessor
|
||||
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-R1",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Question: Is Paris the Capital of France?",
|
||||
}
|
||||
],
|
||||
max_tokens=1024,
|
||||
extra_body={
|
||||
"custom_logit_processor": DeepSeekR1ThinkingBudgetLogitProcessor().to_str(),
|
||||
"custom_params": {
|
||||
"thinking_budget": 512,
|
||||
},
|
||||
},
|
||||
)
|
||||
pprint(response)
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
## FAQ
|
||||
|
||||
**Q: Model loading is taking too long, and I'm encountering an NCCL timeout. What should I do?**
|
||||
|
||||
A: If you're experiencing extended model loading times and an NCCL timeout, you can try increasing the timeout duration. Add the argument `--dist-timeout 3600` when launching your model. This will set the timeout to one hour, which often resolves the issue.
|
||||
@@ -0,0 +1,575 @@
|
||||
---
|
||||
title: "DeepSeek V3.2 Usage"
|
||||
metatags:
|
||||
description: "Deploy DeepSeek V3.2 with SGLang: DeepSeek Sparse Attention (DSA), long-context optimization, MTP speculative decoding, function calling. Supports H200, B200, MI300X, MI350."
|
||||
---
|
||||
DeepSeek-V3.2 model family equips DeepSeek-V3.1-Terminus with DeepSeek Sparse Attention (DSA) through continued training. With DSA, a fine-grained sparse attention mechanism powered by a lightning indexer, DeepSeek-V3.2 achieves efficiency improvements in long-context scenarios.
|
||||
|
||||
For reporting issues or tracking upcoming features, please refer to this [Roadmap](https://github.com/sgl-project/sglang/issues/11060).
|
||||
|
||||
Note: This document is originally written for the usage of [DeepSeek-V3.2-Exp](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp) model. The usage of [DeepSeek-V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2) or [DeepSeek-V3.2-Speciale](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Speciale) is the same as DeepSeek-V3.2-Exp except for the tool call parser.
|
||||
|
||||
|
||||
## Installation
|
||||
|
||||
### Docker
|
||||
|
||||
```bash Command
|
||||
# H200/B200
|
||||
docker pull lmsysorg/sglang:latest
|
||||
|
||||
# MI350/MI355
|
||||
docker pull lmsysorg/sglang:v0.5.8-rocm700-mi35x
|
||||
|
||||
# MI300
|
||||
# v0.5.8-rocm700-mi30x does not include PR #17504. Prefer the newest MI30x ROCm
|
||||
# image tag from Docker Hub when available, or build from source (below).
|
||||
docker pull lmsysorg/sglang:v0.5.8-rocm700-mi30x
|
||||
|
||||
|
||||
# NPUs
|
||||
docker pull lmsysorg/sglang:dsv32-a2
|
||||
docker pull lmsysorg/sglang:dsv32-a3
|
||||
```
|
||||
|
||||
### Build From Source
|
||||
|
||||
```bash Command
|
||||
# Install SGLang
|
||||
git clone https://github.com/sgl-project/sglang
|
||||
cd sglang
|
||||
pip3 install pip --upgrade
|
||||
pip3 install -e "python"
|
||||
```
|
||||
## Launch DeepSeek V3.2 with SGLang
|
||||
|
||||
To serve [DeepSeek-V3.2-Exp](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp) on 8xH200/B200 GPUs:
|
||||
|
||||
```bash Command
|
||||
# Launch with TP + DP (Recommended)
|
||||
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --dp 8 --enable-dp-attention
|
||||
|
||||
# Launch with EP + DP
|
||||
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --ep 8 --dp 8 --enable-dp-attention
|
||||
|
||||
# Launch with Pure TP
|
||||
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8
|
||||
|
||||
# Launch with TP on MI30x/MI35x
|
||||
python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --nsa-prefill-backend tilelang --nsa-decode-backend tilelang
|
||||
```
|
||||
|
||||
### Configuration Tips
|
||||
- **DP Attention (Recommended)**: For DeepSeek V3.2 model, the kernels are customized for the use case of `dp_size=8`, so DP attention (`--dp 8 --enable-dp-attention`) is the recommended configuration for better stability and performance. All test cases use this configuration by default.
|
||||
- **Pure TP Mode**: Launching with pure TP (without `--dp` and `--enable-dp-attention`) is also supported. Note that this mode has not been fully validated in PD disaggregation scenarios.
|
||||
- **Short-sequence MHA prefill (adaptive)**: For short prefill sequences (default threshold: **2048 tokens**), the NSA backend uses standard MHA automatically (no extra flags). On H200 (SM90) this path uses the FlashAttention variable-length kernel; on B200 (SM100) it uses TRT-LLM ragged MHA. MHA uses `MHA_ONE_SHOT` for best performance. `MHA_ONE_SHOT` computes multi-head attention over all tokens (both cached prefix and newly extended tokens) in a single kernel invocation, avoiding the overhead of chunked KV cache processing. This achieves optimal throughput for short sequences where total sequence length fits within the chunk capacity limit.
|
||||
- **Choices of Attention Kernels**: The attention backend is automatically set to `nsa` attention backend for DeepSeek V3.2 model. In this backend, different kernels for sparse prefilling/decoding are implemented, which can be specified by `--nsa-prefill-backend` and `--nsa-decode-backend` server arguments. The choices of nsa prefill/decode attention kernels include:
|
||||
- `flashmla_sparse`: `flash_mla_sparse_fwd` kernel from `flash_mla` library. Can run on both Hopper and Blackwell GPUs. It requires bf16 q, kv inputs.
|
||||
- `flashmla_kv`: `flash_mla_with_kvcache` kernel from `flash_mla` library. Can run on both Hopper and Blackwell GPUs. It requires bf16 q, fp8 k_cache inputs.
|
||||
- `fa3`: `flash_attn_with_kvcache` kernel from `flash_attn` library. Can only run on Hopper GPUs. It requires bf16 q, kv inputs.
|
||||
- `tilelang`: `tilelang` implementation that can run on GPU, HPU and NPU.
|
||||
- `aiter`: Aiter kernel on AMD HPUs. Can only be used as decode kernel.
|
||||
- On the basis of performance benchmarks, the default configuration on H200 and B200 are set as follows :
|
||||
- H200: `flashmla_sparse` prefill attention (short-seq prefill uses MHA via FlashAttention varlen), `fa3` decode attention, `bf16` kv cache dtype.
|
||||
- B200: `flashmla_auto` prefill attention (short-seq prefill uses MHA via TRT-LLM ragged), `flashmla_kv` decode attention, `fp8_e4m3` kv cache dtype. `flashmla_auto` enables automatic selection of either `flashmla_sparse` or `flashmla_kv` kernel for prefill based on KV cache dtype, hardware, and heuristics. When FP8 KV cache is enabled and `total_kv_tokens < total_q_tokens * 512`, it uses the `flashmla_sparse` kernel; otherwise, it falls back to the `flashmla_kv` kernel. The heuristics may need to be tuned if the performance of either the `flashmla_sparse` or `flashmla_kv` kernel changes significantly.
|
||||
|
||||
## Multi-token Prediction
|
||||
SGLang implements Multi-Token Prediction (MTP) for DeepSeek V3.2 based on [EAGLE speculative decoding](../advanced_features/speculative_decoding#EAGLE-Decoding). With this optimization, the decoding speed can be improved significantly on small batch sizes. Please look at [this PR](https://github.com/sgl-project/sglang/pull/11652) for more information.
|
||||
|
||||
Example usage with DP Attention:
|
||||
```bash Command
|
||||
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --dp 8 --enable-dp-attention --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
|
||||
```
|
||||
|
||||
Example usage with Pure TP:
|
||||
```bash Command
|
||||
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
|
||||
```
|
||||
|
||||
- The best configuration for `--speculative-num-steps`, `--speculative-eagle-topk` and `--speculative-num-draft-tokens` can be searched with [bench_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py) script for given batch size. The minimum configuration is `--speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2`, which can achieve speedup for larger batch sizes.
|
||||
- The default value of `--max-running-requests` is set to `48` for MTP. For larger batch sizes, this value should be increased beyond the default value.
|
||||
|
||||
<Tip>
|
||||
To enable the experimental overlap scheduler for EAGLE speculative decoding, set the environment variable `SGLANG_ENABLE_SPEC_V2=1`. This can improve performance by enabling overlap scheduling between draft and verification stages.
|
||||
</Tip>
|
||||
|
||||
|
||||
## Function Calling and Reasoning Parser
|
||||
The usage of function calling and reasoning parser is the same as DeepSeek V3.1. Please refer to [Reasoning Parser](../advanced_features/separate_reasoning) and [Tool Parser](../advanced_features/tool_parser) documents.
|
||||
|
||||
To launch `DeepSeek-V3.2-Exp` with function calling and reasoning parser:
|
||||
<Note>
|
||||
It is recommended to specify the chat-template, ensuring that you are within the sglang's root directory.
|
||||
</Note>
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3.2-Exp \
|
||||
--trust-remote-code \
|
||||
--tp-size 8 --dp-size 8 --enable-dp-attention \
|
||||
--tool-call-parser deepseekv31 \
|
||||
--reasoning-parser deepseek-v3 \
|
||||
--chat-template ./examples/chat_template/tool_chat_template_deepseekv32.jinja
|
||||
```
|
||||
|
||||
To launch `DeepSeek-V3.2` with function calling and reasoning parser:
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3.2 \
|
||||
--trust-remote-code \
|
||||
--tp-size 8 --dp-size 8 --enable-dp-attention \
|
||||
--tool-call-parser deepseekv32 \
|
||||
--reasoning-parser deepseek-v3
|
||||
```
|
||||
|
||||
`DeepSeek-V3.2-Speciale` doesn't support tool calling, so can only be launched with reasoning parser:
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3.2-Speciale \
|
||||
--trust-remote-code \
|
||||
--tp-size 8 --dp-size 8 --enable-dp-attention \
|
||||
--reasoning-parser deepseek-v3
|
||||
```
|
||||
|
||||
|
||||
## PD Disaggregation
|
||||
|
||||
Prefill Command:
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3.2-Exp \
|
||||
--disaggregation-mode prefill \
|
||||
--host $LOCAL_IP \
|
||||
--port $PORT \
|
||||
--tp 8 \
|
||||
--dp 8 \
|
||||
--enable-dp-attention \
|
||||
--dist-init-addr ${HOST}:${DIST_PORT} \
|
||||
--trust-remote-code \
|
||||
--disaggregation-bootstrap-port 8998 \
|
||||
--mem-fraction-static 0.9 \
|
||||
```
|
||||
|
||||
Decode command:
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3.2-Exp \
|
||||
--disaggregation-mode decode \
|
||||
--host $LOCAL_IP \
|
||||
--port $PORT \
|
||||
--tp 8 \
|
||||
--dp 8 \
|
||||
--enable-dp-attention \
|
||||
--dist-init-addr ${HOST}:${DIST_PORT} \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.9 \
|
||||
```
|
||||
|
||||
Router command:
|
||||
```bash Command
|
||||
python -m sglang_router.launch_router --pd-disaggregation \
|
||||
--prefill $PREFILL_ADDR 8998 \
|
||||
--decode $DECODE_ADDR \
|
||||
--host 127.0.0.1 \
|
||||
--port 8000 \
|
||||
```
|
||||
|
||||
If you need more advanced deployment methods or production-ready deployment methods, such as RBG or LWS-based deployment, please refer to [references/multi_node_deployment/rbg_pd/deepseekv32_pd](../references/multi_node_deployment/rbg_pd/deepseekv32_pd). Additionally, you can also find startup commands for DeepEP-based EP parallelism in the aforementioned documentation.
|
||||
|
||||
|
||||
## Benchmarking Results
|
||||
|
||||
### Accuracy Test with `gsm8k`
|
||||
A simple accuracy benchmark can be tested with `gsm8k` dataset:
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1319 --parallel 1319
|
||||
```
|
||||
|
||||
The result is 0.956, which matches our expectation:
|
||||
```bash Command
|
||||
Accuracy: 0.956
|
||||
Invalid: 0.000
|
||||
Latency: 25.109 s
|
||||
Output throughput: 5226.235 token/s
|
||||
```
|
||||
|
||||
To test long-context accuracy, run gsm8k with `--num-shots 20`. The results are very close to the 8 shots results:
|
||||
```text Output
|
||||
Accuracy: 0.956
|
||||
Invalid: 0.000
|
||||
Latency: 29.545 s
|
||||
Output throughput: 4418.617 token/s
|
||||
```
|
||||
|
||||
|
||||
### Accuracy Test with `gpqa-diamond`
|
||||
|
||||
Accuracy benchmark on long context can be tested on GPQA-diamond dataset with long output tokens and thinking enabled:
|
||||
```bash Command
|
||||
python3 -m sglang.test.run_eval --port 30000 --eval-name gpqa --num-examples 198 --max-tokens 128000 --repeat 8 --thinking-mode deepseek-v3
|
||||
```
|
||||
|
||||
The mean accuracy over 8 runs shows 0.797, which matches the number 0.799 in official tech report.
|
||||
```bash Command
|
||||
Repeat: 8, mean: 0.797
|
||||
Scores: ['0.808', '0.798', '0.808', '0.798', '0.783', '0.788', '0.803', '0.793']
|
||||
```
|
||||
|
||||
For Deepseek V3.2, Deepseek recommends setting the sampling parameters to temperature = 1.0, top_p = 0.95:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.test.run_eval --port 30000 --eval-name gpqa --num-examples 198 --max-tokens 128000 --repeat 8 --top-p 0.95 --temperature 1.0 --thinking-mode deepseek-v3
|
||||
|
||||
Repeat: 8, mean: 0.840
|
||||
Scores: ['0.848', '0.808', '0.848', '0.838', '0.879', '0.813', '0.838', '0.848']
|
||||
```
|
||||
which matches the official score, 0.824, as reported in the [Deepseek-V3.2 technical report](https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/main/assets/paper.pdf).
|
||||
|
||||
### Accuracy Test with `aime 2025`
|
||||
|
||||
Prepare the environment by installing NeMo-Skills in the docker or your own virtual environment:
|
||||
|
||||
```text Output
|
||||
pip install git+https://github.com/NVIDIA/NeMo-Skills.git --ignore-installed blinker
|
||||
```
|
||||
|
||||
Then launch the SGLang server:
|
||||
```text Output
|
||||
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --dp 8 --enable-dp-attention
|
||||
```
|
||||
|
||||
**For `DeepSeek-V3.2` and `DeepSeek-V3.2-Speciale`**:
|
||||
|
||||
```text Output
|
||||
python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3.2 --trust-remote-code --tp-size 8 --dp-size 8 --enable-dp-attention --tool-call-parser deepseekv32 --reasoning-parser deepseek-v3
|
||||
```
|
||||
|
||||
Run the following script to evaluate AIME 2025:
|
||||
```text Output
|
||||
#! /bin/bash
|
||||
export NEMO_SKILLS_DISABLE_UNCOMMITTED_CHANGES_CHECK=1
|
||||
|
||||
ns prepare_data aime25
|
||||
|
||||
PORT=30000
|
||||
BACKEND=sglang
|
||||
MODEL="deepseek-ai/DeepSeek-V3.2-Exp" # Should be changed to the model name
|
||||
MODEL_NAME="dsv32-fp8"
|
||||
|
||||
echo "Starting AIME25 evaluation with model $MODEL on port $PORT using backend $BACKEND..."
|
||||
ns eval \
|
||||
--benchmarks=aime25:4 \
|
||||
--server_type=$BACKEND \
|
||||
--model=$MODEL \
|
||||
--server_address=http://localhost:${PORT}/v1 \
|
||||
--output_dir=nemo_skills_aime25_${MODEL_NAME}_output_${BACKEND}_$(date +%Y%m%d_%H%M%S) \
|
||||
++chat_template_kwargs.thinking=true \
|
||||
++inference.temperature=1.0 \
|
||||
++inference.top_p=0.95 \
|
||||
++inference.tokens_to_generate=64000
|
||||
# ++inference.tokens_to_generate=120000 for Speciale model
|
||||
```
|
||||
|
||||
Test results (8*B200):
|
||||
|
||||
DeepSeek-V3.2-Exp：
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "16%"}} />
|
||||
<col style={{width: "16%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>evaluation_mode</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>num_entries</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>avg_tokens</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>gen_seconds</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>symbolic_correct</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>no_answer</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@1[avg-of-4]</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>30</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>15040</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1673</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>87.50% ± 1.67%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.00%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>majority@4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>30</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>15040</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1673</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>90.00%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.00%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>30</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>15040</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1673</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>90.00%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.00%</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
DeepSeek-V3.2:
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "16%"}} />
|
||||
<col style={{width: "16%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>evaluation_mode</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>num_entries</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>avg_tokens</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>gen_seconds</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>symbolic_correct</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>no_answer</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@1[avg-of-4]</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>30</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>13550</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1632</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>92.50% ± 1.67%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.00%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>majority@4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>30</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>13550</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1632</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>94.71%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.00%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>30</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>13550</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1632</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>96.67%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.00%</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
DeepSeek-V3.2-Speciale:
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "16%"}} />
|
||||
<col style={{width: "16%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>evaluation_mode</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>num_entries</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>avg_tokens</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>gen_seconds</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>symbolic_correct</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>no_answer</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@1[avg-of-4]</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>30</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>24155</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>3583</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>95.00% ± 1.92%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.00%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>majority@4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>30</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>24155</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>3583</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>95.83%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.00%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>30</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>24155</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>3583</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>100.00%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.00%</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
|
||||
## DSA long sequence context parallel optimization(experimental)
|
||||
|
||||
**Note: This feature is only verified on Hopper machines**
|
||||
|
||||
For context parallel in DeepSeek V3.2 model, we provide two different modes of splitting tokens, which can be controlled with argument `--nsa-prefill-cp-mode`.
|
||||
|
||||
### In sequence splitting (default setting)
|
||||
|
||||
The first mode can be enabled by `--nsa-prefill-cp-mode in-seq-split`. This mode implements context parallel for DSA by splitting the sequence uniformly between context parallel ranks. At attention stage, each cp rank computes the indexer results of sharded sequence, and collects the whole kv cache through all gather operator.
|
||||
|
||||
The communication group for context parallel reuses the one for attention tp, thus `cp_size` equals `atten_tp_size = tp_size / dp_size`.
|
||||
|
||||
Note that in sequence splitting mode has the following restrictions:
|
||||
- The batch size is restricted to 1 for prefill batches
|
||||
- Multi-node/PD disaggregation is still not supported
|
||||
- `moe_dense_tp_size=1`, `kv_cache_dtype = "bf16"`, `moe_a2a_backend = "deepep"`
|
||||
- To ensure `cp_size > 1`, the passed in `tp_size` must be larger than `dp_size`
|
||||
|
||||
For more details, please refer to PR https://github.com/sgl-project/sglang/pull/12065.
|
||||
|
||||
Example:
|
||||
```bash Command
|
||||
# In-seq splitting mode launched with EP + DP
|
||||
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --ep 8 --dp 2 --enable-dp-attention --enable-nsa-prefill-context-parallel --nsa-prefill-cp-mode in-seq-split --max-running-requests 32
|
||||
```
|
||||
|
||||
### Round robin splitting
|
||||
|
||||
This mode can be enabled by specifying the parameter `--nsa-prefill-cp-mode round-robin-split`, which distributes tokens across ranks based on `token_idx % cp_size`.
|
||||
|
||||
In this scenario, compared with the aforementioned method, it additionally supports the fused MoE backend (the fused MoE backend may deliver better performance than DeepEP in single-machine scenarios), FP8 KV-cache, and multi-batch prefill inference. But it cannot be enabled with dp attention together.
|
||||
|
||||
For more details, please refer to PR https://github.com/sgl-project/sglang/pull/13959.
|
||||
|
||||
Example usage:
|
||||
```bash Command
|
||||
# Launch with FusedMoe + CP8
|
||||
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --enable-nsa-prefill-context-parallel --nsa-prefill-cp-mode round-robin-split --max-running-requests 32
|
||||
```
|
||||
### Pipeline Parallel + Context Parallel (PP + CP)
|
||||
|
||||
This mode combines Pipeline Parallelism (PP) and Context Parallelism (CP) to scale across multiple nodes, which can achieve better throughput and Time To First Token (TTFT). Note that this method has only been tested on H20 96G.
|
||||
|
||||
#### Standard Usage
|
||||
|
||||
To launch with PP=2 and CP (via `round-robin-split` mode) on 2 nodes. This configuration uses the fused MoE kernel by default, which generally provides better performance.
|
||||
|
||||
For related development details, please refer to:
|
||||
- Fused MoE + CP support: [PR #13959](https://github.com/sgl-project/sglang/pull/13959)
|
||||
- PP + CP support: [Issue #15358](https://github.com/sgl-project/sglang/issues/15358) and [PR #16380](https://github.com/sgl-project/sglang/pull/16380)
|
||||
|
||||
Node 0:
|
||||
```bash Command
|
||||
export SGLANG_PP_LAYER_PARTITION=30,31
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3.2-Exp \
|
||||
--nnodes 2 --node-rank 0 \
|
||||
--dist-init-addr <HEAD_NODE_IP>:62001 \
|
||||
--tp 8 --pp-size 2 \
|
||||
--dp-size 1 --moe-dense-tp-size 1 \
|
||||
--enable-nsa-prefill-context-parallel \
|
||||
--nsa-prefill-cp-mode round-robin-split \
|
||||
--trust-remote-code \
|
||||
--disable-radix-cache \
|
||||
--mem-fraction-static 0.8 \
|
||||
--max-running-requests 128 \
|
||||
--chunked-prefill-size 16384 \
|
||||
--cuda-graph-max-bs 8 \
|
||||
--page-size 64 \
|
||||
--watchdog-timeout 3600 \
|
||||
--host 0.0.0.0 --port 8000 \
|
||||
--tool-call-parser deepseekv32
|
||||
```
|
||||
|
||||
Node 1:
|
||||
```bash Command
|
||||
export SGLANG_PP_LAYER_PARTITION=30,31
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3.2-Exp \
|
||||
--nnodes 2 --node-rank 1 \
|
||||
--dist-init-addr <HEAD_NODE_IP>:62001 \
|
||||
--tp 8 --pp-size 2 \
|
||||
--dp-size 1 --moe-dense-tp-size 1 \
|
||||
--enable-nsa-prefill-context-parallel \
|
||||
--nsa-prefill-cp-mode round-robin-split \
|
||||
--trust-remote-code \
|
||||
--disable-radix-cache \
|
||||
--mem-fraction-static 0.8 \
|
||||
--max-running-requests 128 \
|
||||
--chunked-prefill-size 16384 \
|
||||
--cuda-graph-max-bs 8 \
|
||||
--page-size 64 \
|
||||
--watchdog-timeout 3600 \
|
||||
--host 0.0.0.0 --port 8000 \
|
||||
--tool-call-parser deepseekv32
|
||||
```
|
||||
|
||||
#### PD Disaggregation with PP + CP
|
||||
|
||||
If using PD (Prefill-Decode) Disaggregation, the Prefill nodes can be configured with PP + CP as follows.
|
||||
|
||||
Prefill Node 0:
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3.2-Exp \
|
||||
--served-model-name deepseek-v32 \
|
||||
--nnodes 2 --node-rank 0 \
|
||||
--dist-init-addr <PREFILL_HEAD_IP>:20102 \
|
||||
--tp 8 --pp-size 2 \
|
||||
--dp-size 1 --moe-dense-tp-size 1 \
|
||||
--enable-nsa-prefill-context-parallel \
|
||||
--nsa-prefill-cp-mode round-robin-split \
|
||||
--disaggregation-ib-device mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3 \
|
||||
--trust-remote-code \
|
||||
--disable-radix-cache \
|
||||
--max-running-requests 512 \
|
||||
--chunked-prefill-size 4096 \
|
||||
--context-length 131072 \
|
||||
--mem-fraction-static 0.9 \
|
||||
--page-size 64 \
|
||||
--enable-metrics \
|
||||
--collect-tokens-histogram \
|
||||
--tokenizer-worker-num 8 \
|
||||
--host 0.0.0.0 --port 30000
|
||||
```
|
||||
|
||||
Prefill Node 1:
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3.2-Exp \
|
||||
--served-model-name deepseek-v32-prefill \
|
||||
--nnodes 2 --node-rank 1 \
|
||||
--dist-init-addr <PREFILL_HEAD_IP>:20102 \
|
||||
--tp 8 --pp-size 2 \
|
||||
--dp-size 1 --moe-dense-tp-size 1 \
|
||||
--enable-nsa-prefill-context-parallel \
|
||||
--nsa-prefill-cp-mode round-robin-split \
|
||||
--disaggregation-ib-device mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3 \
|
||||
--trust-remote-code \
|
||||
--disable-radix-cache \
|
||||
--max-running-requests 512 \
|
||||
--chunked-prefill-size 4096 \
|
||||
--context-length 131072 \
|
||||
--mem-fraction-static 0.9 \
|
||||
--page-size 64 \
|
||||
--enable-metrics \
|
||||
--collect-tokens-histogram \
|
||||
--tokenizer-worker-num 8 \
|
||||
--host 0.0.0.0 --port 30000
|
||||
```
|
||||
|
||||
For the Decode nodes, it is recommended to use the **EP mode**.
|
||||
@@ -0,0 +1,76 @@
|
||||
---
|
||||
title: "Launch GLM-4.5 / GLM-4.6 / GLM-4.7 with SGLang"
|
||||
metatags:
|
||||
description: "Deploy GLM-4.5/4.6/4.7 models with SGLang: FP8 inference, EAGLE speculative decoding, function calling support. Optimized for H100/H200 GPUs."
|
||||
---
|
||||
|
||||
## Launch GLM-4.5 / GLM-4.6 / GLM-4.7 with SGLang
|
||||
|
||||
To serve GLM-4.5 / GLM-4.6 FP8 models on 8xH100/H200 GPUs:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server --model zai-org/GLM-4.6-FP8 --tp 8
|
||||
```
|
||||
|
||||
### EAGLE Speculative Decoding
|
||||
|
||||
**Description**: SGLang has supported GLM-4.5 / GLM-4.6 models
|
||||
with [EAGLE speculative decoding](../advanced_features/speculative_decoding#EAGLE-Decoding).
|
||||
|
||||
**Usage**:
|
||||
Add arguments `--speculative-algorithm`, `--speculative-num-steps`, `--speculative-eagle-topk` and
|
||||
`--speculative-num-draft-tokens` to enable this feature. For example:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path zai-org/GLM-4.6-FP8 \
|
||||
--tp-size 8 \
|
||||
--tool-call-parser glm45 \
|
||||
--reasoning-parser glm45 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--mem-fraction-static 0.9 \
|
||||
--served-model-name glm-4.6-fp8 \
|
||||
--enable-custom-logit-processor
|
||||
```
|
||||
|
||||
<Tip>
|
||||
To enable the experimental overlap scheduler for EAGLE speculative decoding, set the environment variable `SGLANG_ENABLE_SPEC_V2=1`. This can improve performance by enabling overlap scheduling between draft and verification stages.
|
||||
</Tip>
|
||||
|
||||
### Thinking Budget for GLM-4.5 / GLM-4.6
|
||||
**Note**: For GLM-4.7, `--tool-call-parser` should be set to `glm47`, for GLM-4.5 and GLM-4.6, it should be set to `glm45`.
|
||||
|
||||
In SGLang, we can implement thinking budget with `CustomLogitProcessor`.
|
||||
|
||||
Launch a server with `--enable-custom-logit-processor` flag on.
|
||||
|
||||
Sample Request:
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
from rich.pretty import pprint
|
||||
from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor
|
||||
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.6",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Question: Is Paris the Capital of France?",
|
||||
}
|
||||
],
|
||||
max_tokens=1024,
|
||||
extra_body={
|
||||
"custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(),
|
||||
"custom_params": {
|
||||
"thinking_budget": 512,
|
||||
},
|
||||
},
|
||||
)
|
||||
pprint(response)
|
||||
```
|
||||
@@ -0,0 +1,139 @@
|
||||
---
|
||||
title: "GLM-4.6V / GLM-4.5V Usage"
|
||||
metatags:
|
||||
description: "Deploy GLM-4.6V/4.5V vision models with SGLang: FP8 and BF16 modes, expert parallelism, video understanding. Supports H100, H200, A100 GPUs."
|
||||
---
|
||||
## Launch commands for SGLang
|
||||
|
||||
Below are suggested launch commands tailored for different hardware / precision modes
|
||||
|
||||
### FP8 (quantised) mode
|
||||
|
||||
For high memory-efficiency and latency optimized deployments (e.g., on H100, H200) where FP8 checkpoint is supported:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path zai-org/GLM-4.6V-FP8 \
|
||||
--tp 2 \
|
||||
--ep 2 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--keep-mm-feature-on-device
|
||||
```
|
||||
|
||||
### Non-FP8 (BF16 / full precision) mode
|
||||
For deployments on A100/H100 where BF16 is used (or FP8 snapshot not used):
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path zai-org/GLM-4.6V \
|
||||
--tp 4 \
|
||||
--ep 4 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
## Hardware-specific notes / recommendations
|
||||
|
||||
- On H100 with FP8: Use the FP8 checkpoint for best memory efficiency.
|
||||
- On A100 / H100 with BF16 (non-FP8): It’s recommended to use `--mm-max-concurrent-calls` to control parallel throughput and GPU memory usage during image/video inference.
|
||||
- On H200 & B200: The model can be run “out of the box”, supporting full context length plus concurrent image + video processing.
|
||||
|
||||
## Sending Image/Video Requests
|
||||
|
||||
### Image input:
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
url = f"http://localhost:30000/v1/chat/completions"
|
||||
|
||||
data = {
|
||||
"model": "zai-org/GLM-4.6V",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "What’s in this image?"},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true"
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
"max_tokens": 300,
|
||||
}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
print(response.text)
|
||||
```
|
||||
|
||||
### Video Input:
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
url = f"http://localhost:30000/v1/chat/completions"
|
||||
|
||||
data = {
|
||||
"model": "zai-org/GLM-4.6V",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "What’s happening in this video?"},
|
||||
{
|
||||
"type": "video_url",
|
||||
"video_url": {
|
||||
"url": "https://github.com/sgl-project/sgl-test-files/raw/refs/heads/main/videos/jobs_presenting_ipod.mp4"
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
"max_tokens": 300,
|
||||
}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
print(response.text)
|
||||
```
|
||||
|
||||
## Important Server Parameters and Flags
|
||||
|
||||
When launching the model server for **multimodal support**, you can use the following command-line arguments to fine-tune performance and behavior:
|
||||
|
||||
- `--mm-attention-backend`: Specify multimodal attention backend. Eg. `fa3`(Flash Attention 3)
|
||||
- `--mm-max-concurrent-calls <value>`: Specifies the **maximum number of concurrent asynchronous multimodal data processing calls** allowed on the server. Use this to control parallel throughput and GPU memory usage during image/video inference.
|
||||
- `--mm-per-request-timeout <seconds>`: Defines the **timeout duration (in seconds)** for each multimodal request. If a request exceeds this time limit (e.g., for very large video inputs), it will be automatically terminated.
|
||||
- `--keep-mm-feature-on-device`: Instructs the server to **retain multimodal feature tensors on the GPU** after processing. This avoids device-to-host (D2H) memory copies and improves performance for repeated or high-frequency inference workloads.
|
||||
- `--mm-enable-dp-encoder`: Placing the ViT in data parallel while keeping the LLM in tensor parallel consistently lowers TTFT and boosts end-to-end throughput.
|
||||
- `SGLANG_USE_CUDA_IPC_TRANSPORT=1`: Shared memory pool based CUDA IPC for multi-modal data transport. For significantly improving e2e latency.
|
||||
|
||||
### Example usage with the above optimizations:
|
||||
```bash Command
|
||||
SGLANG_USE_CUDA_IPC_TRANSPORT=1 \
|
||||
SGLANG_VLM_CACHE_SIZE_MB=0 \
|
||||
python -m sglang.launch_server \
|
||||
--model-path zai-org/GLM-4.6V \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--trust-remote-code \
|
||||
--tp-size 8 \
|
||||
--enable-cache-report \
|
||||
--log-level info \
|
||||
--max-running-requests 64 \
|
||||
--mem-fraction-static 0.65 \
|
||||
--chunked-prefill-size 8192 \
|
||||
--attention-backend fa3 \
|
||||
--mm-attention-backend fa3 \
|
||||
--mm-enable-dp-encoder \
|
||||
--enable-metrics
|
||||
```
|
||||
|
||||
### Thinking Budget for GLM-4.5V / GLM-4.6V
|
||||
|
||||
In SGLang, we can implement thinking budget with `CustomLogitProcessor`.
|
||||
|
||||
Launch a server with `--enable-custom-logit-processor` flag on. and using `Glm4MoeThinkingBudgetLogitProcessor` in the request likes `GLM-4.6` example in [glm45](./glm45).
|
||||
@@ -0,0 +1,181 @@
|
||||
---
|
||||
title: "GPT OSS Usage"
|
||||
metatags:
|
||||
description: "Deploy GPT-OSS with SGLang: OpenAI Responses API compatible, built-in tools for web search and Python execution, reasoning levels, MCP tool server support."
|
||||
---
|
||||
Please refer to [#8833](https://github.com/sgl-project/sglang/issues/8833).
|
||||
|
||||
## Responses API & Built-in Tools
|
||||
|
||||
### Responses API
|
||||
|
||||
GPT‑OSS is compatible with the OpenAI Responses API. Use `client.responses.create(...)` with `model`, `instructions`, `input`, and optional `tools` to enable built‑in tool use. You can set reasoning level via `instructions`, e.g., "Reasoning: high" (also supports "medium" and "low") — levels: low (fast), medium (balanced), high (deep).
|
||||
|
||||
### Built-in Tools
|
||||
|
||||
GPT‑OSS can call built‑in tools for web search and Python execution. You can use the demo tool server or connect to external MCP tool servers.
|
||||
|
||||
#### Python Tool
|
||||
|
||||
- Executes short Python snippets for calculations, parsing, and quick scripts.
|
||||
- By default runs in a Docker-based sandbox. To run on the host, set `PYTHON_EXECUTION_BACKEND=UV` (this executes model-generated code locally; use with care).
|
||||
- Ensure Docker is available if you are not using the UV backend. It is recommended to run `docker pull python:3.11` in advance.
|
||||
|
||||
#### Web Search Tool
|
||||
|
||||
- Uses the Exa backend for web search.
|
||||
- Requires an Exa API key; set `EXA_API_KEY` in your environment. Create a key at `https://exa.ai`.
|
||||
|
||||
### Tool & Reasoning Parser
|
||||
|
||||
- We support OpenAI Reasoning and Tool Call parser, as well as our SGLang native api for tool call and reasoning. Refer to [reasoning parser](../advanced_features/separate_reasoning) and [tool call parser](../advanced_features/tool_parser) for more details.
|
||||
|
||||
|
||||
## Notes
|
||||
|
||||
- Use **Python 3.12** for the demo tools. And install the required `gpt-oss` packages.
|
||||
- The default demo integrates the web search tool (Exa backend) and a demo Python interpreter via Docker.
|
||||
- For search, set `EXA_API_KEY`. For Python execution, either have Docker available or set `PYTHON_EXECUTION_BACKEND=UV`.
|
||||
|
||||
Examples:
|
||||
```bash Command
|
||||
export EXA_API_KEY=YOUR_EXA_KEY
|
||||
# Optional: run Python tool locally instead of Docker (use with care)
|
||||
export PYTHON_EXECUTION_BACKEND=UV
|
||||
```
|
||||
|
||||
Launch the server with the demo tool server:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path openai/gpt-oss-120b \
|
||||
--tool-server demo \
|
||||
--tp 2
|
||||
```
|
||||
|
||||
For production usage, sglang can act as an MCP client for multiple services. An [example tool server](https://github.com/openai/gpt-oss/tree/main/gpt-oss-mcp-server) is provided. Start the servers and point sglang to them:
|
||||
```bash Command
|
||||
mcp run -t sse browser_server.py:mcp
|
||||
mcp run -t sse python_server.py:mcp
|
||||
|
||||
python -m sglang.launch_server ... --tool-server ip-1:port-1,ip-2:port-2
|
||||
```
|
||||
The URLs should be MCP SSE servers that expose server information and well-documented tools. These tools are added to the system prompt so the model can use them.
|
||||
|
||||
## Speculative Decoding
|
||||
|
||||
SGLang supports speculative decoding for GPT-OSS models using EAGLE3 algorithm. This can significantly improve decoding speed, especially for small batch sizes.
|
||||
|
||||
**Usage**:
|
||||
Add `--speculative-algorithm EAGLE3` along with the draft model path.
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path openai/gpt-oss-120b \
|
||||
--speculative-algorithm EAGLE3 \
|
||||
--speculative-draft-model-path lmsys/EAGLE3-gpt-oss-120b-bf16 \
|
||||
--tp 2
|
||||
```
|
||||
|
||||
<Tip>
|
||||
To enable the experimental overlap scheduler for EAGLE3 speculative decoding, set the environment variable `SGLANG_ENABLE_SPEC_V2=1`. This can improve performance by enabling overlap scheduling between draft and verification stages.
|
||||
</Tip>
|
||||
|
||||
### Quick Demo
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="sk-123456"
|
||||
)
|
||||
|
||||
tools = [
|
||||
{"type": "code_interpreter"},
|
||||
{"type": "web_search_preview"},
|
||||
]
|
||||
|
||||
# Reasoning level example
|
||||
response = client.responses.create(
|
||||
model="openai/gpt-oss-120b",
|
||||
instructions="You are a helpful assistant."
|
||||
reasoning_effort="high" # Supports high, medium, or low
|
||||
input="In one sentence, explain the transformer architecture.",
|
||||
)
|
||||
print("====== reasoning: high ======")
|
||||
print(response.output_text)
|
||||
|
||||
# Test python tool
|
||||
response = client.responses.create(
|
||||
model="openai/gpt-oss-120b",
|
||||
instructions="You are a helfpul assistant, you could use python tool to execute code.",
|
||||
input="Use python tool to calculate the sum of 29138749187 and 29138749187", # 58,277,498,374
|
||||
tools=tools
|
||||
)
|
||||
print("====== test python tool ======")
|
||||
print(response.output_text)
|
||||
|
||||
# Test browser tool
|
||||
response = client.responses.create(
|
||||
model="openai/gpt-oss-120b",
|
||||
instructions="You are a helfpul assistant, you could use browser to search the web",
|
||||
input="Search the web for the latest news about Nvidia stock price",
|
||||
tools=tools
|
||||
)
|
||||
print("====== test browser tool ======")
|
||||
print(response.output_text)
|
||||
```
|
||||
|
||||
Example output:
|
||||
```text Output
|
||||
====== test python tool ======
|
||||
The sum of 29,138,749,187 and 29,138,749,187 is **58,277,498,374**.
|
||||
====== test browser tool ======
|
||||
**Recent headlines on Nvidia (NVDA) stock**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Date (2025)</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Source</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Key news points</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Stock‑price detail</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**May 13**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Reuters</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The market data page shows Nvidia trading “higher” at **$116.61** with no change from the previous close.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>**$116.61** – latest trade (delayed ≈ 15 min)【14†L34-L38】</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Aug 18**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>CNBC</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Morgan Stanley kept an **overweight** rating and lifted its price target to **$206** (up from $200), implying a 14 % upside from the Friday close. The firm notes Nvidia shares have already **jumped 34 % this year**.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No exact price quoted, but the article signals strong upside expectations【9†L27-L31】</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Aug 20**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The Motley Fool</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Nvidia is set to release its Q2 earnings on Aug 27. The article lists the **current price of $175.36**, down 0.16 % on the day (as of 3:58 p.m. ET).</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>**$175.36** – current price on Aug 20【10†L12-L15】【10†L53-L57】</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**What the news tells us**
|
||||
|
||||
* Nvidia’s share price has risen sharply this year – up roughly a third according to Morgan Stanley – and analysts are still raising targets (now $206).
|
||||
* The most recent market quote (Reuters, May 13) was **$116.61**, but the stock has surged since then, reaching **$175.36** by mid‑August.
|
||||
* Upcoming earnings on **Aug 27** are a focal point; both the Motley Fool and Morgan Stanley expect the results could keep the rally going.
|
||||
|
||||
**Bottom line:** Nvidia’s stock is on a strong upward trajectory in 2025, with price targets climbing toward $200‑$210 and the market price already near $175 as of late August.
|
||||
|
||||
```
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
title: "Kimi-K2.5 Usage"
|
||||
metatags:
|
||||
description: "Deploy Kimi-K2.5 with SGLang: 1T-parameter multimodal MoE model, 256K context, MLA attention, MoonViT vision encoder, thinking and instant modes, tool calling support."
|
||||
---
|
||||
[Kimi-K2.5](https://huggingface.co/moonshotai/Kimi-K2.5) is Moonshot AI's open-source, native multimodal, agentic MoE. It is a 1T-parameter model (32B active) with 256K context, MLA attention, and a MoonViT vision encoder, supporting both thinking and instant modes.
|
||||
|
||||
In SGLang, Kimi-K2.5 uses the `kimi_k2` reasoning and tool-call parsers for correct thinking and tool handling.
|
||||
|
||||
```{note} Example
|
||||
Kimi-K2.5 support is in SGLang main and will land in the next release. Use the latest main or a nightly image until then.
|
||||
```
|
||||
|
||||
Official deployment guide: [Kimi-K2.5 deployment guide](https://huggingface.co/moonshotai/Kimi-K2.5/blob/main/docs/deploy_guidance)
|
||||
|
||||
## Install (Latest Main)
|
||||
|
||||
```bash Command
|
||||
uv pip install "sglang @ git+https://github.com/sgl-project/sglang.git#subdirectory=python"
|
||||
# For CUDA 12:
|
||||
uv pip install "nvidia-cudnn-cu12==9.16.0.29"
|
||||
# For CUDA 13:
|
||||
uv pip install "nvidia-cudnn-cu13==9.16.0.29"
|
||||
```
|
||||
|
||||
## Launch Kimi-K2.5 with SGLang
|
||||
|
||||
Example: single node, TP8 on H200.
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path moonshotai/Kimi-K2.5 \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--tool-call-parser kimi_k2 \
|
||||
--reasoning-parser kimi_k2
|
||||
```
|
||||
|
||||
### Parser Requirements
|
||||
|
||||
- `--tool-call-parser kimi_k2`: Required for tool calling.
|
||||
- `--reasoning-parser kimi_k2`: Required to parse thinking content; thinking mode is enabled by default.
|
||||
|
||||
## Test the Deployment
|
||||
|
||||
Thinking mode is enabled by default. To disable thinking (instant mode), pass `extra_body.chat_template_kwargs.thinking=false`.
|
||||
|
||||
```bash Command
|
||||
# Thinking mode (default)
|
||||
curl http://localhost:30000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "moonshotai/Kimi-K2.5",
|
||||
"messages": [
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "Explain mixture-of-experts in one sentence."}
|
||||
],
|
||||
"max_tokens": 256
|
||||
}'
|
||||
```
|
||||
|
||||
```bash Command
|
||||
# Instant mode (thinking disabled)
|
||||
curl http://localhost:30000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "moonshotai/Kimi-K2.5",
|
||||
"messages": [
|
||||
{"role": "user", "content": "Give one sentence on MoE models."}
|
||||
],
|
||||
"max_tokens": 128,
|
||||
"extra_body": {"chat_template_kwargs": {"thinking": false}}
|
||||
}'
|
||||
```
|
||||
|
||||
## Multimodal Inputs (Image/Video)
|
||||
|
||||
Kimi-K2.5 is multimodal. Image inputs are supported via the OpenAI-compatible vision API. For more details, see `openai_api_vision.ipynb`.
|
||||
|
||||
```bash Command
|
||||
# Image input (SGLang)
|
||||
curl http://localhost:30000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "moonshotai/Kimi-K2.5",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "Describe this image."},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"max_tokens": 256
|
||||
}'
|
||||
```
|
||||
|
||||
<Note>
|
||||
Video chat is experimental and is only supported in the official Moonshot API for now.
|
||||
</Note>
|
||||
@@ -0,0 +1,117 @@
|
||||
---
|
||||
title: "Llama4 Usage"
|
||||
metatags:
|
||||
description: "Deploy Llama 4 Scout (109B) and Maverick (400B) with SGLang: up to 10M context, hybrid KV cache, vision support. Optimized for H100/H200 GPUs."
|
||||
---
|
||||
[Llama 4](https://github.com/meta-llama/llama-models/blob/main/models/llama4/MODEL_CARD) is Meta's latest generation of open-source LLM model with industry-leading performance.
|
||||
|
||||
SGLang has supported Llama 4 Scout (109B) and Llama 4 Maverick (400B) since [v0.4.5](https://github.com/sgl-project/sglang/releases/tag/v0.4.5).
|
||||
|
||||
Ongoing optimizations are tracked in the [Roadmap](https://github.com/sgl-project/sglang/issues/5118).
|
||||
|
||||
## Launch Llama 4 with SGLang
|
||||
|
||||
To serve Llama 4 models on 8xH100/H200 GPUs:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
|
||||
--tp 8 \
|
||||
--context-length 1000000
|
||||
```
|
||||
|
||||
### Configuration Tips
|
||||
|
||||
- **OOM Mitigation**: Adjust `--context-length` to avoid a GPU out-of-memory issue. For the Scout model, we recommend setting this value up to 1M on 8\*H100 and up to 2.5M on 8\*H200. For the Maverick model, we don't need to set context length on 8\*H200. When hybrid kv cache is enabled, `--context-length` can be set up to 5M on 8\*H100 and up to 10M on 8\*H200 for the Scout model.
|
||||
|
||||
- **Attention Backend Auto-Selection**: SGLang automatically selects the optimal attention backend for Llama 4 based on your hardware. You typically don't need to specify `--attention-backend` manually:
|
||||
- **Blackwell GPUs (B200/GB200)**: `trtllm_mha`
|
||||
- **Hopper GPUs (H100/H200)**: `fa3`
|
||||
- **AMD GPUs**: `aiter`
|
||||
- **Intel XPU**: `intel_xpu`
|
||||
- **Other platforms**: `triton` (fallback)
|
||||
|
||||
To override the auto-selection, explicitly specify `--attention-backend` with one of the supported backends: `fa3`, `aiter`, `triton`, `trtllm_mha`, or `intel_xpu`.
|
||||
|
||||
- **Chat Template**: Add `--chat-template llama-4` for chat completion tasks.
|
||||
- **Enable Multi-Modal**: Add `--enable-multimodal` for multi-modal capabilities.
|
||||
- **Enable Hybrid-KVCache**: Set `--swa-full-tokens-ratio` to adjust the ratio of SWA layer (for Llama4, it's local attention layer) KV tokens / full layer KV tokens. (default: 0.8, range: 0-1)
|
||||
|
||||
|
||||
### EAGLE Speculative Decoding
|
||||
**Description**: SGLang has supported Llama 4 Maverick (400B) with [EAGLE speculative decoding](../advanced_features/speculative_decoding#EAGLE-Decoding).
|
||||
|
||||
**Usage**:
|
||||
Add arguments `--speculative-draft-model-path`, `--speculative-algorithm`, `--speculative-num-steps`, `--speculative-eagle-topk` and `--speculative-num-draft-tokens` to enable this feature. For example:
|
||||
```text Output
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--speculative-algorithm EAGLE3 \
|
||||
--speculative-draft-model-path nvidia/Llama-4-Maverick-17B-128E-Eagle3 \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--trust-remote-code \
|
||||
--tp 8 \
|
||||
--context-length 1000000
|
||||
```
|
||||
|
||||
- **Note** The Llama 4 draft model *nvidia/Llama-4-Maverick-17B-128E-Eagle3* can only recognize conversations in chat mode.
|
||||
|
||||
## Benchmarking Results
|
||||
|
||||
### Accuracy Test with `lm_eval`
|
||||
|
||||
The accuracy on SGLang for both Llama4 Scout and Llama4 Maverick can match the [official benchmark numbers](https://ai.meta.com/blog/llama-4-multimodal-intelligence/).
|
||||
|
||||
Benchmark results on MMLU Pro dataset with 8*H100:
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "34%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}></th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama-4-Scout-17B-16E-Instruct</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-4-Maverick-17B-128E-Instruct</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Official Benchmark</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>74.3</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>80.5</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>SGLang</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>75.2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>80.7</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
Commands:
|
||||
|
||||
```bash Command
|
||||
# Llama-4-Scout-17B-16E-Instruct model
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
|
||||
--port 30000 \
|
||||
--tp 8 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--context-length 65536
|
||||
lm_eval --model local-chat-completions --model_args model=meta-llama/Llama-4-Scout-17B-16E-Instruct,base_url=http://localhost:30000/v1/chat/completions,num_concurrent=128,timeout=999999,max_gen_toks=2048 --tasks mmlu_pro --batch_size 128 --apply_chat_template --num_fewshot 0
|
||||
|
||||
# Llama-4-Maverick-17B-128E-Instruct
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--port 30000 \
|
||||
--tp 8 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--context-length 65536
|
||||
lm_eval --model local-chat-completions --model_args model=meta-llama/Llama-4-Maverick-17B-128E-Instruct,base_url=http://localhost:30000/v1/chat/completions,num_concurrent=128,timeout=999999,max_gen_toks=2048 --tasks mmlu_pro --batch_size 128 --apply_chat_template --num_fewshot 0
|
||||
```
|
||||
|
||||
Details can be seen in [this PR](https://github.com/sgl-project/sglang/pull/5092).
|
||||
@@ -0,0 +1,69 @@
|
||||
---
|
||||
title: "MiniMax M2.1/M2 Usage"
|
||||
metatags:
|
||||
description: "Deploy MiniMax M2.1/M2 with SGLang: 230B MoE model (10B active), up to 3M context, optimized for coding and agentic tasks, tool use support."
|
||||
---
|
||||
[MiniMax-M2.1](https://huggingface.co/MiniMaxAI/MiniMax-M2.1) and [MiniMax-M2](https://huggingface.co/MiniMaxAI/MiniMax-M2) are advanced large language models created by [MiniMax](https://www.minimax.io/).
|
||||
|
||||
MiniMax-M2 series redefines efficiency for agents. It's a compact, fast, and cost-effective MoE model (230 billion total parameters with 10 billion active parameters) built for elite performance in coding and agentic tasks, all while maintaining powerful general intelligence. With just 10 billion activated parameters, MiniMax-M2 provides the sophisticated, end-to-end tool use performance expected from today's leading models, but in a streamlined form factor that makes deployment and scaling easier than ever.
|
||||
|
||||
## Supported Models
|
||||
|
||||
This guide applies to the following models. You only need to update the model name during deployment. The following examples use **MiniMax-M2**:
|
||||
|
||||
- [MiniMaxAI/MiniMax-M2.1](https://huggingface.co/MiniMaxAI/MiniMax-M2.1)
|
||||
- [MiniMaxAI/MiniMax-M2](https://huggingface.co/MiniMaxAI/MiniMax-M2)
|
||||
|
||||
## System Requirements
|
||||
|
||||
The following are recommended configurations; actual requirements should be adjusted based on your use case:
|
||||
|
||||
- 4x 96GB GPUs: Supported context length of up to 400K tokens.
|
||||
- 8x 144GB GPUs: Supported context length of up to 3M tokens.
|
||||
|
||||
## Deployment with Python
|
||||
|
||||
4-GPU deployment command:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path MiniMaxAI/MiniMax-M2 \
|
||||
--tp-size 4 \
|
||||
--tool-call-parser minimax-m2 \
|
||||
--reasoning-parser minimax-append-think \
|
||||
--host 0.0.0.0 \
|
||||
--trust-remote-code \
|
||||
--port 8000 \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
|
||||
8-GPU deployment command:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path MiniMaxAI/MiniMax-M2 \
|
||||
--tp-size 8 \
|
||||
--ep-size 8 \
|
||||
--tool-call-parser minimax-m2 \
|
||||
--reasoning-parser minimax-append-think \
|
||||
--host 0.0.0.0 \
|
||||
--trust-remote-code \
|
||||
--port 8000 \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
|
||||
## Testing Deployment
|
||||
|
||||
After startup, you can test the SGLang OpenAI-compatible API with the following command:
|
||||
|
||||
```bash Command
|
||||
curl http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "MiniMaxAI/MiniMax-M2",
|
||||
"messages": [
|
||||
{"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]},
|
||||
{"role": "user", "content": [{"type": "text", "text": "Who won the world series in 2020?"}]}
|
||||
]
|
||||
}'
|
||||
```
|
||||
@@ -0,0 +1,667 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# SGLang Native APIs\n",
|
||||
"\n",
|
||||
"Apart from the OpenAI compatible APIs, the SGLang Runtime also provides its native server APIs. We introduce the following APIs:\n",
|
||||
"\n",
|
||||
"- `/generate` (text generation model)\n",
|
||||
"- `/get_model_info`\n",
|
||||
"- `/get_server_info`\n",
|
||||
"- `/health`\n",
|
||||
"- `/health_generate`\n",
|
||||
"- `/flush_cache`\n",
|
||||
"- `/update_weights`\n",
|
||||
"- `/encode`(embedding model)\n",
|
||||
"- `/v1/rerank`(cross encoder rerank model)\n",
|
||||
"- `/v1/score`(decoder-only scoring)\n",
|
||||
"- `/classify`(reward model)\n",
|
||||
"- `/start_expert_distribution_record`\n",
|
||||
"- `/stop_expert_distribution_record`\n",
|
||||
"- `/dump_expert_distribution_record`\n",
|
||||
"- `/tokenize`\n",
|
||||
"- `/detokenize`\n",
|
||||
"- A full list of these APIs can be found at [http_server.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/entrypoints/http_server.py)\n",
|
||||
"\n",
|
||||
"We mainly use `requests` to test these APIs in the following examples. You can also use `curl`.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Launch A Server"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sglang.test.doc_patch import launch_server_cmd\n",
|
||||
"from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
|
||||
"\n",
|
||||
"server_process, port = launch_server_cmd(\n",
|
||||
" \"python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --log-level warning\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"wait_for_server(f\"http://localhost:{port}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Generate (text generation model)\n",
|
||||
"Generate completions. This is similar to the `/v1/completions` in OpenAI API. Detailed parameters can be found in the [sampling parameters](sampling_params)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import requests\n",
|
||||
"\n",
|
||||
"url = f\"http://localhost:{port}/generate\"\n",
|
||||
"data = {\"text\": \"What is the capital of France?\"}\n",
|
||||
"\n",
|
||||
"response = requests.post(url, json=data)\n",
|
||||
"print_highlight(response.json())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Get Model Info\n",
|
||||
"\n",
|
||||
"Get the information of the model.\n",
|
||||
"\n",
|
||||
"- `model_path`: The path/name of the model.\n",
|
||||
"- `is_generation`: Whether the model is used as generation model or embedding model.\n",
|
||||
"- `tokenizer_path`: The path/name of the tokenizer.\n",
|
||||
"- `preferred_sampling_params`: The default sampling params specified via `--preferred-sampling-params`. `None` is returned in this example as we did not explicitly configure it in server args.\n",
|
||||
"- `weight_version`: This field contains the version of the model weights. This is often used to track changes or updates to the model’s trained parameters.\n",
|
||||
"- `has_image_understanding`: Whether the model has image-understanding capability.\n",
|
||||
"- `has_audio_understanding`: Whether the model has audio-understanding capability.\n",
|
||||
"- `model_type`: The model type from the HuggingFace config (e.g., \"qwen2\", \"llama\").\n",
|
||||
"- `architectures`: The model architectures from the HuggingFace config (e.g., [\"Qwen2ForCausalLM\"])."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"url = f\"http://localhost:{port}/get_model_info\"\n",
|
||||
"\n",
|
||||
"response = requests.get(url)\n",
|
||||
"response_json = response.json()\n",
|
||||
"print_highlight(response_json)\n",
|
||||
"assert response_json[\"model_path\"] == \"qwen/qwen2.5-0.5b-instruct\"\n",
|
||||
"assert response_json[\"is_generation\"] is True\n",
|
||||
"assert response_json[\"tokenizer_path\"] == \"qwen/qwen2.5-0.5b-instruct\"\n",
|
||||
"assert response_json[\"preferred_sampling_params\"] is None\n",
|
||||
"assert response_json.keys() == {\n",
|
||||
" \"model_path\",\n",
|
||||
" \"is_generation\",\n",
|
||||
" \"tokenizer_path\",\n",
|
||||
" \"preferred_sampling_params\",\n",
|
||||
" \"weight_version\",\n",
|
||||
" \"has_image_understanding\",\n",
|
||||
" \"has_audio_understanding\",\n",
|
||||
" \"model_type\",\n",
|
||||
" \"architectures\",\n",
|
||||
"}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Get Server Info\n",
|
||||
"Gets the server information including CLI arguments, token limits, and memory pool sizes.\n",
|
||||
"- Note: `get_server_info` merges the following deprecated endpoints:\n",
|
||||
" - `get_server_args`\n",
|
||||
" - `get_memory_pool_size`\n",
|
||||
" - `get_max_total_num_tokens`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"url = f\"http://localhost:{port}/get_server_info\"\n",
|
||||
"\n",
|
||||
"response = requests.get(url)\n",
|
||||
"print_highlight(response.text)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Health Check\n",
|
||||
"- `/health`: Check the health of the server.\n",
|
||||
"- `/health_generate`: Check the health of the server by generating one token."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"url = f\"http://localhost:{port}/health_generate\"\n",
|
||||
"\n",
|
||||
"response = requests.get(url)\n",
|
||||
"print_highlight(response.text)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"url = f\"http://localhost:{port}/health\"\n",
|
||||
"\n",
|
||||
"response = requests.get(url)\n",
|
||||
"print_highlight(response.text)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Flush Cache\n",
|
||||
"\n",
|
||||
"Flush the radix cache. It will be automatically triggered when the model weights are updated by the `/update_weights` API."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"url = f\"http://localhost:{port}/flush_cache\"\n",
|
||||
"\n",
|
||||
"response = requests.post(url)\n",
|
||||
"print_highlight(response.text)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Update Weights From Disk\n",
|
||||
"\n",
|
||||
"Update model weights from disk without restarting the server. Only applicable for models with the same architecture and parameter size.\n",
|
||||
"\n",
|
||||
"SGLang support `update_weights_from_disk` API for continuous evaluation during training (save checkpoint to disk and update weights from disk).\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# successful update with same architecture and size\n",
|
||||
"\n",
|
||||
"url = f\"http://localhost:{port}/update_weights_from_disk\"\n",
|
||||
"data = {\"model_path\": \"qwen/qwen2.5-0.5b-instruct\"}\n",
|
||||
"\n",
|
||||
"response = requests.post(url, json=data)\n",
|
||||
"print_highlight(response.text)\n",
|
||||
"assert response.json()[\"success\"] is True\n",
|
||||
"assert response.json()[\"message\"] == \"Succeeded to update model weights.\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# failed update with different parameter size or wrong name\n",
|
||||
"\n",
|
||||
"url = f\"http://localhost:{port}/update_weights_from_disk\"\n",
|
||||
"data = {\"model_path\": \"qwen/qwen2.5-0.5b-instruct-wrong\"}\n",
|
||||
"\n",
|
||||
"response = requests.post(url, json=data)\n",
|
||||
"response_json = response.json()\n",
|
||||
"print_highlight(response_json)\n",
|
||||
"assert response_json[\"success\"] is False\n",
|
||||
"assert response_json[\"message\"] == (\n",
|
||||
" \"Failed to get weights iterator: \"\n",
|
||||
" \"qwen/qwen2.5-0.5b-instruct-wrong\"\n",
|
||||
" \" (repository not found).\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(server_process)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Encode (embedding model)\n",
|
||||
"\n",
|
||||
"Encode text into embeddings. Note that this API is only available for [embedding models](openai_api_embeddings) and will raise an error for generation models.\n",
|
||||
"Therefore, we launch a new server to server an embedding model."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"embedding_process, port = launch_server_cmd(\"\"\"\n",
|
||||
"python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \\\n",
|
||||
" --host 0.0.0.0 --is-embedding --log-level warning\n",
|
||||
"\"\"\")\n",
|
||||
"\n",
|
||||
"wait_for_server(f\"http://localhost:{port}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# successful encode for embedding model\n",
|
||||
"\n",
|
||||
"url = f\"http://localhost:{port}/encode\"\n",
|
||||
"data = {\"model\": \"Alibaba-NLP/gte-Qwen2-1.5B-instruct\", \"text\": \"Once upon a time\"}\n",
|
||||
"\n",
|
||||
"response = requests.post(url, json=data)\n",
|
||||
"response_json = response.json()\n",
|
||||
"print_highlight(f\"Text embedding (first 10): {response_json['embedding'][:10]}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(embedding_process)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## v1/rerank (cross encoder rerank model)\n",
|
||||
"Rerank a list of documents given a query using a cross-encoder model. Note that this API is only available for cross encoder model like [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) with `attention-backend` `triton` and `torch_native`.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"reranker_process, port = launch_server_cmd(\"\"\"\n",
|
||||
"python3 -m sglang.launch_server --model-path BAAI/bge-reranker-v2-m3 \\\n",
|
||||
" --host 0.0.0.0 --disable-radix-cache --chunked-prefill-size -1 --attention-backend triton --is-embedding --log-level warning\n",
|
||||
"\"\"\")\n",
|
||||
"\n",
|
||||
"wait_for_server(f\"http://localhost:{port}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# compute rerank scores for query and documents\n",
|
||||
"\n",
|
||||
"url = f\"http://localhost:{port}/v1/rerank\"\n",
|
||||
"data = {\n",
|
||||
" \"model\": \"BAAI/bge-reranker-v2-m3\",\n",
|
||||
" \"query\": \"what is panda?\",\n",
|
||||
" \"documents\": [\n",
|
||||
" \"hi\",\n",
|
||||
" \"The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.\",\n",
|
||||
" ],\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"response = requests.post(url, json=data)\n",
|
||||
"response_json = response.json()\n",
|
||||
"for item in response_json:\n",
|
||||
" print_highlight(f\"Score: {item['score']:.2f} - Document: '{item['document']}'\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(reranker_process)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## v1/score (decoder-only scoring)\n",
|
||||
"\n",
|
||||
"Compute token probabilities for specified tokens given a query and items. This is useful for classification tasks, scoring responses, or computing log-probabilities.\n",
|
||||
"\n",
|
||||
"Parameters:\n",
|
||||
"- `query`: Query text\n",
|
||||
"- `items`: Item text(s) to score\n",
|
||||
"- `label_token_ids`: Token IDs to compute probabilities for\n",
|
||||
"- `apply_softmax`: Whether to apply softmax to get normalized probabilities (default: False)\n",
|
||||
"- `item_first`: Whether items come first in concatenation order (default: False)\n",
|
||||
"- `model`: Model name\n",
|
||||
"\n",
|
||||
"The response contains `scores` - a list of probability lists, one per item, each in the order of `label_token_ids`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"score_process, port = launch_server_cmd(\"\"\"\n",
|
||||
"python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct \\\n",
|
||||
" --host 0.0.0.0 --log-level warning\n",
|
||||
"\"\"\")\n",
|
||||
"\n",
|
||||
"wait_for_server(f\"http://localhost:{port}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Score the probability of different completions given a query\n",
|
||||
"query = \"The capital of France is\"\n",
|
||||
"items = [\"Paris\", \"London\", \"Berlin\"]\n",
|
||||
"\n",
|
||||
"url = f\"http://localhost:{port}/v1/score\"\n",
|
||||
"data = {\n",
|
||||
" \"model\": \"qwen/qwen2.5-0.5b-instruct\",\n",
|
||||
" \"query\": query,\n",
|
||||
" \"items\": items,\n",
|
||||
" \"label_token_ids\": [9454, 2753], # e.g. \"Yes\" and \"No\" token ids\n",
|
||||
" \"apply_softmax\": True, # Normalize probabilities to sum to 1\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"response = requests.post(url, json=data)\n",
|
||||
"response_json = response.json()\n",
|
||||
"\n",
|
||||
"# Display scores for each item\n",
|
||||
"for item, scores in zip(items, response_json[\"scores\"]):\n",
|
||||
" print_highlight(f\"Item '{item}': probabilities = {[f'{s:.4f}' for s in scores]}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(score_process)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Classify (reward model)\n",
|
||||
"\n",
|
||||
"SGLang Runtime also supports reward models. Here we use a reward model to classify the quality of pairwise generations."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Note that SGLang now treats embedding models and reward models as the same type of models.\n",
|
||||
"# This will be updated in the future.\n",
|
||||
"\n",
|
||||
"reward_process, port = launch_server_cmd(\"\"\"\n",
|
||||
"python3 -m sglang.launch_server --model-path Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 --host 0.0.0.0 --is-embedding --log-level warning\n",
|
||||
"\"\"\")\n",
|
||||
"\n",
|
||||
"wait_for_server(f\"http://localhost:{port}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from transformers import AutoTokenizer\n",
|
||||
"\n",
|
||||
"PROMPT = (\n",
|
||||
" \"What is the range of the numeric output of a sigmoid node in a neural network?\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"RESPONSE1 = \"The output of a sigmoid node is bounded between -1 and 1.\"\n",
|
||||
"RESPONSE2 = \"The output of a sigmoid node is bounded between 0 and 1.\"\n",
|
||||
"\n",
|
||||
"CONVS = [\n",
|
||||
" [{\"role\": \"user\", \"content\": PROMPT}, {\"role\": \"assistant\", \"content\": RESPONSE1}],\n",
|
||||
" [{\"role\": \"user\", \"content\": PROMPT}, {\"role\": \"assistant\", \"content\": RESPONSE2}],\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"tokenizer = AutoTokenizer.from_pretrained(\"Skywork/Skywork-Reward-Llama-3.1-8B-v0.2\")\n",
|
||||
"prompts = tokenizer.apply_chat_template(CONVS, tokenize=False, return_dict=False)\n",
|
||||
"\n",
|
||||
"url = f\"http://localhost:{port}/classify\"\n",
|
||||
"data = {\"model\": \"Skywork/Skywork-Reward-Llama-3.1-8B-v0.2\", \"text\": prompts}\n",
|
||||
"\n",
|
||||
"responses = requests.post(url, json=data).json()\n",
|
||||
"for response in responses:\n",
|
||||
" print_highlight(f\"reward: {response['embedding'][0]}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(reward_process)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Capture expert selection distribution in MoE models\n",
|
||||
"\n",
|
||||
"SGLang Runtime supports recording the number of times an expert is selected in a MoE model run for each expert in the model. This is useful when analyzing the throughput of the model and plan for optimization.\n",
|
||||
"\n",
|
||||
"*Note: We only print out the first 10 lines of the csv below for better readability. Please adjust accordingly if you want to analyze the results more deeply.*"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"expert_record_server_process, port = launch_server_cmd(\n",
|
||||
" \"python3 -m sglang.launch_server --model-path Qwen/Qwen1.5-MoE-A2.7B --host 0.0.0.0 --expert-distribution-recorder-mode stat --log-level warning\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"wait_for_server(f\"http://localhost:{port}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"response = requests.post(f\"http://localhost:{port}/start_expert_distribution_record\")\n",
|
||||
"print_highlight(response)\n",
|
||||
"\n",
|
||||
"url = f\"http://localhost:{port}/generate\"\n",
|
||||
"data = {\"text\": \"What is the capital of France?\"}\n",
|
||||
"\n",
|
||||
"response = requests.post(url, json=data)\n",
|
||||
"print_highlight(response.json())\n",
|
||||
"\n",
|
||||
"response = requests.post(f\"http://localhost:{port}/stop_expert_distribution_record\")\n",
|
||||
"print_highlight(response)\n",
|
||||
"\n",
|
||||
"response = requests.post(f\"http://localhost:{port}/dump_expert_distribution_record\")\n",
|
||||
"print_highlight(response)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(expert_record_server_process)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Tokenize/Detokenize Example (Round Trip)\n",
|
||||
"\n",
|
||||
"This example demonstrates how to use the /tokenize and /detokenize endpoints together. We first tokenize a string, then detokenize the resulting IDs to reconstruct the original text. This workflow is useful when you need to handle tokenization externally but still leverage the server for detokenization."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"tokenizer_free_server_process, port = launch_server_cmd(\"\"\"\n",
|
||||
"python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct\n",
|
||||
"\"\"\")\n",
|
||||
"\n",
|
||||
"wait_for_server(f\"http://localhost:{port}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import requests\n",
|
||||
"from sglang.utils import print_highlight\n",
|
||||
"\n",
|
||||
"base_url = f\"http://localhost:{port}\"\n",
|
||||
"tokenize_url = f\"{base_url}/tokenize\"\n",
|
||||
"detokenize_url = f\"{base_url}/detokenize\"\n",
|
||||
"\n",
|
||||
"model_name = \"qwen/qwen2.5-0.5b-instruct\"\n",
|
||||
"input_text = \"SGLang provides efficient tokenization endpoints.\"\n",
|
||||
"print_highlight(f\"Original Input Text:\\n'{input_text}'\")\n",
|
||||
"\n",
|
||||
"# --- tokenize the input text ---\n",
|
||||
"tokenize_payload = {\n",
|
||||
" \"model\": model_name,\n",
|
||||
" \"prompt\": input_text,\n",
|
||||
" \"add_special_tokens\": False,\n",
|
||||
"}\n",
|
||||
"try:\n",
|
||||
" tokenize_response = requests.post(tokenize_url, json=tokenize_payload)\n",
|
||||
" tokenize_response.raise_for_status()\n",
|
||||
" tokenization_result = tokenize_response.json()\n",
|
||||
" token_ids = tokenization_result.get(\"tokens\")\n",
|
||||
"\n",
|
||||
" if not token_ids:\n",
|
||||
" raise ValueError(\"Tokenization returned empty tokens.\")\n",
|
||||
"\n",
|
||||
" print_highlight(f\"\\nTokenized Output (IDs):\\n{token_ids}\")\n",
|
||||
" print_highlight(f\"Token Count: {tokenization_result.get('count')}\")\n",
|
||||
" print_highlight(f\"Max Model Length: {tokenization_result.get('max_model_len')}\")\n",
|
||||
"\n",
|
||||
" # --- detokenize the obtained token IDs ---\n",
|
||||
" detokenize_payload = {\n",
|
||||
" \"model\": model_name,\n",
|
||||
" \"tokens\": token_ids,\n",
|
||||
" \"skip_special_tokens\": True,\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" detokenize_response = requests.post(detokenize_url, json=detokenize_payload)\n",
|
||||
" detokenize_response.raise_for_status()\n",
|
||||
" detokenization_result = detokenize_response.json()\n",
|
||||
" reconstructed_text = detokenization_result.get(\"text\")\n",
|
||||
"\n",
|
||||
" print_highlight(f\"\\nDetokenized Output (Text):\\n'{reconstructed_text}'\")\n",
|
||||
"\n",
|
||||
" if input_text == reconstructed_text:\n",
|
||||
" print_highlight(\n",
|
||||
" \"\\nRound Trip Successful: Original and reconstructed text match.\"\n",
|
||||
" )\n",
|
||||
" else:\n",
|
||||
" print_highlight(\n",
|
||||
" \"\\nRound Trip Mismatch: Original and reconstructed text differ.\"\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"except requests.exceptions.RequestException as e:\n",
|
||||
" print_highlight(f\"\\nHTTP Request Error: {e}\")\n",
|
||||
"except Exception as e:\n",
|
||||
" print_highlight(f\"\\nAn error occurred: {e}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(tokenizer_free_server_process)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
@@ -0,0 +1,450 @@
|
||||
---
|
||||
title: "SGLang Native APIs"
|
||||
metatags:
|
||||
description: "SGLang native server APIs for text generation, embedding, reranking, model info, cache management, and more."
|
||||
---
|
||||
Apart from the OpenAI compatible APIs, the SGLang Runtime also provides its native server APIs. We introduce the following APIs:
|
||||
|
||||
- `/generate` (text generation model)
|
||||
- `/get_model_info`
|
||||
- `/get_server_info`
|
||||
- `/health`
|
||||
- `/health_generate`
|
||||
- `/flush_cache`
|
||||
- `/update_weights`
|
||||
- `/encode`(embedding model)
|
||||
- `/v1/rerank`(cross encoder rerank model)
|
||||
- `/v1/score`(decoder-only scoring)
|
||||
- `/classify`(reward model)
|
||||
- `/start_expert_distribution_record`
|
||||
- `/stop_expert_distribution_record`
|
||||
- `/dump_expert_distribution_record`
|
||||
- `/tokenize`
|
||||
- `/detokenize`
|
||||
- A full list of these APIs can be found at [http_server.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/entrypoints/http_server.py)
|
||||
|
||||
We mainly use `requests` to test these APIs in the following examples. You can also use `curl`.
|
||||
|
||||
## Launch A Server
|
||||
|
||||
```python Example
|
||||
from sglang.test.doc_patch import launch_server_cmd
|
||||
from sglang.utils import wait_for_server, print_highlight, terminate_process
|
||||
|
||||
server_process, port = launch_server_cmd(
|
||||
"python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --log-level warning"
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
```
|
||||
|
||||
## Generate (text generation model)
|
||||
Generate completions. This is similar to the `/v1/completions` in OpenAI API. Detailed parameters can be found in the [sampling parameters](./sampling_params).
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
url = f"http://localhost:{port}/generate"
|
||||
data = {"text": "What is the capital of France?"}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
print_highlight(response.json())
|
||||
```
|
||||
|
||||
## Get Model Info
|
||||
|
||||
Get the information of the model.
|
||||
|
||||
- `model_path`: The path/name of the model.
|
||||
- `is_generation`: Whether the model is used as generation model or embedding model.
|
||||
- `tokenizer_path`: The path/name of the tokenizer.
|
||||
- `preferred_sampling_params`: The default sampling params specified via `--preferred-sampling-params`. `None` is returned in this example as we did not explicitly configure it in server args.
|
||||
- `weight_version`: This field contains the version of the model weights. This is often used to track changes or updates to the model’s trained parameters.
|
||||
- `has_image_understanding`: Whether the model has image-understanding capability.
|
||||
- `has_audio_understanding`: Whether the model has audio-understanding capability.
|
||||
- `model_type`: The model type from the HuggingFace config (e.g., "qwen2", "llama").
|
||||
- `architectures`: The model architectures from the HuggingFace config (e.g., ["Qwen2ForCausalLM"]).
|
||||
|
||||
```python Example
|
||||
url = f"http://localhost:{port}/get_model_info"
|
||||
|
||||
response = requests.get(url)
|
||||
response_json = response.json()
|
||||
print_highlight(response_json)
|
||||
assert response_json["model_path"] == "qwen/qwen2.5-0.5b-instruct"
|
||||
assert response_json["is_generation"] is True
|
||||
assert response_json["tokenizer_path"] == "qwen/qwen2.5-0.5b-instruct"
|
||||
assert response_json["preferred_sampling_params"] is None
|
||||
assert response_json.keys() == {
|
||||
"model_path",
|
||||
"is_generation",
|
||||
"tokenizer_path",
|
||||
"preferred_sampling_params",
|
||||
"weight_version",
|
||||
"has_image_understanding",
|
||||
"has_audio_understanding",
|
||||
"model_type",
|
||||
"architectures",
|
||||
}
|
||||
```
|
||||
|
||||
## Get Server Info
|
||||
Gets the server information including CLI arguments, token limits, and memory pool sizes.
|
||||
- Note: `get_server_info` merges the following deprecated endpoints:
|
||||
- `get_server_args`
|
||||
- `get_memory_pool_size`
|
||||
- `get_max_total_num_tokens`
|
||||
|
||||
```python Example
|
||||
url = f"http://localhost:{port}/get_server_info"
|
||||
|
||||
response = requests.get(url)
|
||||
print_highlight(response.text)
|
||||
```
|
||||
|
||||
## Health Check
|
||||
- `/health`: Check the health of the server.
|
||||
- `/health_generate`: Check the health of the server by generating one token.
|
||||
|
||||
```python Example
|
||||
url = f"http://localhost:{port}/health_generate"
|
||||
|
||||
response = requests.get(url)
|
||||
print_highlight(response.text)
|
||||
```
|
||||
|
||||
```python Example
|
||||
url = f"http://localhost:{port}/health"
|
||||
|
||||
response = requests.get(url)
|
||||
print_highlight(response.text)
|
||||
```
|
||||
|
||||
## Flush Cache
|
||||
|
||||
Flush the radix cache. It will be automatically triggered when the model weights are updated by the `/update_weights` API.
|
||||
|
||||
```python Example
|
||||
url = f"http://localhost:{port}/flush_cache"
|
||||
|
||||
response = requests.post(url)
|
||||
print_highlight(response.text)
|
||||
```
|
||||
|
||||
## Update Weights From Disk
|
||||
|
||||
Update model weights from disk without restarting the server. Only applicable for models with the same architecture and parameter size.
|
||||
|
||||
SGLang support `update_weights_from_disk` API for continuous evaluation during training (save checkpoint to disk and update weights from disk).
|
||||
|
||||
```python Example
|
||||
# successful update with same architecture and size
|
||||
|
||||
url = f"http://localhost:{port}/update_weights_from_disk"
|
||||
data = {"model_path": "qwen/qwen2.5-0.5b-instruct"}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
print_highlight(response.text)
|
||||
assert response.json()["success"] is True
|
||||
assert response.json()["message"] == "Succeeded to update model weights."
|
||||
```
|
||||
|
||||
```python Example
|
||||
# failed update with different parameter size or wrong name
|
||||
|
||||
url = f"http://localhost:{port}/update_weights_from_disk"
|
||||
data = {"model_path": "qwen/qwen2.5-0.5b-instruct-wrong"}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
response_json = response.json()
|
||||
print_highlight(response_json)
|
||||
assert response_json["success"] is False
|
||||
assert response_json["message"] == (
|
||||
"Failed to get weights iterator: "
|
||||
"qwen/qwen2.5-0.5b-instruct-wrong"
|
||||
" (repository not found)."
|
||||
)
|
||||
```
|
||||
|
||||
```python Example
|
||||
terminate_process(server_process)
|
||||
```
|
||||
|
||||
## Encode (embedding model)
|
||||
|
||||
Encode text into embeddings. Note that this API is only available for [embedding models](./openai_api_embeddings) and will raise an error for generation models.
|
||||
Therefore, we launch a new server to server an embedding model.
|
||||
|
||||
```python Example
|
||||
embedding_process, port = launch_server_cmd(
|
||||
"""
|
||||
python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \
|
||||
--host 0.0.0.0 --is-embedding --log-level warning
|
||||
"""
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
```
|
||||
|
||||
```python Example
|
||||
# successful encode for embedding model
|
||||
|
||||
url = f"http://localhost:{port}/encode"
|
||||
data = {"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "text": "Once upon a time"}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
response_json = response.json()
|
||||
print_highlight(f"Text embedding (first 10): {response_json['embedding'][:10]}")
|
||||
```
|
||||
|
||||
```python Example
|
||||
terminate_process(embedding_process)
|
||||
```
|
||||
|
||||
## v1/rerank (cross encoder rerank model)
|
||||
Rerank a list of documents given a query using a cross-encoder model. Note that this API is only available for cross encoder model like [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) with `attention-backend` `triton` and `torch_native`.
|
||||
|
||||
```python Example
|
||||
reranker_process, port = launch_server_cmd(
|
||||
"""
|
||||
python3 -m sglang.launch_server --model-path BAAI/bge-reranker-v2-m3 \
|
||||
--host 0.0.0.0 --disable-radix-cache --chunked-prefill-size -1 --attention-backend triton --is-embedding --log-level warning
|
||||
"""
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
```
|
||||
|
||||
```python Example
|
||||
# compute rerank scores for query and documents
|
||||
|
||||
url = f"http://localhost:{port}/v1/rerank"
|
||||
data = {
|
||||
"model": "BAAI/bge-reranker-v2-m3",
|
||||
"query": "what is panda?",
|
||||
"documents": [
|
||||
"hi",
|
||||
"The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.",
|
||||
],
|
||||
}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
response_json = response.json()
|
||||
for item in response_json:
|
||||
print_highlight(f"Score: {item['score']:.2f} - Document: '{item['document']}'")
|
||||
```
|
||||
|
||||
```python Example
|
||||
terminate_process(reranker_process)
|
||||
```
|
||||
|
||||
## v1/score (decoder-only scoring)
|
||||
|
||||
Compute token probabilities for specified tokens given a query and items. This is useful for classification tasks, scoring responses, or computing log-probabilities.
|
||||
|
||||
Parameters:
|
||||
- `query`: Query text
|
||||
- `items`: Item text(s) to score
|
||||
- `label_token_ids`: Token IDs to compute probabilities for
|
||||
- `apply_softmax`: Whether to apply softmax to get normalized probabilities (default: False)
|
||||
- `item_first`: Whether items come first in concatenation order (default: False)
|
||||
- `model`: Model name
|
||||
|
||||
The response contains `scores` - a list of probability lists, one per item, each in the order of `label_token_ids`.
|
||||
|
||||
```python Example
|
||||
score_process, port = launch_server_cmd(
|
||||
"""
|
||||
python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct \
|
||||
--host 0.0.0.0 --log-level warning
|
||||
"""
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
```
|
||||
|
||||
```python Example
|
||||
# Score the probability of different completions given a query
|
||||
query = "The capital of France is"
|
||||
items = ["Paris", "London", "Berlin"]
|
||||
|
||||
url = f"http://localhost:{port}/v1/score"
|
||||
data = {
|
||||
"model": "qwen/qwen2.5-0.5b-instruct",
|
||||
"query": query,
|
||||
"items": items,
|
||||
"label_token_ids": [9454, 2753], # e.g. "Yes" and "No" token ids
|
||||
"apply_softmax": True, # Normalize probabilities to sum to 1
|
||||
}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
response_json = response.json()
|
||||
|
||||
# Display scores for each item
|
||||
for item, scores in zip(items, response_json["scores"]):
|
||||
print_highlight(f"Item '{item}': probabilities = {[f'{s:.4f}' for s in scores]}")
|
||||
```
|
||||
|
||||
```python Example
|
||||
terminate_process(score_process)
|
||||
```
|
||||
|
||||
## Classify (reward model)
|
||||
|
||||
SGLang Runtime also supports reward models. Here we use a reward model to classify the quality of pairwise generations.
|
||||
|
||||
```python Example
|
||||
# Note that SGLang now treats embedding models and reward models as the same type of models.
|
||||
# This will be updated in the future.
|
||||
|
||||
reward_process, port = launch_server_cmd(
|
||||
"""
|
||||
python3 -m sglang.launch_server --model-path Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 --host 0.0.0.0 --is-embedding --log-level warning
|
||||
"""
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
```
|
||||
|
||||
```python Example
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
PROMPT = (
|
||||
"What is the range of the numeric output of a sigmoid node in a neural network?"
|
||||
)
|
||||
|
||||
RESPONSE1 = "The output of a sigmoid node is bounded between -1 and 1."
|
||||
RESPONSE2 = "The output of a sigmoid node is bounded between 0 and 1."
|
||||
|
||||
CONVS = [
|
||||
[{"role": "user", "content": PROMPT}, {"role": "assistant", "content": RESPONSE1}],
|
||||
[{"role": "user", "content": PROMPT}, {"role": "assistant", "content": RESPONSE2}],
|
||||
]
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("Skywork/Skywork-Reward-Llama-3.1-8B-v0.2")
|
||||
prompts = tokenizer.apply_chat_template(CONVS, tokenize=False, return_dict=False)
|
||||
|
||||
url = f"http://localhost:{port}/classify"
|
||||
data = {"model": "Skywork/Skywork-Reward-Llama-3.1-8B-v0.2", "text": prompts}
|
||||
|
||||
responses = requests.post(url, json=data).json()
|
||||
for response in responses:
|
||||
print_highlight(f"reward: {response['embedding'][0]}")
|
||||
```
|
||||
|
||||
```python Example
|
||||
terminate_process(reward_process)
|
||||
```
|
||||
|
||||
## Capture expert selection distribution in MoE models
|
||||
|
||||
SGLang Runtime supports recording the number of times an expert is selected in a MoE model run for each expert in the model. This is useful when analyzing the throughput of the model and plan for optimization.
|
||||
|
||||
*Note: We only print out the first 10 lines of the csv below for better readability. Please adjust accordingly if you want to analyze the results more deeply.*
|
||||
|
||||
```python Example
|
||||
expert_record_server_process, port = launch_server_cmd(
|
||||
"python3 -m sglang.launch_server --model-path Qwen/Qwen1.5-MoE-A2.7B --host 0.0.0.0 --expert-distribution-recorder-mode stat --log-level warning"
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
```
|
||||
|
||||
```python Example
|
||||
response = requests.post(f"http://localhost:{port}/start_expert_distribution_record")
|
||||
print_highlight(response)
|
||||
|
||||
url = f"http://localhost:{port}/generate"
|
||||
data = {"text": "What is the capital of France?"}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
print_highlight(response.json())
|
||||
|
||||
response = requests.post(f"http://localhost:{port}/stop_expert_distribution_record")
|
||||
print_highlight(response)
|
||||
|
||||
response = requests.post(f"http://localhost:{port}/dump_expert_distribution_record")
|
||||
print_highlight(response)
|
||||
```
|
||||
|
||||
```python Example
|
||||
terminate_process(expert_record_server_process)
|
||||
```
|
||||
|
||||
## Tokenize/Detokenize Example (Round Trip)
|
||||
|
||||
This example demonstrates how to use the /tokenize and /detokenize endpoints together. We first tokenize a string, then detokenize the resulting IDs to reconstruct the original text. This workflow is useful when you need to handle tokenization externally but still leverage the server for detokenization.
|
||||
|
||||
```python Example
|
||||
tokenizer_free_server_process, port = launch_server_cmd(
|
||||
"""
|
||||
python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct
|
||||
"""
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
```
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
from sglang.utils import print_highlight
|
||||
|
||||
base_url = f"http://localhost:{port}"
|
||||
tokenize_url = f"{base_url}/tokenize"
|
||||
detokenize_url = f"{base_url}/detokenize"
|
||||
|
||||
model_name = "qwen/qwen2.5-0.5b-instruct"
|
||||
input_text = "SGLang provides efficient tokenization endpoints."
|
||||
print_highlight(f"Original Input Text:\n'{input_text}'")
|
||||
|
||||
# --- tokenize the input text ---
|
||||
tokenize_payload = {
|
||||
"model": model_name,
|
||||
"prompt": input_text,
|
||||
"add_special_tokens": False,
|
||||
}
|
||||
try:
|
||||
tokenize_response = requests.post(tokenize_url, json=tokenize_payload)
|
||||
tokenize_response.raise_for_status()
|
||||
tokenization_result = tokenize_response.json()
|
||||
token_ids = tokenization_result.get("tokens")
|
||||
|
||||
if not token_ids:
|
||||
raise ValueError("Tokenization returned empty tokens.")
|
||||
|
||||
print_highlight(f"\nTokenized Output (IDs):\n{token_ids}")
|
||||
print_highlight(f"Token Count: {tokenization_result.get('count')}")
|
||||
print_highlight(f"Max Model Length: {tokenization_result.get('max_model_len')}")
|
||||
|
||||
# --- detokenize the obtained token IDs ---
|
||||
detokenize_payload = {
|
||||
"model": model_name,
|
||||
"tokens": token_ids,
|
||||
"skip_special_tokens": True,
|
||||
}
|
||||
|
||||
detokenize_response = requests.post(detokenize_url, json=detokenize_payload)
|
||||
detokenize_response.raise_for_status()
|
||||
detokenization_result = detokenize_response.json()
|
||||
reconstructed_text = detokenization_result.get("text")
|
||||
|
||||
print_highlight(f"\nDetokenized Output (Text):\n'{reconstructed_text}'")
|
||||
|
||||
if input_text == reconstructed_text:
|
||||
print_highlight(
|
||||
"\nRound Trip Successful: Original and reconstructed text match."
|
||||
)
|
||||
else:
|
||||
print_highlight(
|
||||
"\nRound Trip Mismatch: Original and reconstructed text differ."
|
||||
)
|
||||
|
||||
except requests.exceptions.RequestException as e:
|
||||
print_highlight(f"\nHTTP Request Error: {e}")
|
||||
except Exception as e:
|
||||
print_highlight(f"\nAn error occurred: {e}")
|
||||
```
|
||||
|
||||
```python Example
|
||||
terminate_process(tokenizer_free_server_process)
|
||||
```
|
||||
@@ -0,0 +1,235 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Offline Engine API\n",
|
||||
"\n",
|
||||
"SGLang provides a direct inference engine without the need for an HTTP server, especially for use cases where additional HTTP server adds unnecessary complexity or overhead. Here are two general use cases:\n",
|
||||
"\n",
|
||||
"- Offline Batch Inference\n",
|
||||
"- Custom Server on Top of the Engine\n",
|
||||
"\n",
|
||||
"This document focuses on the offline batch inference, demonstrating four different inference modes:\n",
|
||||
"\n",
|
||||
"- Non-streaming synchronous generation\n",
|
||||
"- Streaming synchronous generation\n",
|
||||
"- Non-streaming asynchronous generation\n",
|
||||
"- Streaming asynchronous generation\n",
|
||||
"\n",
|
||||
"Additionally, you can easily build a custom server on top of the SGLang offline engine. A detailed example working in a python script can be found in [custom_server](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/custom_server.py).\n",
|
||||
"\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Nest Asyncio\n",
|
||||
"Note that if you want to use **Offline Engine** in ipython or some other nested loop code, you need to add the following code:\n",
|
||||
"```python\n",
|
||||
"import nest_asyncio\n",
|
||||
"\n",
|
||||
"nest_asyncio.apply()\n",
|
||||
"\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Advanced Usage\n",
|
||||
"\n",
|
||||
"The engine supports [vlm inference](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/offline_batch_inference_vlm.py) as well as [extracting hidden states](https://github.com/sgl-project/sglang/blob/main/examples/runtime/hidden_states). \n",
|
||||
"\n",
|
||||
"Please see [the examples](https://github.com/sgl-project/sglang/tree/main/examples/runtime/engine) for further use cases."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Offline Batch Inference\n",
|
||||
"\n",
|
||||
"SGLang offline engine supports batch inference with efficient scheduling."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# launch the offline engine\n",
|
||||
"import asyncio\n",
|
||||
"\n",
|
||||
"import sglang as sgl\n",
|
||||
"import sglang.test.doc_patch\n",
|
||||
"from sglang.utils import async_stream_and_merge, stream_and_merge\n",
|
||||
"\n",
|
||||
"llm = sgl.Engine(model_path=\"qwen/qwen2.5-0.5b-instruct\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Non-streaming Synchronous Generation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"prompts = [\n",
|
||||
" \"Hello, my name is\",\n",
|
||||
" \"The president of the United States is\",\n",
|
||||
" \"The capital of France is\",\n",
|
||||
" \"The future of AI is\",\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"sampling_params = {\"temperature\": 0.8, \"top_p\": 0.95}\n",
|
||||
"\n",
|
||||
"outputs = llm.generate(prompts, sampling_params)\n",
|
||||
"for prompt, output in zip(prompts, outputs):\n",
|
||||
" print(\"===============================\")\n",
|
||||
" print(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Streaming Synchronous Generation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"prompts = [\n",
|
||||
" \"Write a short, neutral self-introduction for a fictional character. Hello, my name is\",\n",
|
||||
" \"Provide a concise factual statement about France’s capital city. The capital of France is\",\n",
|
||||
" \"Explain possible future trends in artificial intelligence. The future of AI is\",\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"sampling_params = {\n",
|
||||
" \"temperature\": 0.2,\n",
|
||||
" \"top_p\": 0.9,\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"print(\"\\n=== Testing synchronous streaming generation with overlap removal ===\\n\")\n",
|
||||
"\n",
|
||||
"for prompt in prompts:\n",
|
||||
" print(f\"Prompt: {prompt}\")\n",
|
||||
" merged_output = stream_and_merge(llm, prompt, sampling_params)\n",
|
||||
" print(\"Generated text:\", merged_output)\n",
|
||||
" print()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Non-streaming Asynchronous Generation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"prompts = [\n",
|
||||
" \"Write a short, neutral self-introduction for a fictional character. Hello, my name is\",\n",
|
||||
" \"Provide a concise factual statement about France’s capital city. The capital of France is\",\n",
|
||||
" \"Explain possible future trends in artificial intelligence. The future of AI is\",\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"sampling_params = {\"temperature\": 0.8, \"top_p\": 0.95}\n",
|
||||
"\n",
|
||||
"print(\"\\n=== Testing asynchronous batch generation ===\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"async def main():\n",
|
||||
" outputs = await llm.async_generate(prompts, sampling_params)\n",
|
||||
"\n",
|
||||
" for prompt, output in zip(prompts, outputs):\n",
|
||||
" print(f\"\\nPrompt: {prompt}\")\n",
|
||||
" print(f\"Generated text: {output['text']}\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"asyncio.run(main())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Streaming Asynchronous Generation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"prompts = [\n",
|
||||
" \"Write a short, neutral self-introduction for a fictional character. Hello, my name is\",\n",
|
||||
" \"Provide a concise factual statement about France’s capital city. The capital of France is\",\n",
|
||||
" \"Explain possible future trends in artificial intelligence. The future of AI is\",\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"sampling_params = {\"temperature\": 0.8, \"top_p\": 0.95}\n",
|
||||
"\n",
|
||||
"print(\"\\n=== Testing asynchronous streaming generation (no repeats) ===\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"async def main():\n",
|
||||
" for prompt in prompts:\n",
|
||||
" print(f\"\\nPrompt: {prompt}\")\n",
|
||||
" print(\"Generated text: \", end=\"\", flush=True)\n",
|
||||
"\n",
|
||||
" # Replace direct calls to async_generate with our custom overlap-aware version\n",
|
||||
" async for cleaned_chunk in async_stream_and_merge(llm, prompt, sampling_params):\n",
|
||||
" print(cleaned_chunk, end=\"\", flush=True)\n",
|
||||
"\n",
|
||||
" print() # New line after each prompt\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"asyncio.run(main())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"llm.shutdown()"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,143 @@
|
||||
---
|
||||
title: "Offline Engine API"
|
||||
metatags:
|
||||
description: "Use SGLang's offline engine for direct batch inference without HTTP server overhead. Supports sync/async and streaming modes."
|
||||
---
|
||||
SGLang provides a direct inference engine without the need for an HTTP server, especially for use cases where additional HTTP server adds unnecessary complexity or overhead. Here are two general use cases:
|
||||
|
||||
- Offline Batch Inference
|
||||
- Custom Server on Top of the Engine
|
||||
|
||||
This document focuses on the offline batch inference, demonstrating four different inference modes:
|
||||
|
||||
- Non-streaming synchronous generation
|
||||
- Streaming synchronous generation
|
||||
- Non-streaming asynchronous generation
|
||||
- Streaming asynchronous generation
|
||||
|
||||
Additionally, you can easily build a custom server on top of the SGLang offline engine. A detailed example working in a python script can be found in [custom_server](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/custom_server.py).
|
||||
|
||||
## Nest Asyncio
|
||||
Note that if you want to use **Offline Engine** in ipython or some other nested loop code, you need to add the following code:
|
||||
```python Example
|
||||
import nest_asyncio
|
||||
|
||||
nest_asyncio.apply()
|
||||
|
||||
```
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
The engine supports [vlm inference](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/offline_batch_inference_vlm.py) as well as [extracting hidden states](https://github.com/sgl-project/sglang/tree/main/examples/runtime/hidden_states).
|
||||
|
||||
Please see [the examples](https://github.com/sgl-project/sglang/tree/main/examples/runtime/engine) for further use cases.
|
||||
|
||||
## Offline Batch Inference
|
||||
|
||||
SGLang offline engine supports batch inference with efficient scheduling.
|
||||
|
||||
```python Example
|
||||
# launch the offline engine
|
||||
import asyncio
|
||||
|
||||
import sglang as sgl
|
||||
import sglang.test.doc_patch
|
||||
from sglang.utils import async_stream_and_merge, stream_and_merge
|
||||
|
||||
llm = sgl.Engine(model_path="qwen/qwen2.5-0.5b-instruct")
|
||||
```
|
||||
|
||||
### Non-streaming Synchronous Generation
|
||||
|
||||
```python Example
|
||||
prompts = [
|
||||
"Hello, my name is",
|
||||
"The president of the United States is",
|
||||
"The capital of France is",
|
||||
"The future of AI is",
|
||||
]
|
||||
|
||||
sampling_params = {"temperature": 0.8, "top_p": 0.95}
|
||||
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
for prompt, output in zip(prompts, outputs):
|
||||
print("===============================")
|
||||
print(f"Prompt: {prompt}\nGenerated text: {output['text']}")
|
||||
```
|
||||
|
||||
### Streaming Synchronous Generation
|
||||
|
||||
```python Example
|
||||
prompts = [
|
||||
"Write a short, neutral self-introduction for a fictional character. Hello, my name is",
|
||||
"Provide a concise factual statement about France’s capital city. The capital of France is",
|
||||
"Explain possible future trends in artificial intelligence. The future of AI is",
|
||||
]
|
||||
|
||||
sampling_params = {
|
||||
"temperature": 0.2,
|
||||
"top_p": 0.9,
|
||||
}
|
||||
|
||||
print("\n=== Testing synchronous streaming generation with overlap removal ===\n")
|
||||
|
||||
for prompt in prompts:
|
||||
print(f"Prompt: {prompt}")
|
||||
merged_output = stream_and_merge(llm, prompt, sampling_params)
|
||||
print("Generated text:", merged_output)
|
||||
print()
|
||||
```
|
||||
|
||||
### Non-streaming Asynchronous Generation
|
||||
|
||||
```python Example
|
||||
prompts = [
|
||||
"Write a short, neutral self-introduction for a fictional character. Hello, my name is",
|
||||
"Provide a concise factual statement about France’s capital city. The capital of France is",
|
||||
"Explain possible future trends in artificial intelligence. The future of AI is",
|
||||
]
|
||||
|
||||
sampling_params = {"temperature": 0.8, "top_p": 0.95}
|
||||
|
||||
print("\n=== Testing asynchronous batch generation ===")
|
||||
|
||||
async def main():
|
||||
outputs = await llm.async_generate(prompts, sampling_params)
|
||||
|
||||
for prompt, output in zip(prompts, outputs):
|
||||
print(f"\nPrompt: {prompt}")
|
||||
print(f"Generated text: {output['text']}")
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Streaming Asynchronous Generation
|
||||
|
||||
```python Example
|
||||
prompts = [
|
||||
"Write a short, neutral self-introduction for a fictional character. Hello, my name is",
|
||||
"Provide a concise factual statement about France’s capital city. The capital of France is",
|
||||
"Explain possible future trends in artificial intelligence. The future of AI is",
|
||||
]
|
||||
|
||||
sampling_params = {"temperature": 0.8, "top_p": 0.95}
|
||||
|
||||
print("\n=== Testing asynchronous streaming generation (no repeats) ===")
|
||||
|
||||
async def main():
|
||||
for prompt in prompts:
|
||||
print(f"\nPrompt: {prompt}")
|
||||
print("Generated text: ", end="", flush=True)
|
||||
|
||||
# Replace direct calls to async_generate with our custom overlap-aware version
|
||||
async for cleaned_chunk in async_stream_and_merge(llm, prompt, sampling_params):
|
||||
print(cleaned_chunk, end="", flush=True)
|
||||
|
||||
print() # New line after each prompt
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
```python Example
|
||||
llm.shutdown()
|
||||
```
|
||||
@@ -0,0 +1,157 @@
|
||||
---
|
||||
title: "Ollama-Compatible API"
|
||||
metatags:
|
||||
description: "SGLang provides Ollama API compatibility, allowing you to use the Ollama CLI and Python library with SGLang as the inference backend."
|
||||
---
|
||||
SGLang provides Ollama API compatibility, allowing you to use the Ollama CLI and Python library with SGLang as the inference backend.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
<CodeGroup>
|
||||
```bash Command
|
||||
# Install the Ollama Python library (for Python client usage)
|
||||
pip install ollama
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
<Note>You don't need the Ollama server installed - SGLang acts as the backend. You only need the `ollama` CLI or Python library as the client.</Note>
|
||||
|
||||
## Endpoints
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "34%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Endpoint</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Method</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`/`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GET, HEAD</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Health check for Ollama CLI</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`/api/tags`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GET</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>List available models</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`/api/chat`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>POST</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Chat completions (streaming & non-streaming)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`/api/generate`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>POST</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Text generation (streaming & non-streaming)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`/api/show`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>POST</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Model information</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Launch SGLang Server
|
||||
|
||||
<CodeGroup>
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model Qwen/Qwen2.5-1.5B-Instruct \
|
||||
--port 30001 \
|
||||
--host 0.0.0.0
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
<Note>The model name used with `ollama run` must match exactly what you passed to `--model`.</Note>
|
||||
|
||||
### 2. Use Ollama CLI
|
||||
|
||||
<CodeGroup>
|
||||
```bash Command
|
||||
# List available models
|
||||
OLLAMA_HOST=http://localhost:30001 ollama list
|
||||
|
||||
# Interactive chat
|
||||
OLLAMA_HOST=http://localhost:30001 ollama run "Qwen/Qwen2.5-1.5B-Instruct"
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
If connecting to a remote server behind a firewall:
|
||||
|
||||
<CodeGroup>
|
||||
```bash Command
|
||||
# SSH tunnel
|
||||
ssh -L 30001:localhost:30001 user@gpu-server -N &
|
||||
|
||||
# Then use Ollama CLI as above
|
||||
OLLAMA_HOST=http://localhost:30001 ollama list
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### 3. Use Ollama Python Library
|
||||
|
||||
```python Example
|
||||
import ollama
|
||||
|
||||
client = ollama.Client(host='http://localhost:30001')
|
||||
|
||||
# Non-streaming
|
||||
response = client.chat(
|
||||
model='Qwen/Qwen2.5-1.5B-Instruct',
|
||||
messages=[{'role': 'user', 'content': 'Hello!'}]
|
||||
)
|
||||
print(response['message']['content'])
|
||||
|
||||
# Streaming
|
||||
stream = client.chat(
|
||||
model='Qwen/Qwen2.5-1.5B-Instruct',
|
||||
messages=[{'role': 'user', 'content': 'Tell me a story'}],
|
||||
stream=True
|
||||
)
|
||||
for chunk in stream:
|
||||
print(chunk['message']['content'], end='', flush=True)
|
||||
```
|
||||
|
||||
## Smart Router
|
||||
|
||||
For intelligent routing between local Ollama (fast) and remote SGLang (powerful) using an LLM judge, see the [Smart Router documentation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/entrypoints/ollama/README).
|
||||
|
||||
## Summary
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "50%"}} />
|
||||
<col style={{width: "50%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Component</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Purpose</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Ollama API**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Familiar CLI/API that developers already know</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**SGLang Backend**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>High-performance inference engine</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Smart Router**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Intelligent routing - fast local for simple tasks, powerful remote for complex tasks</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
@@ -0,0 +1,7 @@
|
||||
---
|
||||
title: "OpenAI-Compatible APIs"
|
||||
description: "Documentation for OpenAI-Compatible APIs"
|
||||
---
|
||||
- [Openai Api Completions](./openai_api_completions)
|
||||
- [Openai Api Vision](./openai_api_vision)
|
||||
- [Openai Api Embeddings](./openai_api_embeddings)
|
||||
@@ -0,0 +1,9 @@
|
||||
OpenAI-Compatible APIs
|
||||
======================
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
openai_api_completions.ipynb
|
||||
openai_api_vision.ipynb
|
||||
openai_api_embeddings.ipynb
|
||||
@@ -0,0 +1,552 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# OpenAI APIs - Completions\n",
|
||||
"\n",
|
||||
"SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.\n",
|
||||
"A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/api-reference).\n",
|
||||
"\n",
|
||||
"This tutorial covers the following popular APIs:\n",
|
||||
"\n",
|
||||
"- `chat/completions`\n",
|
||||
"- `completions`\n",
|
||||
"\n",
|
||||
"Check out other tutorials to learn about [vision APIs](openai_api_vision) for vision-language models and [embedding APIs](openai_api_embeddings) for embedding models."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Launch A Server\n",
|
||||
"\n",
|
||||
"Launch the server in your terminal and wait for it to initialize."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sglang.test.doc_patch import launch_server_cmd\n",
|
||||
"from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
|
||||
"\n",
|
||||
"server_process, port = launch_server_cmd(\n",
|
||||
" \"python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --log-level warning\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"wait_for_server(f\"http://localhost:{port}\")\n",
|
||||
"print(f\"Server started on http://localhost:{port}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Chat Completions\n",
|
||||
"\n",
|
||||
"### Usage\n",
|
||||
"\n",
|
||||
"The server fully implements the OpenAI API.\n",
|
||||
"It will automatically apply the chat template specified in the Hugging Face tokenizer, if one is available.\n",
|
||||
"You can also specify a custom chat template with `--chat-template` when launching the server."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import openai\n",
|
||||
"\n",
|
||||
"client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
|
||||
"\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"qwen/qwen2.5-0.5b-instruct\",\n",
|
||||
" messages=[\n",
|
||||
" {\"role\": \"user\", \"content\": \"List 3 countries and their capitals.\"},\n",
|
||||
" ],\n",
|
||||
" temperature=0,\n",
|
||||
" max_tokens=64,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(f\"Response: {response}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Model Thinking/Reasoning Support\n",
|
||||
"\n",
|
||||
"Some models support internal reasoning or thinking processes that can be exposed in the API response. SGLang provides unified support for various reasoning models through the `chat_template_kwargs` parameter and compatible reasoning parsers.\n",
|
||||
"\n",
|
||||
"#### Supported Models and Configuration\n",
|
||||
"\n",
|
||||
"| Model Family | Chat Template Parameter | Reasoning Parser | Notes |\n",
|
||||
"|--------------|------------------------|------------------|--------|\n",
|
||||
"| DeepSeek-R1 (R1, R1-0528, R1-Distill) | `enable_thinking` | `--reasoning-parser deepseek-r1` | Standard reasoning models |\n",
|
||||
"| DeepSeek-V3.1 | `thinking` | `--reasoning-parser deepseek-v3` | Hybrid model (thinking/non-thinking modes) |\n",
|
||||
"| Qwen3 (standard) | `enable_thinking` | `--reasoning-parser qwen3` | Hybrid model (thinking/non-thinking modes) |\n",
|
||||
"| Qwen3-Thinking | N/A (always enabled) | `--reasoning-parser qwen3-thinking` | Always generates reasoning |\n",
|
||||
"| Kimi | N/A (always enabled) | `--reasoning-parser kimi` | Kimi thinking models |\n",
|
||||
"| Gpt-Oss | N/A (always enabled) | `--reasoning-parser gpt-oss` | Gpt-Oss thinking models |\n",
|
||||
"\n",
|
||||
"#### Basic Usage\n",
|
||||
"\n",
|
||||
"To enable reasoning output, you need to:\n",
|
||||
"1. Launch the server with the appropriate reasoning parser\n",
|
||||
"2. Set the model-specific parameter in `chat_template_kwargs`\n",
|
||||
"3. Optionally use `separate_reasoning: False` to not get reasoning content separately (default to `True`)\n",
|
||||
"\n",
|
||||
"**Note for Qwen3-Thinking models:** These models always generate thinking content and do not support the `enable_thinking` parameter. Use `--reasoning-parser qwen3-thinking` or `--reasoning-parser qwen3` to parse the thinking content.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Example: Qwen3 Models\n",
|
||||
"\n",
|
||||
"```python\n",
|
||||
"# Launch server:\n",
|
||||
"# python3 -m sglang.launch_server --model Qwen/Qwen3-4B --reasoning-parser qwen3\n",
|
||||
"\n",
|
||||
"from openai import OpenAI\n",
|
||||
"\n",
|
||||
"client = OpenAI(\n",
|
||||
" api_key=\"EMPTY\",\n",
|
||||
" base_url=f\"http://127.0.0.1:30000/v1\",\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"model = \"Qwen/Qwen3-4B\"\n",
|
||||
"messages = [{\"role\": \"user\", \"content\": \"How many r's are in 'strawberry'?\"}]\n",
|
||||
"\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=model,\n",
|
||||
" messages=messages,\n",
|
||||
" extra_body={\n",
|
||||
" \"chat_template_kwargs\": {\"enable_thinking\": True},\n",
|
||||
" \"separate_reasoning\": True\n",
|
||||
" }\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(\"Reasoning:\", response.choices[0].message.reasoning_content)\n",
|
||||
"print(\"-\"*100)\n",
|
||||
"print(\"Answer:\", response.choices[0].message.content)\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"**ExampleOutput:**\n",
|
||||
"```\n",
|
||||
"Reasoning: Okay, so the user is asking how many 'r's are in the word 'strawberry'. Let me think. First, I need to make sure I have the word spelled correctly. Strawberry... S-T-R-A-W-B-E-R-R-Y. Wait, is that right? Let me break it down.\n",
|
||||
"\n",
|
||||
"Starting with 'strawberry', let's write out the letters one by one. S, T, R, A, W, B, E, R, R, Y. Hmm, wait, that's 10 letters. Let me check again. S (1), T (2), R (3), A (4), W (5), B (6), E (7), R (8), R (9), Y (10). So the letters are S-T-R-A-W-B-E-R-R-Y. \n",
|
||||
"...\n",
|
||||
"Therefore, the answer should be three R's in 'strawberry'. But I need to make sure I'm not counting any other letters as R. Let me check again. S, T, R, A, W, B, E, R, R, Y. No other R's. So three in total. Yeah, that seems right.\n",
|
||||
"\n",
|
||||
"----------------------------------------------------------------------------------------------------\n",
|
||||
"Answer: The word \"strawberry\" contains **three** letters 'r'. Here's the breakdown:\n",
|
||||
"\n",
|
||||
"1. **S-T-R-A-W-B-E-R-R-Y** \n",
|
||||
" - The **third letter** is 'R'. \n",
|
||||
" - The **eighth and ninth letters** are also 'R's. \n",
|
||||
"\n",
|
||||
"Thus, the total count is **3**. \n",
|
||||
"\n",
|
||||
"**Answer:** 3.\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"**Note:** Setting `\"enable_thinking\": False` (or omitting it) will result in `reasoning_content` being `None`. Qwen3-Thinking models always generate reasoning content and don't support the `enable_thinking` parameter.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Logit Bias Support\n",
|
||||
"\n",
|
||||
"SGLang supports the `logit_bias` parameter for both chat completions and completions APIs. This parameter allows you to modify the likelihood of specific tokens being generated by adding bias values to their logits. The bias values can range from -100 to 100, where:\n",
|
||||
"\n",
|
||||
"- **Positive values** (0 to 100) increase the likelihood of the token being selected\n",
|
||||
"- **Negative values** (-100 to 0) decrease the likelihood of the token being selected\n",
|
||||
"- **-100** effectively prevents the token from being generated\n",
|
||||
"\n",
|
||||
"The `logit_bias` parameter accepts a dictionary where keys are token IDs (as strings) and values are the bias amounts (as floats).\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Getting Token IDs\n",
|
||||
"\n",
|
||||
"To use `logit_bias` effectively, you need to know the token IDs for the words you want to bias. Here's how to get token IDs:\n",
|
||||
"\n",
|
||||
"```python\n",
|
||||
"# Get tokenizer to find token IDs\n",
|
||||
"import tiktoken\n",
|
||||
"\n",
|
||||
"# For OpenAI models, use the appropriate encoding\n",
|
||||
"tokenizer = tiktoken.encoding_for_model(\"gpt-3.5-turbo\") # or your model\n",
|
||||
"\n",
|
||||
"# Get token IDs for specific words\n",
|
||||
"word = \"sunny\"\n",
|
||||
"token_ids = tokenizer.encode(word)\n",
|
||||
"print(f\"Token IDs for '{word}': {token_ids}\")\n",
|
||||
"\n",
|
||||
"# For SGLang models, you can access the tokenizer through the client\n",
|
||||
"# and get token IDs for bias\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"**Important:** The `logit_bias` parameter uses token IDs as string keys, not the actual words.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Example: DeepSeek-V3 Models\n",
|
||||
"\n",
|
||||
"DeepSeek-V3 models support thinking mode through the `thinking` parameter:\n",
|
||||
"\n",
|
||||
"```python\n",
|
||||
"# Launch server:\n",
|
||||
"# python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.1 --tp 8 --reasoning-parser deepseek-v3\n",
|
||||
"\n",
|
||||
"from openai import OpenAI\n",
|
||||
"\n",
|
||||
"client = OpenAI(\n",
|
||||
" api_key=\"EMPTY\",\n",
|
||||
" base_url=f\"http://127.0.0.1:30000/v1\",\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"model = \"deepseek-ai/DeepSeek-V3.1\"\n",
|
||||
"messages = [{\"role\": \"user\", \"content\": \"How many r's are in 'strawberry'?\"}]\n",
|
||||
"\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=model,\n",
|
||||
" messages=messages,\n",
|
||||
" extra_body={\n",
|
||||
" \"chat_template_kwargs\": {\"thinking\": True},\n",
|
||||
" \"separate_reasoning\": True\n",
|
||||
" }\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(\"Reasoning:\", response.choices[0].message.reasoning_content)\n",
|
||||
"print(\"-\"*100)\n",
|
||||
"print(\"Answer:\", response.choices[0].message.content)\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"**Example Output:**\n",
|
||||
"```\n",
|
||||
"Reasoning: First, the question is: \"How many r's are in 'strawberry'?\"\n",
|
||||
"\n",
|
||||
"I need to count the number of times the letter 'r' appears in the word \"strawberry\".\n",
|
||||
"\n",
|
||||
"Let me write out the word: S-T-R-A-W-B-E-R-R-Y.\n",
|
||||
"\n",
|
||||
"Now, I'll go through each letter and count the 'r's.\n",
|
||||
"...\n",
|
||||
"So, I have three 'r's in \"strawberry\".\n",
|
||||
"\n",
|
||||
"I should double-check. The word is spelled S-T-R-A-W-B-E-R-R-Y. The letters are at positions: 3, 8, and 9 are 'r's. Yes, that's correct.\n",
|
||||
"\n",
|
||||
"Therefore, the answer should be 3.\n",
|
||||
"----------------------------------------------------------------------------------------------------\n",
|
||||
"Answer: The word \"strawberry\" contains **3** instances of the letter \"r\". Here's a breakdown for clarity:\n",
|
||||
"\n",
|
||||
"- The word is spelled: S-T-R-A-W-B-E-R-R-Y\n",
|
||||
"- The \"r\" appears at the 3rd, 8th, and 9th positions.\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"**Note:** DeepSeek-V3 models use the `thinking` parameter (not `enable_thinking`) to control reasoning output.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Example with logit_bias parameter\n",
|
||||
"# Note: You need to get the actual token IDs from your tokenizer\n",
|
||||
"# For demonstration, we'll use some example token IDs\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"qwen/qwen2.5-0.5b-instruct\",\n",
|
||||
" messages=[\n",
|
||||
" {\"role\": \"user\", \"content\": \"Complete this sentence: The weather today is\"}\n",
|
||||
" ],\n",
|
||||
" temperature=0.7,\n",
|
||||
" max_tokens=20,\n",
|
||||
" logit_bias={\n",
|
||||
" \"12345\": 50, # Increase likelihood of token ID 12345\n",
|
||||
" \"67890\": -50, # Decrease likelihood of token ID 67890\n",
|
||||
" \"11111\": 25, # Slightly increase likelihood of token ID 11111\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(f\"Response with logit bias: {response.choices[0].message.content}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Parameters\n",
|
||||
"\n",
|
||||
"The chat completions API accepts OpenAI Chat Completions API's parameters. Refer to [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat/create) for more details.\n",
|
||||
"\n",
|
||||
"SGLang extends the standard API with the `extra_body` parameter, allowing for additional customization. One key option within `extra_body` is `chat_template_kwargs`, which can be used to pass arguments to the chat template processor."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"qwen/qwen2.5-0.5b-instruct\",\n",
|
||||
" messages=[\n",
|
||||
" {\n",
|
||||
" \"role\": \"system\",\n",
|
||||
" \"content\": \"You are a knowledgeable historian who provides concise responses.\",\n",
|
||||
" },\n",
|
||||
" {\"role\": \"user\", \"content\": \"Tell me about ancient Rome\"},\n",
|
||||
" {\n",
|
||||
" \"role\": \"assistant\",\n",
|
||||
" \"content\": \"Ancient Rome was a civilization centered in Italy.\",\n",
|
||||
" },\n",
|
||||
" {\"role\": \"user\", \"content\": \"What were their major achievements?\"},\n",
|
||||
" ],\n",
|
||||
" temperature=0.3, # Lower temperature for more focused responses\n",
|
||||
" max_tokens=128, # Reasonable length for a concise response\n",
|
||||
" top_p=0.95, # Slightly higher for better fluency\n",
|
||||
" presence_penalty=0.2, # Mild penalty to avoid repetition\n",
|
||||
" frequency_penalty=0.2, # Mild penalty for more natural language\n",
|
||||
" n=1, # Single response is usually more stable\n",
|
||||
" seed=42, # Keep for reproducibility\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(response.choices[0].message.content)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Streaming mode is also supported."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Logit Bias Support\n",
|
||||
"\n",
|
||||
"The completions API also supports the `logit_bias` parameter with the same functionality as described in the chat completions section above.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"stream = client.chat.completions.create(\n",
|
||||
" model=\"qwen/qwen2.5-0.5b-instruct\",\n",
|
||||
" messages=[{\"role\": \"user\", \"content\": \"Say this is a test\"}],\n",
|
||||
" stream=True,\n",
|
||||
")\n",
|
||||
"for chunk in stream:\n",
|
||||
" if chunk.choices[0].delta.content is not None:\n",
|
||||
" print(chunk.choices[0].delta.content, end=\"\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Returning Routed Experts (MoE Models)\n",
|
||||
"\n",
|
||||
"For MoE models, set `return_routed_experts: true` in `extra_body` to return expert routing data. Requires `--enable-return-routed-experts` server flag. The `routed_experts` field will be returned in the `sgl_ext` object on each choice, containing base64-encoded int32 expert IDs as a flattened array with logical shape `[num_tokens, num_layers, top_k]`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Example with logit_bias parameter for completions API\n",
|
||||
"# Note: You need to get the actual token IDs from your tokenizer\n",
|
||||
"# For demonstration, we'll use some example token IDs\n",
|
||||
"response = client.completions.create(\n",
|
||||
" model=\"qwen/qwen2.5-0.5b-instruct\",\n",
|
||||
" prompt=\"The best programming language for AI is\",\n",
|
||||
" temperature=0.7,\n",
|
||||
" max_tokens=20,\n",
|
||||
" logit_bias={\n",
|
||||
" \"12345\": 75, # Strongly favor token ID 12345\n",
|
||||
" \"67890\": -100, # Completely avoid token ID 67890\n",
|
||||
" \"11111\": -25, # Slightly discourage token ID 11111\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(f\"Response with logit bias: {response.choices[0].text}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Completions\n",
|
||||
"\n",
|
||||
"### Usage\n",
|
||||
"Completions API is similar to Chat Completions API, but without the `messages` parameter or chat templates."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"response = client.completions.create(\n",
|
||||
" model=\"qwen/qwen2.5-0.5b-instruct\",\n",
|
||||
" prompt=\"List 3 countries and their capitals.\",\n",
|
||||
" temperature=0,\n",
|
||||
" max_tokens=64,\n",
|
||||
" n=1,\n",
|
||||
" stop=None,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(f\"Response: {response}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Parameters\n",
|
||||
"\n",
|
||||
"The completions API accepts OpenAI Completions API's parameters. Refer to [OpenAI Completions API](https://platform.openai.com/docs/api-reference/completions/create) for more details.\n",
|
||||
"\n",
|
||||
"Here is an example of a detailed completions request:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"response = client.completions.create(\n",
|
||||
" model=\"qwen/qwen2.5-0.5b-instruct\",\n",
|
||||
" prompt=\"Write a short story about a space explorer.\",\n",
|
||||
" temperature=0.7, # Moderate temperature for creative writing\n",
|
||||
" max_tokens=150, # Longer response for a story\n",
|
||||
" top_p=0.9, # Balanced diversity in word choice\n",
|
||||
" stop=[\"\\n\\n\", \"THE END\"], # Multiple stop sequences\n",
|
||||
" presence_penalty=0.3, # Encourage novel elements\n",
|
||||
" frequency_penalty=0.3, # Reduce repetitive phrases\n",
|
||||
" n=1, # Generate one completion\n",
|
||||
" seed=123, # For reproducible results\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(f\"Response: {response}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Returning Routed Experts (MoE Models)\n",
|
||||
"\n",
|
||||
"For MoE models, set `return_routed_experts: true` in `extra_body` to return expert routing data. Requires `--enable-return-routed-experts` server flag. The `routed_experts` field will be returned in the `sgl_ext` object on each choice, containing base64-encoded int32 expert IDs as a flattened array with logical shape `[num_tokens, num_layers, top_k]`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Structured Outputs (JSON, Regex, EBNF)\n",
|
||||
"\n",
|
||||
"For OpenAI compatible structured outputs API, refer to [Structured Outputs](../advanced_features/structured_outputs) for more details.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Using LoRA Adapters\n",
|
||||
"\n",
|
||||
"SGLang supports LoRA (Low-Rank Adaptation) adapters with OpenAI-compatible APIs. You can specify which adapter to use directly in the `model` parameter using the `base-model:adapter-name` syntax.\n",
|
||||
"\n",
|
||||
"**Server Setup:**\n",
|
||||
"```bash\n",
|
||||
"python -m sglang.launch_server \\\n",
|
||||
" --model-path qwen/qwen2.5-0.5b-instruct \\\n",
|
||||
" --enable-lora \\\n",
|
||||
" --lora-paths adapter_a=/path/to/adapter_a adapter_b=/path/to/adapter_b\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"For more details on LoRA serving configuration, see the [LoRA documentation](../advanced_features/lora).\n",
|
||||
"\n",
|
||||
"**API Call:**\n",
|
||||
"\n",
|
||||
"(Recommended) Use the `model:adapter` syntax to specify which adapter to use:\n",
|
||||
"```python\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"qwen/qwen2.5-0.5b-instruct:adapter_a\", # ← base-model:adapter-name\n",
|
||||
" messages=[{\"role\": \"user\", \"content\": \"Convert to SQL: show all users\"}],\n",
|
||||
" max_tokens=50,\n",
|
||||
")\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"**Backward Compatible: Using `extra_body`**\n",
|
||||
"\n",
|
||||
"The old `extra_body` method is still supported for backward compatibility:\n",
|
||||
"```python\n",
|
||||
"# Backward compatible method\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"qwen/qwen2.5-0.5b-instruct\",\n",
|
||||
" messages=[{\"role\": \"user\", \"content\": \"Convert to SQL: show all users\"}],\n",
|
||||
" extra_body={\"lora_path\": \"adapter_a\"}, # ← old method\n",
|
||||
" max_tokens=50,\n",
|
||||
")\n",
|
||||
"```\n",
|
||||
"**Note:** When both `model:adapter` and `extra_body[\"lora_path\"]` are specified, the `model:adapter` syntax takes precedence."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(server_process)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,456 @@
|
||||
---
|
||||
title: "OpenAI APIs - Completions"
|
||||
metatags:
|
||||
description: "This tutorial covers the following popular APIs: 'chat/completions' and 'completions'"
|
||||
---
|
||||
SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.
|
||||
A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/api-reference).
|
||||
|
||||
This tutorial covers the following popular APIs:
|
||||
|
||||
- `chat/completions`
|
||||
- `completions`
|
||||
|
||||
Check out other tutorials to learn about [vision APIs](./openai_api_vision) for vision-language models and [embedding APIs](./openai_api_embeddings) for embedding models.
|
||||
|
||||
## Launch A Server
|
||||
|
||||
Launch the server in your terminal and wait for it to initialize.
|
||||
|
||||
```python Example
|
||||
from sglang.test.doc_patch import launch_server_cmd
|
||||
from sglang.utils import wait_for_server, print_highlight, terminate_process
|
||||
|
||||
server_process, port = launch_server_cmd(
|
||||
"python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --log-level warning"
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
print(f"Server started on http://localhost:{port}")
|
||||
```
|
||||
|
||||
## Chat Completions
|
||||
|
||||
### Usage
|
||||
|
||||
The server fully implements the OpenAI API.
|
||||
It will automatically apply the chat template specified in the Hugging Face tokenizer, if one is available.
|
||||
You can also specify a custom chat template with `--chat-template` when launching the server.
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
|
||||
client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="qwen/qwen2.5-0.5b-instruct",
|
||||
messages=[
|
||||
{"role": "user", "content": "List 3 countries and their capitals."},
|
||||
],
|
||||
temperature=0,
|
||||
max_tokens=64,
|
||||
)
|
||||
|
||||
print_highlight(f"Response: {response}")
|
||||
```
|
||||
|
||||
### Model Thinking/Reasoning Support
|
||||
|
||||
Some models support internal reasoning or thinking processes that can be exposed in the API response. SGLang provides unified support for various reasoning models through the `chat_template_kwargs` parameter and compatible reasoning parsers.
|
||||
|
||||
#### Supported Models and Configuration
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model Family</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Chat Template Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Reasoning Parser</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Notes</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-R1 (R1, R1-0528, R1-Distill)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`enable_thinking`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser deepseek-r1`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Standard reasoning models</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-V3.1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`thinking`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser deepseek-v3`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Hybrid model (thinking/non-thinking modes)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3 (standard)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`enable_thinking`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser qwen3`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Hybrid model (thinking/non-thinking modes)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3-Thinking</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A (always enabled)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser qwen3-thinking`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Always generates reasoning</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Kimi</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A (always enabled)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser kimi`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Kimi thinking models</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Gpt-Oss</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A (always enabled)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser gpt-oss`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Gpt-Oss thinking models</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### Basic Usage
|
||||
|
||||
To enable reasoning output, you need to:
|
||||
1. Launch the server with the appropriate reasoning parser
|
||||
2. Set the model-specific parameter in `chat_template_kwargs`
|
||||
3. Optionally use `separate_reasoning: False` to not get reasoning content separately (default to `True`)
|
||||
|
||||
<Note>
|
||||
**Note for Qwen3-Thinking models:** These models always generate thinking content and do not support the `enable_thinking` parameter. Use `--reasoning-parser qwen3-thinking` or `--reasoning-parser qwen3` to parse the thinking content.
|
||||
</Note>
|
||||
|
||||
#### Example: Qwen3 Models
|
||||
|
||||
```python Example
|
||||
# Launch server:
|
||||
# python3 -m sglang.launch_server --model Qwen/Qwen3-4B --reasoning-parser qwen3
|
||||
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url=f"http://127.0.0.1:30000/v1",
|
||||
)
|
||||
|
||||
model = "Qwen/Qwen3-4B"
|
||||
messages = [{"role": "user", "content": "How many r's are in 'strawberry'?"}]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model=model,
|
||||
messages=messages,
|
||||
extra_body={
|
||||
"chat_template_kwargs": {"enable_thinking": True},
|
||||
"separate_reasoning": True
|
||||
}
|
||||
)
|
||||
|
||||
print("Reasoning:", response.choices[0].message.reasoning_content)
|
||||
print("-"*100)
|
||||
print("Answer:", response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**ExampleOutput:**
|
||||
```text Output
|
||||
Reasoning: Okay, so the user is asking how many 'r's are in the word 'strawberry'. Let me think. First, I need to make sure I have the word spelled correctly. Strawberry... S-T-R-A-W-B-E-R-R-Y. Wait, is that right? Let me break it down.
|
||||
|
||||
Starting with 'strawberry', let's write out the letters one by one. S, T, R, A, W, B, E, R, R, Y. Hmm, wait, that's 10 letters. Let me check again. S (1), T (2), R (3), A (4), W (5), B (6), E (7), R (8), R (9), Y (10). So the letters are S-T-R-A-W-B-E-R-R-Y.
|
||||
...
|
||||
Therefore, the answer should be three R's in 'strawberry'. But I need to make sure I'm not counting any other letters as R. Let me check again. S, T, R, A, W, B, E, R, R, Y. No other R's. So three in total. Yeah, that seems right.
|
||||
|
||||
----------------------------------------------------------------------------------------------------
|
||||
Answer: The word "strawberry" contains **three** letters 'r'. Here's the breakdown:
|
||||
|
||||
1. **S-T-R-A-W-B-E-R-R-Y**
|
||||
- The **third letter** is 'R'.
|
||||
- The **eighth and ninth letters** are also 'R's.
|
||||
|
||||
Thus, the total count is **3**.
|
||||
|
||||
**Answer:** 3.
|
||||
```
|
||||
<Note>
|
||||
Setting `"enable_thinking": False` (or omitting it) will result in `reasoning_content` being `None`. Qwen3-Thinking models always generate reasoning content and don't support the `enable_thinking` parameter.
|
||||
</Note>
|
||||
|
||||
#### Logit Bias Support
|
||||
|
||||
SGLang supports the `logit_bias` parameter for both chat completions and completions APIs. This parameter allows you to modify the likelihood of specific tokens being generated by adding bias values to their logits. The bias values can range from -100 to 100, where:
|
||||
|
||||
- **Positive values** (0 to 100) increase the likelihood of the token being selected
|
||||
- **Negative values** (-100 to 0) decrease the likelihood of the token being selected
|
||||
- **-100** effectively prevents the token from being generated
|
||||
|
||||
The `logit_bias` parameter accepts a dictionary where keys are token IDs (as strings) and values are the bias amounts (as floats).
|
||||
|
||||
#### Getting Token IDs
|
||||
|
||||
To use `logit_bias` effectively, you need to know the token IDs for the words you want to bias. Here's how to get token IDs:
|
||||
|
||||
```python Example
|
||||
# Get tokenizer to find token IDs
|
||||
import tiktoken
|
||||
|
||||
# For OpenAI models, use the appropriate encoding
|
||||
tokenizer = tiktoken.encoding_for_model("gpt-3.5-turbo") # or your model
|
||||
|
||||
# Get token IDs for specific words
|
||||
word = "sunny"
|
||||
token_ids = tokenizer.encode(word)
|
||||
print(f"Token IDs for '{word}': {token_ids}")
|
||||
|
||||
# For SGLang models, you can access the tokenizer through the client
|
||||
# and get token IDs for bias
|
||||
```
|
||||
<Tip>
|
||||
**Important:** The `logit_bias` parameter uses token IDs as string keys, not the actual words.
|
||||
</Tip>
|
||||
|
||||
#### Example: DeepSeek-V3 Models
|
||||
|
||||
DeepSeek-V3 models support thinking mode through the `thinking` parameter:
|
||||
|
||||
```python Example
|
||||
# Launch server:
|
||||
# python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.1 --tp 8 --reasoning-parser deepseek-v3
|
||||
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url=f"http://127.0.0.1:30000/v1",
|
||||
)
|
||||
|
||||
model = "deepseek-ai/DeepSeek-V3.1"
|
||||
messages = [{"role": "user", "content": "How many r's are in 'strawberry'?"}]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model=model,
|
||||
messages=messages,
|
||||
extra_body={
|
||||
"chat_template_kwargs": {"thinking": True},
|
||||
"separate_reasoning": True
|
||||
}
|
||||
)
|
||||
|
||||
print("Reasoning:", response.choices[0].message.reasoning_content)
|
||||
print("-"*100)
|
||||
print("Answer:", response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Example Output:**
|
||||
```text Output
|
||||
Reasoning: First, the question is: "How many r's are in 'strawberry'?"
|
||||
|
||||
I need to count the number of times the letter 'r' appears in the word "strawberry".
|
||||
|
||||
Let me write out the word: S-T-R-A-W-B-E-R-R-Y.
|
||||
|
||||
Now, I'll go through each letter and count the 'r's.
|
||||
...
|
||||
So, I have three 'r's in "strawberry".
|
||||
|
||||
I should double-check. The word is spelled S-T-R-A-W-B-E-R-R-Y. The letters are at positions: 3, 8, and 9 are 'r's. Yes, that's correct.
|
||||
|
||||
Therefore, the answer should be 3.
|
||||
----------------------------------------------------------------------------------------------------
|
||||
Answer: The word "strawberry" contains **3** instances of the letter "r". Here's a breakdown for clarity:
|
||||
|
||||
- The word is spelled: S-T-R-A-W-B-E-R-R-Y
|
||||
- The "r" appears at the 3rd, 8th, and 9th positions.
|
||||
```
|
||||
<Note>
|
||||
DeepSeek-V3 models use the `thinking` parameter (not `enable_thinking`) to control reasoning output.
|
||||
</Note>
|
||||
|
||||
```python Example
|
||||
# Example with logit_bias parameter
|
||||
# Note: You need to get the actual token IDs from your tokenizer
|
||||
# For demonstration, we'll use some example token IDs
|
||||
response = client.chat.completions.create(
|
||||
model="qwen/qwen2.5-0.5b-instruct",
|
||||
messages=[
|
||||
{"role": "user", "content": "Complete this sentence: The weather today is"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=20,
|
||||
logit_bias={
|
||||
"12345": 50, # Increase likelihood of token ID 12345
|
||||
"67890": -50, # Decrease likelihood of token ID 67890
|
||||
"11111": 25, # Slightly increase likelihood of token ID 11111
|
||||
},
|
||||
)
|
||||
|
||||
print_highlight(f"Response with logit bias: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
### Parameters
|
||||
|
||||
The chat completions API accepts OpenAI Chat Completions API's parameters. Refer to [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat/create) for more details.
|
||||
|
||||
SGLang extends the standard API with the `extra_body` parameter, allowing for additional customization. One key option within `extra_body` is `chat_template_kwargs`, which can be used to pass arguments to the chat template processor.
|
||||
|
||||
```python Example
|
||||
response = client.chat.completions.create(
|
||||
model="qwen/qwen2.5-0.5b-instruct",
|
||||
messages=[
|
||||
{
|
||||
"role": "system",
|
||||
"content": "You are a knowledgeable historian who provides concise responses.",
|
||||
},
|
||||
{"role": "user", "content": "Tell me about ancient Rome"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Ancient Rome was a civilization centered in Italy.",
|
||||
},
|
||||
{"role": "user", "content": "What were their major achievements?"},
|
||||
],
|
||||
temperature=0.3, # Lower temperature for more focused responses
|
||||
max_tokens=128, # Reasonable length for a concise response
|
||||
top_p=0.95, # Slightly higher for better fluency
|
||||
presence_penalty=0.2, # Mild penalty to avoid repetition
|
||||
frequency_penalty=0.2, # Mild penalty for more natural language
|
||||
n=1, # Single response is usually more stable
|
||||
seed=42, # Keep for reproducibility
|
||||
)
|
||||
|
||||
print_highlight(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
Streaming mode is also supported.
|
||||
|
||||
#### Logit Bias Support
|
||||
|
||||
The completions API also supports the `logit_bias` parameter with the same functionality as described in the chat completions section above.
|
||||
|
||||
```python Example
|
||||
stream = client.chat.completions.create(
|
||||
model="qwen/qwen2.5-0.5b-instruct",
|
||||
messages=[{"role": "user", "content": "Say this is a test"}],
|
||||
stream=True,
|
||||
)
|
||||
for chunk in stream:
|
||||
if chunk.choices[0].delta.content is not None:
|
||||
print(chunk.choices[0].delta.content, end="")
|
||||
```
|
||||
|
||||
#### Returning Routed Experts (MoE Models)
|
||||
|
||||
For MoE models, set `return_routed_experts: true` in `extra_body` to return expert routing data. Requires `--enable-return-routed-experts` server flag. The `routed_experts` field will be returned in the `sgl_ext` object on each choice, containing base64-encoded int32 expert IDs as a flattened array with logical shape `[num_tokens, num_layers, top_k]`.
|
||||
|
||||
```python Example
|
||||
# Example with logit_bias parameter for completions API
|
||||
# Note: You need to get the actual token IDs from your tokenizer
|
||||
# For demonstration, we'll use some example token IDs
|
||||
response = client.completions.create(
|
||||
model="qwen/qwen2.5-0.5b-instruct",
|
||||
prompt="The best programming language for AI is",
|
||||
temperature=0.7,
|
||||
max_tokens=20,
|
||||
logit_bias={
|
||||
"12345": 75, # Strongly favor token ID 12345
|
||||
"67890": -100, # Completely avoid token ID 67890
|
||||
"11111": -25, # Slightly discourage token ID 11111
|
||||
},
|
||||
)
|
||||
|
||||
print_highlight(f"Response with logit bias: {response.choices[0].text}")
|
||||
```
|
||||
|
||||
## Completions
|
||||
|
||||
### Usage
|
||||
Completions API is similar to Chat Completions API, but without the `messages` parameter or chat templates.
|
||||
|
||||
```python Example
|
||||
response = client.completions.create(
|
||||
model="qwen/qwen2.5-0.5b-instruct",
|
||||
prompt="List 3 countries and their capitals.",
|
||||
temperature=0,
|
||||
max_tokens=64,
|
||||
n=1,
|
||||
stop=None,
|
||||
)
|
||||
|
||||
print_highlight(f"Response: {response}")
|
||||
```
|
||||
|
||||
### Parameters
|
||||
|
||||
The completions API accepts OpenAI Completions API's parameters. Refer to [OpenAI Completions API](https://platform.openai.com/docs/api-reference/completions/create) for more details.
|
||||
|
||||
Here is an example of a detailed completions request:
|
||||
|
||||
```python Example
|
||||
response = client.completions.create(
|
||||
model="qwen/qwen2.5-0.5b-instruct",
|
||||
prompt="Write a short story about a space explorer.",
|
||||
temperature=0.7, # Moderate temperature for creative writing
|
||||
max_tokens=150, # Longer response for a story
|
||||
top_p=0.9, # Balanced diversity in word choice
|
||||
stop=["\n\n", "THE END"], # Multiple stop sequences
|
||||
presence_penalty=0.3, # Encourage novel elements
|
||||
frequency_penalty=0.3, # Reduce repetitive phrases
|
||||
n=1, # Generate one completion
|
||||
seed=123, # For reproducible results
|
||||
)
|
||||
|
||||
print_highlight(f"Response: {response}")
|
||||
```
|
||||
|
||||
#### Returning Routed Experts (MoE Models)
|
||||
|
||||
For MoE models, set `return_routed_experts: true` in `extra_body` to return expert routing data. Requires `--enable-return-routed-experts` server flag. The `routed_experts` field will be returned in the `sgl_ext` object on each choice, containing base64-encoded int32 expert IDs as a flattened array with logical shape `[num_tokens, num_layers, top_k]`.
|
||||
|
||||
## Structured Outputs (JSON, Regex, EBNF)
|
||||
|
||||
For OpenAI compatible structured outputs API, refer to [Structured Outputs](../advanced_features/structured_outputs) for more details.
|
||||
|
||||
## Using LoRA Adapters
|
||||
|
||||
SGLang supports LoRA (Low-Rank Adaptation) adapters with OpenAI-compatible APIs. You can specify which adapter to use directly in the `model` parameter using the `base-model:adapter-name` syntax.
|
||||
|
||||
**Server Setup:**
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path qwen/qwen2.5-0.5b-instruct \
|
||||
--enable-lora \
|
||||
--lora-paths adapter_a=/path/to/adapter_a adapter_b=/path/to/adapter_b
|
||||
```
|
||||
|
||||
For more details on LoRA serving configuration, see the [LoRA documentation](../advanced_features/lora).
|
||||
|
||||
**API Call:**
|
||||
|
||||
(Recommended) Use the `model:adapter` syntax to specify which adapter to use:
|
||||
```python Example
|
||||
response = client.chat.completions.create(
|
||||
model="qwen/qwen2.5-0.5b-instruct:adapter_a", # ← base-model:adapter-name
|
||||
messages=[{"role": "user", "content": "Convert to SQL: show all users"}],
|
||||
max_tokens=50,
|
||||
)
|
||||
```
|
||||
|
||||
**Backward Compatible: Using `extra_body`**
|
||||
|
||||
The old `extra_body` method is still supported for backward compatibility:
|
||||
```python Example
|
||||
# Backward compatible method
|
||||
response = client.chat.completions.create(
|
||||
model="qwen/qwen2.5-0.5b-instruct",
|
||||
messages=[{"role": "user", "content": "Convert to SQL: show all users"}],
|
||||
extra_body={"lora_path": "adapter_a"}, # ← old method
|
||||
max_tokens=50,
|
||||
)
|
||||
```
|
||||
**Note:** When both `model:adapter` and `extra_body["lora_path"]` are specified, the `model:adapter` syntax takes precedence.
|
||||
|
||||
```python Example
|
||||
terminate_process(server_process)
|
||||
```
|
||||
@@ -0,0 +1,193 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# OpenAI APIs - Embedding\n",
|
||||
"\n",
|
||||
"SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.\n",
|
||||
"A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/guides/embeddings).\n",
|
||||
"\n",
|
||||
"This tutorial covers the embedding APIs for embedding models. For a list of the supported models see the [corresponding overview page](../supported_models/embedding_models)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Launch A Server\n",
|
||||
"\n",
|
||||
"Launch the server in your terminal and wait for it to initialize. Remember to add `--is-embedding` to the command."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sglang.test.doc_patch import launch_server_cmd\n",
|
||||
"from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
|
||||
"\n",
|
||||
"embedding_process, port = launch_server_cmd(\"\"\"\n",
|
||||
"python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \\\n",
|
||||
" --host 0.0.0.0 --is-embedding --log-level warning\n",
|
||||
"\"\"\")\n",
|
||||
"\n",
|
||||
"wait_for_server(f\"http://localhost:{port}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Using cURL"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import subprocess, json\n",
|
||||
"\n",
|
||||
"text = \"Once upon a time\"\n",
|
||||
"\n",
|
||||
"curl_text = f\"\"\"curl -s http://localhost:{port}/v1/embeddings \\\n",
|
||||
" -H \"Content-Type: application/json\" \\\n",
|
||||
" -d '{{\"model\": \"Alibaba-NLP/gte-Qwen2-1.5B-instruct\", \"input\": \"{text}\"}}'\"\"\"\n",
|
||||
"\n",
|
||||
"result = subprocess.check_output(curl_text, shell=True)\n",
|
||||
"\n",
|
||||
"print(result)\n",
|
||||
"\n",
|
||||
"text_embedding = json.loads(result)[\"data\"][0][\"embedding\"]\n",
|
||||
"\n",
|
||||
"print_highlight(f\"Text embedding (first 10): {text_embedding[:10]}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Using Python Requests"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import requests\n",
|
||||
"\n",
|
||||
"text = \"Once upon a time\"\n",
|
||||
"\n",
|
||||
"response = requests.post(\n",
|
||||
" f\"http://localhost:{port}/v1/embeddings\",\n",
|
||||
" json={\"model\": \"Alibaba-NLP/gte-Qwen2-1.5B-instruct\", \"input\": text},\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"text_embedding = response.json()[\"data\"][0][\"embedding\"]\n",
|
||||
"\n",
|
||||
"print_highlight(f\"Text embedding (first 10): {text_embedding[:10]}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Using OpenAI Python Client"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import openai\n",
|
||||
"\n",
|
||||
"client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
|
||||
"\n",
|
||||
"# Text embedding example\n",
|
||||
"response = client.embeddings.create(\n",
|
||||
" model=\"Alibaba-NLP/gte-Qwen2-1.5B-instruct\",\n",
|
||||
" input=text,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"embedding = response.data[0].embedding[:10]\n",
|
||||
"print_highlight(f\"Text embedding (first 10): {embedding}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Using Input IDs\n",
|
||||
"\n",
|
||||
"SGLang also supports `input_ids` as input to get the embedding."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"import os\n",
|
||||
"from transformers import AutoTokenizer\n",
|
||||
"\n",
|
||||
"os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n",
|
||||
"\n",
|
||||
"tokenizer = AutoTokenizer.from_pretrained(\"Alibaba-NLP/gte-Qwen2-1.5B-instruct\")\n",
|
||||
"input_ids = tokenizer.encode(text)\n",
|
||||
"\n",
|
||||
"curl_ids = f\"\"\"curl -s http://localhost:{port}/v1/embeddings \\\n",
|
||||
" -H \"Content-Type: application/json\" \\\n",
|
||||
" -d '{{\"model\": \"Alibaba-NLP/gte-Qwen2-1.5B-instruct\", \"input\": {json.dumps(input_ids)}}}'\"\"\"\n",
|
||||
"\n",
|
||||
"input_ids_embedding = json.loads(subprocess.check_output(curl_ids, shell=True))[\"data\"][\n",
|
||||
" 0\n",
|
||||
"][\"embedding\"]\n",
|
||||
"\n",
|
||||
"print_highlight(f\"Input IDs embedding (first 10): {input_ids_embedding[:10]}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(embedding_process)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Multi-Modal Embedding Model\n",
|
||||
"Please refer to [Multi-Modal Embedding Model](../supported_models/embedding_models)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,126 @@
|
||||
---
|
||||
title: "OpenAI APIs - Embedding"
|
||||
metatags:
|
||||
description: "This tutorial covers the embedding APIs for embedding models."
|
||||
---
|
||||
SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.
|
||||
A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/guides/embeddings).
|
||||
|
||||
This tutorial covers the embedding APIs for embedding models. For a list of the supported models see the [corresponding overview page](../supported-models)
|
||||
|
||||
|
||||
|
||||
## Launch A Server
|
||||
|
||||
Launch the server in your terminal and wait for it to initialize. Remember to add `--is-embedding` to the command.
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
from sglang.test.doc_patch import launch_server_cmd
|
||||
from sglang.utils import wait_for_server, print_highlight, terminate_process
|
||||
|
||||
embedding_process, port = launch_server_cmd(
|
||||
"""
|
||||
python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \
|
||||
--host 0.0.0.0 --is-embedding --log-level warning
|
||||
"""
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
```
|
||||
|
||||
## Using cURL
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import subprocess, json
|
||||
|
||||
text = "Once upon a time"
|
||||
|
||||
curl_text = f"""curl -s http://localhost:{port}/v1/embeddings \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{{"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": "{text}"}}'"""
|
||||
|
||||
result = subprocess.check_output(curl_text, shell=True)
|
||||
|
||||
print(result)
|
||||
|
||||
text_embedding = json.loads(result)["data"][0]["embedding"]
|
||||
|
||||
print_highlight(f"Text embedding (first 10): {text_embedding[:10]}")
|
||||
```
|
||||
|
||||
## Using Python Requests
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
text = "Once upon a time"
|
||||
|
||||
response = requests.post(
|
||||
f"http://localhost:{port}/v1/embeddings",
|
||||
json={"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": text},
|
||||
)
|
||||
|
||||
text_embedding = response.json()["data"][0]["embedding"]
|
||||
|
||||
print_highlight(f"Text embedding (first 10): {text_embedding[:10]}")
|
||||
```
|
||||
|
||||
## Using OpenAI Python Client
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
|
||||
client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")
|
||||
|
||||
# Text embedding example
|
||||
response = client.embeddings.create(
|
||||
model="Alibaba-NLP/gte-Qwen2-1.5B-instruct",
|
||||
input=text,
|
||||
)
|
||||
|
||||
embedding = response.data[0].embedding[:10]
|
||||
print_highlight(f"Text embedding (first 10): {embedding}")
|
||||
```
|
||||
|
||||
## Using Input IDs
|
||||
|
||||
SGLang also supports `input_ids` as input to get the embedding.
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import json
|
||||
import os
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
os.environ["TOKENIZERS_PARALLELISM"] = "false"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("Alibaba-NLP/gte-Qwen2-1.5B-instruct")
|
||||
input_ids = tokenizer.encode(text)
|
||||
|
||||
curl_ids = f"""curl -s http://localhost:{port}/v1/embeddings \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{{"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": {json.dumps(input_ids)}}}'"""
|
||||
|
||||
input_ids_embedding = json.loads(subprocess.check_output(curl_ids, shell=True))["data"][
|
||||
0
|
||||
]["embedding"]
|
||||
|
||||
print_highlight(f"Input IDs embedding (first 10): {input_ids_embedding[:10]}")
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
terminate_process(embedding_process)
|
||||
```
|
||||
|
||||
## Multi-Modal Embedding Model
|
||||
Please refer to [Multi-Modal Embedding Model](../supported-models)
|
||||
@@ -0,0 +1,252 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# OpenAI APIs - Vision\n",
|
||||
"\n",
|
||||
"SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.\n",
|
||||
"A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/guides/vision).\n",
|
||||
"This tutorial covers the vision APIs for vision language models.\n",
|
||||
"\n",
|
||||
"SGLang supports various vision language models such as Llama 3.2, LLaVA-OneVision, Qwen2.5-VL, Gemma3 and [more](../supported_models/multimodal_language_models).\n",
|
||||
"\n",
|
||||
"As an alternative to the OpenAI API, you can also use the [SGLang offline engine](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/offline_batch_inference_vlm.py)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Launch A Server\n",
|
||||
"\n",
|
||||
"Launch the server in your terminal and wait for it to initialize."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sglang.test.doc_patch import launch_server_cmd\n",
|
||||
"from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
|
||||
"\n",
|
||||
"vision_process, port = launch_server_cmd(\"\"\"\n",
|
||||
"python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-7B-Instruct --log-level warning\n",
|
||||
"\"\"\")\n",
|
||||
"\n",
|
||||
"wait_for_server(f\"http://localhost:{port}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Using cURL\n",
|
||||
"\n",
|
||||
"Once the server is up, you can send test requests using curl or requests."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import subprocess\n",
|
||||
"\n",
|
||||
"curl_command = f\"\"\"\n",
|
||||
"curl -s http://localhost:{port}/v1/chat/completions \\\\\n",
|
||||
" -H \"Content-Type: application/json\" \\\\\n",
|
||||
" -d '{{\n",
|
||||
" \"model\": \"Qwen/Qwen2.5-VL-7B-Instruct\",\n",
|
||||
" \"messages\": [\n",
|
||||
" {{\n",
|
||||
" \"role\": \"user\",\n",
|
||||
" \"content\": [\n",
|
||||
" {{\n",
|
||||
" \"type\": \"text\",\n",
|
||||
" \"text\": \"What’s in this image?\"\n",
|
||||
" }},\n",
|
||||
" {{\n",
|
||||
" \"type\": \"image_url\",\n",
|
||||
" \"image_url\": {{\n",
|
||||
" \"url\": \"https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true\"\n",
|
||||
" }}\n",
|
||||
" }}\n",
|
||||
" ]\n",
|
||||
" }}\n",
|
||||
" ],\n",
|
||||
" \"max_tokens\": 300\n",
|
||||
" }}'\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"response = subprocess.check_output(curl_command, shell=True).decode()\n",
|
||||
"print_highlight(response)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"response = subprocess.check_output(curl_command, shell=True).decode()\n",
|
||||
"print_highlight(response)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Using Python Requests"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import requests\n",
|
||||
"\n",
|
||||
"url = f\"http://localhost:{port}/v1/chat/completions\"\n",
|
||||
"\n",
|
||||
"data = {\n",
|
||||
" \"model\": \"Qwen/Qwen2.5-VL-7B-Instruct\",\n",
|
||||
" \"messages\": [\n",
|
||||
" {\n",
|
||||
" \"role\": \"user\",\n",
|
||||
" \"content\": [\n",
|
||||
" {\"type\": \"text\", \"text\": \"What’s in this image?\"},\n",
|
||||
" {\n",
|
||||
" \"type\": \"image_url\",\n",
|
||||
" \"image_url\": {\n",
|
||||
" \"url\": \"https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true\"\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
" ],\n",
|
||||
" }\n",
|
||||
" ],\n",
|
||||
" \"max_tokens\": 300,\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"response = requests.post(url, json=data)\n",
|
||||
"print_highlight(response.text)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Using OpenAI Python Client"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from openai import OpenAI\n",
|
||||
"\n",
|
||||
"client = OpenAI(base_url=f\"http://localhost:{port}/v1\", api_key=\"None\")\n",
|
||||
"\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"Qwen/Qwen2.5-VL-7B-Instruct\",\n",
|
||||
" messages=[\n",
|
||||
" {\n",
|
||||
" \"role\": \"user\",\n",
|
||||
" \"content\": [\n",
|
||||
" {\n",
|
||||
" \"type\": \"text\",\n",
|
||||
" \"text\": \"What is in this image?\",\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"type\": \"image_url\",\n",
|
||||
" \"image_url\": {\n",
|
||||
" \"url\": \"https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true\"\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
" ],\n",
|
||||
" }\n",
|
||||
" ],\n",
|
||||
" max_tokens=300,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(response.choices[0].message.content)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Multiple-Image Inputs\n",
|
||||
"\n",
|
||||
"The server also supports multiple images and interleaved text and images if the model supports it."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from openai import OpenAI\n",
|
||||
"\n",
|
||||
"client = OpenAI(base_url=f\"http://localhost:{port}/v1\", api_key=\"None\")\n",
|
||||
"\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"Qwen/Qwen2.5-VL-7B-Instruct\",\n",
|
||||
" messages=[\n",
|
||||
" {\n",
|
||||
" \"role\": \"user\",\n",
|
||||
" \"content\": [\n",
|
||||
" {\n",
|
||||
" \"type\": \"image_url\",\n",
|
||||
" \"image_url\": {\n",
|
||||
" \"url\": \"https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true\",\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"type\": \"image_url\",\n",
|
||||
" \"image_url\": {\n",
|
||||
" \"url\": \"https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png\",\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
" {\n",
|
||||
" \"type\": \"text\",\n",
|
||||
" \"text\": \"I have two very different images. They are not related at all. \"\n",
|
||||
" \"Please describe the first image in one sentence, and then describe the second image in another sentence.\",\n",
|
||||
" },\n",
|
||||
" ],\n",
|
||||
" }\n",
|
||||
" ],\n",
|
||||
" temperature=0,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(response.choices[0].message.content)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(vision_process)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,189 @@
|
||||
---
|
||||
title: "OpenAI APIs - Vision"
|
||||
metatags:
|
||||
description: "This tutorial covers the vision APIs for vision language models."
|
||||
---
|
||||
SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.
|
||||
A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/guides/vision).
|
||||
This tutorial covers the vision APIs for vision language models.
|
||||
|
||||
SGLang supports various vision language models such as Llama 3.2, LLaVA-OneVision, Qwen2.5-VL, Gemma3 and [more](../supported-models).
|
||||
|
||||
As an alternative to the OpenAI API, you can also use the [SGLang offline engine](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/offline_batch_inference_vlm.py).
|
||||
|
||||
|
||||
## Launch A Server
|
||||
|
||||
Launch the server in your terminal and wait for it to initialize.
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
from sglang.test.doc_patch import launch_server_cmd
|
||||
from sglang.utils import wait_for_server, print_highlight, terminate_process
|
||||
|
||||
vision_process, port = launch_server_cmd(
|
||||
"""
|
||||
python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-7B-Instruct --log-level warning
|
||||
"""
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
```
|
||||
|
||||
## Using cURL
|
||||
|
||||
Once the server is up, you can send test requests using curl or requests.
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import subprocess
|
||||
|
||||
curl_command = f"""
|
||||
curl -s http://localhost:{port}/v1/chat/completions \\
|
||||
-H "Content-Type: application/json" \\
|
||||
-d '{{
|
||||
"model": "Qwen/Qwen2.5-VL-7B-Instruct",
|
||||
"messages": [
|
||||
{{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{{
|
||||
"type": "text",
|
||||
"text": "What’s in this image?"
|
||||
}},
|
||||
{{
|
||||
"type": "image_url",
|
||||
"image_url": {{
|
||||
"url": "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true"
|
||||
}}
|
||||
}}
|
||||
]
|
||||
}}
|
||||
],
|
||||
"max_tokens": 300
|
||||
}}'
|
||||
"""
|
||||
|
||||
response = subprocess.check_output(curl_command, shell=True).decode()
|
||||
print_highlight(response)
|
||||
|
||||
|
||||
response = subprocess.check_output(curl_command, shell=True).decode()
|
||||
print_highlight(response)
|
||||
```
|
||||
|
||||
## Using Python Requests
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
url = f"http://localhost:{port}/v1/chat/completions"
|
||||
|
||||
data = {
|
||||
"model": "Qwen/Qwen2.5-VL-7B-Instruct",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "What’s in this image?"},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true"
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
"max_tokens": 300,
|
||||
}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
print_highlight(response.text)
|
||||
```
|
||||
|
||||
## Using OpenAI Python Client
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url=f"http://localhost:{port}/v1", api_key="None")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen2.5-VL-7B-Instruct",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": "What is in this image?",
|
||||
},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true"
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
max_tokens=300,
|
||||
)
|
||||
|
||||
print_highlight(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
## Multiple-Image Inputs
|
||||
|
||||
The server also supports multiple images and interleaved text and images if the model supports it.
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url=f"http://localhost:{port}/v1", api_key="None")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen2.5-VL-7B-Instruct",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true",
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png",
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "I have two very different images. They are not related at all. "
|
||||
"Please describe the first image in one sentence, and then describe the second image in another sentence.",
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
temperature=0,
|
||||
)
|
||||
|
||||
print_highlight(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
|
||||
```python Example
|
||||
terminate_process(vision_process)
|
||||
```
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
title: Basic Usage
|
||||
description: Core APIs and common usage patterns for SGLang.
|
||||
---
|
||||
|
||||
- [OpenAI-Compatible APIs](./openai_api_completions) — Chat completions, vision, and embeddings
|
||||
- [Ollama API](./ollama_api)
|
||||
- [Offline Engine API](./offline_engine_api)
|
||||
- [Native API](./native_api)
|
||||
- [Sampling Parameters](./sampling_params)
|
||||
- [Popular Model Usage](./popular_model_usage) — DeepSeek, GLM, Qwen, Llama, and more
|
||||
@@ -0,0 +1,14 @@
|
||||
---
|
||||
title: "Popular Model Usage (DeepSeek, GPT-OSS, GLM, Llama, MiniMax, Qwen, and more)"
|
||||
description: "Documentation for Popular Model Usage (DeepSeek, GPT-OSS, GLM, Llama, MiniMax, Qwen, and more)"
|
||||
---
|
||||
- [Deepseek V3](./deepseek_v3)
|
||||
- [Deepseek V32](./deepseek_v32)
|
||||
- [Glm45](./glm45)
|
||||
- [Glmv](./glmv)
|
||||
- [Gpt Oss](./gpt_oss)
|
||||
- [Kimi K2 5](./kimi_k2_5)
|
||||
- [Minimax M2](./minimax_m2)
|
||||
- [Qwen3](./qwen3)
|
||||
- [Qwen3 Vl](./qwen3_vl)
|
||||
- [Llama4](./llama4)
|
||||
@@ -0,0 +1,16 @@
|
||||
Popular Model Usage (DeepSeek, GPT-OSS, GLM, Llama, MiniMax, Qwen, and more)
|
||||
===============================================================
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
deepseek_v3.md
|
||||
deepseek_v32.md
|
||||
glm45.md
|
||||
glmv.md
|
||||
gpt_oss.md
|
||||
kimi_k2_5.md
|
||||
minimax_m2.md
|
||||
qwen3.md
|
||||
qwen3_vl.md
|
||||
llama4.md
|
||||
@@ -0,0 +1,42 @@
|
||||
---
|
||||
title: "Qwen3-Next Usage"
|
||||
metatags:
|
||||
description: "Deploy Qwen3-Next with SGLang: 80B hybrid Mamba model, MambaRadixCache prefix caching, EAGLE speculative decoding. Supports H100/H200 GPUs."
|
||||
---
|
||||
SGLang has supported Qwen3-Next-80B-A3B-Instruct and Qwen3-Next-80B-A3B-Thinking since [this PR](https://github.com/sgl-project/sglang/pull/10233).
|
||||
|
||||
## Launch Qwen3-Next with SGLang
|
||||
|
||||
To serve Qwen3-Next models on 4xH100/H200 GPUs:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server --model Qwen/Qwen3-Next-80B-A3B-Instruct --tp 4
|
||||
```
|
||||
|
||||
### Configuration Tips
|
||||
- `--max-mamba-cache-size`: Adjust `--max-mamba-cache-size` to increase mamba cache space and max running requests capability. It will decrease KV cache space as a trade-off. You can adjust it according to workload.
|
||||
- `--mamba-ssm-dtype`: `bfloat16` or `float32`, use `bfloat16` to save mamba cache size and `float32` to get more accurate results. The default setting is `float32`.
|
||||
- `--mamba-full-memory-ratio`: The ratio of mamba state memory to full kv cache memory. The default is 0.9.
|
||||
|
||||
### Mamba Radix Cache
|
||||
SGLang supports prefix caching for Qwen3-Next models named `MambaRadixCache`, which improves inference speed by reusing computation results. There are two versions of `MambaRadixCache`:
|
||||
- `no_buffer`: The default version, which is also other hybrid linear models' choice. When it is enabled, SGLang will automatically close overlap schedule for compatibility reasons.
|
||||
- `extra_buffer`: An optimized version that is compatible with features like page size > 1, overlap schedule, and speculative decoding. It also supports storing mamba state in branching positions. However, it requires two extra mamba spaces for a ping-pong buffer for each request. To enable it, add the argument `--mamba-scheduler-strategy extra_buffer` when launching the server.
|
||||
|
||||
### EAGLE Speculative Decoding
|
||||
**Description**: SGLang has supported Qwen3-Next models with [EAGLE speculative decoding](../advanced_features/speculative_decoding#EAGLE-Decoding).
|
||||
|
||||
**Usage**:
|
||||
Add arguments `--speculative-algorithm`, `--speculative-num-steps`, `--speculative-eagle-topk` and `--speculative-num-draft-tokens` to enable this feature. For example:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model Qwen/Qwen3-Next-80B-A3B-Instruct \
|
||||
--tp 4 \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--speculative-algo NEXTN
|
||||
```
|
||||
|
||||
Details can be seen in [this PR](https://github.com/sgl-project/sglang/pull/10233).
|
||||
@@ -0,0 +1,133 @@
|
||||
---
|
||||
title: "Qwen3-VL Usage"
|
||||
metatags:
|
||||
description: "Deploy Qwen3-VL vision models with SGLang: FP8 and BF16 modes, image and video input, expert parallelism. Supports H100, H200, A100 GPUs."
|
||||
---
|
||||
[Qwen3-VL](https://huggingface.co/collections/Qwen/qwen3-vl)
|
||||
is Alibaba’s latest multimodal large language model with strong text, vision, and reasoning capabilities.
|
||||
SGLang supports Qwen3-VL Family of models with Image and Video input support.
|
||||
|
||||
## Launch commands for SGLang
|
||||
|
||||
Below are suggested launch commands tailored for different hardware / precision modes
|
||||
|
||||
### FP8 (quantised) mode
|
||||
For high memory-efficiency and latency optimized deployments (e.g., on H100, H200) where FP8 checkpoint is supported:
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-VL-235B-A22B-Instruct-FP8 \
|
||||
--tp 8 \
|
||||
--ep 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--keep-mm-feature-on-device
|
||||
```
|
||||
|
||||
### Non-FP8 (BF16 / full precision) mode
|
||||
For deployments on A100/H100 where BF16 is used (or FP8 snapshot not used):
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-VL-235B-A22B-Instruct \
|
||||
--tp 8 \
|
||||
--ep 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
```
|
||||
|
||||
## Hardware-specific notes / recommendations
|
||||
|
||||
- On H100 with FP8: Use the FP8 checkpoint for best memory efficiency.
|
||||
- On A100 / H100 with BF16 (non-FP8): It’s recommended to use `--mm-max-concurrent-calls` to control parallel throughput and GPU memory usage during image/video inference.
|
||||
- On H200 & B200: The model can be run “out of the box”, supporting full context length plus concurrent image + video processing.
|
||||
|
||||
## Sending Image/Video Requests
|
||||
|
||||
### Image input:
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
url = f"http://localhost:30000/v1/chat/completions"
|
||||
|
||||
data = {
|
||||
"model": "Qwen/Qwen3-VL-30B-A3B-Instruct",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "What’s in this image?"},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true"
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
"max_tokens": 300,
|
||||
}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
print(response.text)
|
||||
```
|
||||
|
||||
### Video Input:
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
url = f"http://localhost:30000/v1/chat/completions"
|
||||
|
||||
data = {
|
||||
"model": "Qwen/Qwen3-VL-30B-A3B-Instruct",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "What’s happening in this video?"},
|
||||
{
|
||||
"type": "video_url",
|
||||
"video_url": {
|
||||
"url": "https://github.com/sgl-project/sgl-test-files/raw/refs/heads/main/videos/jobs_presenting_ipod.mp4"
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
"max_tokens": 300,
|
||||
}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
print(response.text)
|
||||
```
|
||||
|
||||
## Important Server Parameters and Flags
|
||||
|
||||
When launching the model server for **multimodal support**, you can use the following command-line arguments to fine-tune performance and behavior:
|
||||
|
||||
- `--mm-attention-backend`: Specify multimodal attention backend. Eg. `fa3`(Flash Attention 3)
|
||||
- `--mm-max-concurrent-calls <value>`: Specifies the **maximum number of concurrent asynchronous multimodal data processing calls** allowed on the server. Use this to control parallel throughput and GPU memory usage during image/video inference.
|
||||
- `--mm-per-request-timeout <seconds>`: Defines the **timeout duration (in seconds)** for each multimodal request. If a request exceeds this time limit (e.g., for very large video inputs), it will be automatically terminated.
|
||||
- `--keep-mm-feature-on-device`: Instructs the server to **retain multimodal feature tensors on the GPU** after processing. This avoids device-to-host (D2H) memory copies and improves performance for repeated or high-frequency inference workloads.
|
||||
- `SGLANG_USE_CUDA_IPC_TRANSPORT=1`: Shared memory pool based CUDA IPC for multi-modal data transport. For significantly improving e2e latency.
|
||||
|
||||
### Example usage with the above optimizations:
|
||||
```bash Command
|
||||
SGLANG_USE_CUDA_IPC_TRANSPORT=1 \
|
||||
SGLANG_VLM_CACHE_SIZE_MB=0 \
|
||||
python -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-VL-235B-A22B-Instruct \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--trust-remote-code \
|
||||
--tp-size 8 \
|
||||
--enable-cache-report \
|
||||
--log-level info \
|
||||
--max-running-requests 64 \
|
||||
--mem-fraction-static 0.65 \
|
||||
--chunked-prefill-size 8192 \
|
||||
--attention-backend fa3 \
|
||||
--mm-attention-backend fa3 \
|
||||
--enable-metrics
|
||||
```
|
||||
@@ -0,0 +1,576 @@
|
||||
---
|
||||
title: "Sampling Parameters"
|
||||
metatags:
|
||||
description: "Complete reference for SGLang sampling parameters: temperature, top_p, top_k, frequency penalty, stop tokens, and more."
|
||||
---
|
||||
This doc describes the sampling parameters of the SGLang Runtime. It is the low-level endpoint of the runtime.
|
||||
If you want a high-level endpoint that can automatically handle chat templates, consider using the [OpenAI Compatible API](./openai_api_completions).
|
||||
|
||||
## `/generate` Endpoint
|
||||
|
||||
The `/generate` endpoint accepts the following parameters in JSON format. For detailed usage, see the [native API doc](./native_api). The object is defined at `io_struct.py::GenerateReqInput`. You can also read the source code to find more arguments and docs.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "34%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type/Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>text</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[str], str]] = None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The input prompt. Can be a single prompt or a batch of prompts.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>input_ids</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[List[int]], List[int]]] = None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The token IDs for text; one can specify either text or input_ids.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>input_embeds</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[List[List[float]]], List[List[float]]]] = None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The embeddings for input_ids; one can specify either text, input_ids, or input_embeds.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>image_data</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[List[ImageDataItem]], List[ImageDataItem], ImageDataItem]] = None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The image input. Supports three formats: (1) **Raw images**: PIL Image, file path, URL, or base64 string; (2) **Processor output**: Dict with `format: "processor_output"` containing HuggingFace processor outputs; (3) **Precomputed embeddings**: Dict with `format: "precomputed_embedding"` and `feature` containing pre-calculated visual embeddings. Can be a single image, list of images, or list of lists of images. See [Multimodal Input Formats](#multimodal-input-formats) for details.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>audio_data</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[AudioDataItem], AudioDataItem]] = None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The audio input. Can be a file name, URL, or base64 encoded string.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>sampling_params</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[Dict], Dict]] = None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The sampling parameters as described in the sections below.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>rid</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[str], str]] = None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The request ID.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>return_logprob</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[bool], bool]] = None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Whether to return log probabilities for tokens.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>logprob_start_len</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[int], int]] = None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>If return_logprob, the start location in the prompt for returning logprobs. Default is "-1", which returns logprobs for output tokens only.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>top_logprobs_num</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[int], int]] = None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>If return_logprob, the number of top logprobs to return at each position.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>token_ids_logprob</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[List[int]], List[int]]] = None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>If return_logprob, the token IDs to return logprob for.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>return_text_in_logprobs</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`bool = False`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Whether to detokenize tokens in text in the returned logprobs.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>stream</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`bool = False`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Whether to stream output.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>lora_path</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[Optional[str]], Optional[str]]] = None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The path to the LoRA.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>custom_logit_processor</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[Optional[str]], str]] = None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Custom logit processor for advanced sampling control. Must be a serialized instance of `CustomLogitProcessor` using its `to_str()` method. For usage see below.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>return_hidden_states</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Union[List[bool], bool] = False`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Whether to return hidden states.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>return_routed_experts</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`bool = False`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Whether to return routed experts for MoE models. Requires `--enable-return-routed-experts` server flag. Returns base64-encoded int32 expert IDs as a flattened array with logical shape `[num_tokens, num_layers, top_k]`.</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Sampling parameters
|
||||
|
||||
The object is defined at `sampling_params.py::SamplingParams`. You can also read the source code to find more arguments and docs.
|
||||
|
||||
### Note on defaults
|
||||
|
||||
By default, SGLang initializes several sampling parameters from the model's `generation_config.json` (when the server is launched with `--sampling-defaults model`, which is the default). To use SGLang/OpenAI constant defaults instead, start the server with `--sampling-defaults openai`. You can always override any parameter per request via `sampling_params`.
|
||||
|
||||
```bash Command
|
||||
# Use model-provided defaults from generation_config.json (default behavior)
|
||||
python -m sglang.launch_server --model-path <MODEL> --sampling-defaults model
|
||||
|
||||
# Use SGLang/OpenAI constant defaults instead
|
||||
python -m sglang.launch_server --model-path <MODEL> --sampling-defaults openai
|
||||
```
|
||||
|
||||
### Core parameters
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "34%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type/Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>max_new_tokens</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int = 128`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The maximum output length measured in tokens.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>stop</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[str, List[str]]] = None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>One or multiple [stop words](https://platform.openai.com/docs/api-reference/chat/create#chat-create-stop). Generation will stop if one of these words is sampled.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>stop_token_ids</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[List[int]] = None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Provide stop words in the form of token IDs. Generation will stop if one of these token IDs is sampled.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>stop_regex</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[str, List[str]]] = None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Stop when hitting any of the regex patterns in this list</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>temperature</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`float (model default; fallback 1.0)`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[Temperature](https://platform.openai.com/docs/api-reference/chat/create#chat-create-temperature) when sampling the next token. `temperature = 0` corresponds to greedy sampling, a higher temperature leads to more diversity.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>top_p</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`float (model default; fallback 1.0)`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[Top-p](https://platform.openai.com/docs/api-reference/chat/create#chat-create-top_p) selects tokens from the smallest sorted set whose cumulative probability exceeds `top_p`. When `top_p = 1`, this reduces to unrestricted sampling from all tokens.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>top_k</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int (model default; fallback -1)`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[Top-k](https://developer.nvidia.com/blog/how-to-get-better-outputs-from-your-large-language-model/#predictability_vs_creativity) randomly selects from the `k` highest-probability tokens.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>min_p</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`float (model default; fallback 0.0)`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[Min-p](https://github.com/huggingface/transformers/issues/27670) samples from tokens with probability larger than `min_p * highest_token_probability`.</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
### Penalizers
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "34%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type/Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>frequency_penalty</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`float = 0.0`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Penalizes tokens based on their frequency in generation so far. Must be between `-2` and `2` where negative numbers encourage repeatment of tokens and positive number encourages sampling of new tokens. The scaling of penalization grows linearly with each appearance of a token.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>presence_penalty</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`float = 0.0`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Penalizes tokens if they appeared in the generation so far. Must be between `-2` and `2` where negative numbers encourage repeatment of tokens and positive number encourages sampling of new tokens. The scaling of the penalization is constant if a token occurred.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>repetition_penalty</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`float = 1.0`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Scales the logits of previously generated tokens to discourage (values > 1) or encourage (values < 1) repetition. Valid range is `[0, 2]`; `1.0` leaves probabilities unchanged.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>min_new_tokens</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int = 0`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Forces the model to generate at least `min_new_tokens` until a stop word or EOS token is sampled. Note that this might lead to unintended behavior, for example, if the distribution is highly skewed towards these tokens.</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
### Constrained decoding
|
||||
|
||||
Please refer to our dedicated guide on [constrained decoding](../advanced_features/structured_outputs) for the following parameters.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "34%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type/Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>json_schema</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[str] = None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>JSON schema for structured outputs.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>regex</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[str] = None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Regex for structured outputs.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ebnf</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[str] = None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>EBNF for structured outputs.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>structural_tag</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[str] = None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The structal tag for structured outputs.</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
### Other options
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "34%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type/Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>n</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int = 1`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Specifies the number of output sequences to generate per request. (Generating multiple outputs in one request (n > 1) is discouraged; repeating the same prompts several times offers better control and efficiency.)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ignore_eos</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`bool = False`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Don't stop generation when EOS token is sampled.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>skip_special_tokens</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`bool = True`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Remove special tokens during decoding.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>spaces_between_special_tokens</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`bool = True`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Whether or not to add spaces between special tokens during detokenization.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>no_stop_trim</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`bool = False`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Don't trim stop words or EOS token from the generated text.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>custom_params</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[List[Optional[Dict[str, Any]]]] = None`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Used when employing `CustomLogitProcessor`. For usage, see below.</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Examples
|
||||
|
||||
### Normal
|
||||
|
||||
Launch a server:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30000
|
||||
```
|
||||
|
||||
Send a request:
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
response = requests.post(
|
||||
"http://localhost:30000/generate",
|
||||
json={
|
||||
"text": "The capital of France is",
|
||||
"sampling_params": {
|
||||
"temperature": 0,
|
||||
"max_new_tokens": 32,
|
||||
},
|
||||
},
|
||||
)
|
||||
print(response.json())
|
||||
```
|
||||
|
||||
Detailed example in [send request](./send_request).
|
||||
|
||||
### Streaming
|
||||
|
||||
Send a request and stream the output:
|
||||
|
||||
```python Example
|
||||
import requests, json
|
||||
|
||||
response = requests.post(
|
||||
"http://localhost:30000/generate",
|
||||
json={
|
||||
"text": "The capital of France is",
|
||||
"sampling_params": {
|
||||
"temperature": 0,
|
||||
"max_new_tokens": 32,
|
||||
},
|
||||
"stream": True,
|
||||
},
|
||||
stream=True,
|
||||
)
|
||||
|
||||
prev = 0
|
||||
for chunk in response.iter_lines(decode_unicode=False):
|
||||
chunk = chunk.decode("utf-8")
|
||||
if chunk and chunk.startswith("data:"):
|
||||
if chunk == "data: [DONE]":
|
||||
break
|
||||
data = json.loads(chunk[5:].strip("\n"))
|
||||
output = data["text"].strip()
|
||||
print(output[prev:], end="", flush=True)
|
||||
prev = len(output)
|
||||
print("")
|
||||
```
|
||||
|
||||
Detailed example in [openai compatible api](./openai_api_completions).
|
||||
|
||||
### Multimodal
|
||||
|
||||
Launch a server:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server --model-path lmms-lab/llava-onevision-qwen2-7b-ov
|
||||
```
|
||||
|
||||
Download an image:
|
||||
|
||||
```bash Command
|
||||
curl -o example_image.png -L https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true
|
||||
```
|
||||
|
||||
Send a request:
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
response = requests.post(
|
||||
"http://localhost:30000/generate",
|
||||
json={
|
||||
"text": "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n"
|
||||
"<|im_start|>user\n<image>\nDescribe this image in a very short sentence.<|im_end|>\n"
|
||||
"<|im_start|>assistant\n",
|
||||
"image_data": "example_image.png",
|
||||
"sampling_params": {
|
||||
"temperature": 0,
|
||||
"max_new_tokens": 32,
|
||||
},
|
||||
},
|
||||
)
|
||||
print(response.json())
|
||||
```
|
||||
|
||||
The `image_data` can be a file name, a URL, or a base64 encoded string. See also `python/sglang/srt/utils.py:load_image`.
|
||||
|
||||
Streaming is supported in a similar manner as [above](#streaming).
|
||||
|
||||
Detailed example in [OpenAI API Vision](./openai_api_vision).
|
||||
|
||||
### Structured Outputs (JSON, Regex, EBNF)
|
||||
|
||||
You can specify a JSON schema, regular expression or [EBNF](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) to constrain the model output. The model output will be guaranteed to follow the given constraints. Only one constraint parameter (`json_schema`, `regex`, or `ebnf`) can be specified for a request.
|
||||
|
||||
SGLang supports two grammar backends:
|
||||
|
||||
- [XGrammar](https://github.com/mlc-ai/xgrammar) (default): Supports JSON schema, regular expression, and EBNF constraints.
|
||||
- XGrammar currently uses the [GGML BNF format](https://github.com/ggerganov/llama.cpp/blob/master/grammars/README).
|
||||
- [Outlines](https://github.com/dottxt-ai/outlines): Supports JSON schema and regular expression constraints.
|
||||
|
||||
If instead you want to initialize the Outlines backend, you can use `--grammar-backend outlines` flag:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--port 30000 --host 0.0.0.0 --grammar-backend [xgrammar|outlines] # xgrammar or outlines (default: xgrammar)
|
||||
```
|
||||
|
||||
```python Example
|
||||
import json
|
||||
import requests
|
||||
|
||||
json_schema = json.dumps({
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": {"type": "string", "pattern": "^[\\w]+$"},
|
||||
"population": {"type": "integer"},
|
||||
},
|
||||
"required": ["name", "population"],
|
||||
})
|
||||
|
||||
# JSON (works with both Outlines and XGrammar)
|
||||
response = requests.post(
|
||||
"http://localhost:30000/generate",
|
||||
json={
|
||||
"text": "Here is the information of the capital of France in the JSON format.\n",
|
||||
"sampling_params": {
|
||||
"temperature": 0,
|
||||
"max_new_tokens": 64,
|
||||
"json_schema": json_schema,
|
||||
},
|
||||
},
|
||||
)
|
||||
print(response.json())
|
||||
|
||||
# Regular expression (Outlines backend only)
|
||||
response = requests.post(
|
||||
"http://localhost:30000/generate",
|
||||
json={
|
||||
"text": "Paris is the capital of",
|
||||
"sampling_params": {
|
||||
"temperature": 0,
|
||||
"max_new_tokens": 64,
|
||||
"regex": "(France|England)",
|
||||
},
|
||||
},
|
||||
)
|
||||
print(response.json())
|
||||
|
||||
# EBNF (XGrammar backend only)
|
||||
response = requests.post(
|
||||
"http://localhost:30000/generate",
|
||||
json={
|
||||
"text": "Write a greeting.",
|
||||
"sampling_params": {
|
||||
"temperature": 0,
|
||||
"max_new_tokens": 64,
|
||||
"ebnf": 'root ::= "Hello" | "Hi" | "Hey"',
|
||||
},
|
||||
},
|
||||
)
|
||||
print(response.json())
|
||||
```
|
||||
|
||||
Detailed example in [structured outputs](../advanced_features/structured_outputs).
|
||||
|
||||
### Custom logit processor
|
||||
|
||||
Launch a server with `--enable-custom-logit-processor` flag on.
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Meta-Llama-3-8B-Instruct \
|
||||
--port 30000 \
|
||||
--enable-custom-logit-processor
|
||||
```
|
||||
|
||||
Define a custom logit processor that will always sample a specific token id.
|
||||
|
||||
```python Example
|
||||
from sglang.srt.sampling.custom_logit_processor import CustomLogitProcessor
|
||||
|
||||
class DeterministicLogitProcessor(CustomLogitProcessor):
|
||||
"""A dummy logit processor that changes the logits to always
|
||||
sample the given token id.
|
||||
"""
|
||||
|
||||
def __call__(self, logits, custom_param_list):
|
||||
# Check that the number of logits matches the number of custom parameters
|
||||
assert logits.shape[0] == len(custom_param_list)
|
||||
key = "token_id"
|
||||
|
||||
for i, param_dict in enumerate(custom_param_list):
|
||||
# Mask all other tokens
|
||||
logits[i, :] = -float("inf")
|
||||
# Assign highest probability to the specified token
|
||||
logits[i, param_dict[key]] = 0.0
|
||||
return logits
|
||||
```
|
||||
|
||||
Send a request:
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
response = requests.post(
|
||||
"http://localhost:30000/generate",
|
||||
json={
|
||||
"text": "The capital of France is",
|
||||
"custom_logit_processor": DeterministicLogitProcessor().to_str(),
|
||||
"sampling_params": {
|
||||
"temperature": 0.0,
|
||||
"max_new_tokens": 32,
|
||||
"custom_params": {"token_id": 5},
|
||||
},
|
||||
},
|
||||
)
|
||||
print(response.json())
|
||||
```
|
||||
|
||||
Send an OpenAI chat completion request:
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
from sglang.utils import print_highlight
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="meta-llama/Meta-Llama-3-8B-Instruct",
|
||||
messages=[
|
||||
{"role": "user", "content": "List 3 countries and their capitals."},
|
||||
],
|
||||
temperature=0.0,
|
||||
max_tokens=32,
|
||||
extra_body={
|
||||
"custom_logit_processor": DeterministicLogitProcessor().to_str(),
|
||||
"custom_params": {"token_id": 5},
|
||||
},
|
||||
)
|
||||
|
||||
print_highlight(f"Response: {response}")
|
||||
```
|
||||
@@ -0,0 +1,251 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Sending Requests\n",
|
||||
"This notebook provides a quick-start guide to use SGLang in chat completions after installation. Once your server is running, API documentation is available at `http://localhost:30000/docs` (Swagger UI), `http://localhost:30000/redoc` (ReDoc), or `http://localhost:30000/openapi.json` (OpenAPI spec, useful for AI agents). Replace `30000` with your port if using a different one.\n",
|
||||
"\n",
|
||||
"- For Vision Language Models, see [OpenAI APIs - Vision](openai_api_vision).\n",
|
||||
"- For Embedding Models, see [OpenAI APIs - Embedding](openai_api_embeddings) and [Encode (embedding model)](native_api#encode-embedding-model).\n",
|
||||
"- For Reward Models, see [Classify (reward model)](native_api#classify-reward-model)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Launch A Server"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sglang.test.doc_patch import launch_server_cmd\n",
|
||||
"from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
|
||||
"\n",
|
||||
"# This is equivalent to running the following command in your terminal\n",
|
||||
"# python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0\n",
|
||||
"\n",
|
||||
"server_process, port = launch_server_cmd(\"\"\"\n",
|
||||
"python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct \\\n",
|
||||
" --host 0.0.0.0 --log-level warning\n",
|
||||
"\"\"\")\n",
|
||||
"\n",
|
||||
"wait_for_server(f\"http://localhost:{port}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Using cURL\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import subprocess, json\n",
|
||||
"\n",
|
||||
"curl_command = f\"\"\"\n",
|
||||
"curl -s http://localhost:{port}/v1/chat/completions \\\n",
|
||||
" -H \"Content-Type: application/json\" \\\n",
|
||||
" -d '{{\"model\": \"qwen/qwen2.5-0.5b-instruct\", \"messages\": [{{\"role\": \"user\", \"content\": \"What is the capital of France?\"}}]}}'\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"response = json.loads(subprocess.check_output(curl_command, shell=True))\n",
|
||||
"print_highlight(response)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Using Python Requests"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import requests\n",
|
||||
"\n",
|
||||
"url = f\"http://localhost:{port}/v1/chat/completions\"\n",
|
||||
"\n",
|
||||
"data = {\n",
|
||||
" \"model\": \"qwen/qwen2.5-0.5b-instruct\",\n",
|
||||
" \"messages\": [{\"role\": \"user\", \"content\": \"What is the capital of France?\"}],\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"response = requests.post(url, json=data)\n",
|
||||
"print_highlight(response.json())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Using OpenAI Python Client"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import openai\n",
|
||||
"\n",
|
||||
"client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
|
||||
"\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"qwen/qwen2.5-0.5b-instruct\",\n",
|
||||
" messages=[\n",
|
||||
" {\"role\": \"user\", \"content\": \"List 3 countries and their capitals.\"},\n",
|
||||
" ],\n",
|
||||
" temperature=0,\n",
|
||||
" max_tokens=64,\n",
|
||||
")\n",
|
||||
"print_highlight(response)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Streaming"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import openai\n",
|
||||
"\n",
|
||||
"client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
|
||||
"\n",
|
||||
"# Use stream=True for streaming responses\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"qwen/qwen2.5-0.5b-instruct\",\n",
|
||||
" messages=[\n",
|
||||
" {\"role\": \"user\", \"content\": \"List 3 countries and their capitals.\"},\n",
|
||||
" ],\n",
|
||||
" temperature=0,\n",
|
||||
" max_tokens=64,\n",
|
||||
" stream=True,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# Handle the streaming output\n",
|
||||
"for chunk in response:\n",
|
||||
" if chunk.choices[0].delta.content:\n",
|
||||
" print(chunk.choices[0].delta.content, end=\"\", flush=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Using Native Generation APIs\n",
|
||||
"\n",
|
||||
"You can also use the native `/generate` endpoint with requests, which provides more flexibility. An API reference is available at [Sampling Parameters](sampling_params)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import requests\n",
|
||||
"\n",
|
||||
"response = requests.post(\n",
|
||||
" f\"http://localhost:{port}/generate\",\n",
|
||||
" json={\n",
|
||||
" \"text\": \"The capital of France is\",\n",
|
||||
" \"sampling_params\": {\n",
|
||||
" \"temperature\": 0,\n",
|
||||
" \"max_new_tokens\": 32,\n",
|
||||
" },\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print_highlight(response.json())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Streaming"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import requests, json\n",
|
||||
"\n",
|
||||
"response = requests.post(\n",
|
||||
" f\"http://localhost:{port}/generate\",\n",
|
||||
" json={\n",
|
||||
" \"text\": \"The capital of France is\",\n",
|
||||
" \"sampling_params\": {\n",
|
||||
" \"temperature\": 0,\n",
|
||||
" \"max_new_tokens\": 32,\n",
|
||||
" },\n",
|
||||
" \"stream\": True,\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"prev = 0\n",
|
||||
"for chunk in response.iter_lines(decode_unicode=False):\n",
|
||||
" chunk = chunk.decode(\"utf-8\")\n",
|
||||
" if chunk and chunk.startswith(\"data:\"):\n",
|
||||
" if chunk == \"data: [DONE]\":\n",
|
||||
" break\n",
|
||||
" data = json.loads(chunk[5:].strip(\"\\n\"))\n",
|
||||
" output = data[\"text\"]\n",
|
||||
" print(output[prev:], end=\"\", flush=True)\n",
|
||||
" prev = len(output)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"terminate_process(server_process)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,172 @@
|
||||
---
|
||||
title: "Tutorial: Sending a request"
|
||||
metatags:
|
||||
description: "This notebook provides a quick-start guide to use SGLang in chat completions after installation. "
|
||||
---
|
||||
This notebook provides a quick-start guide to use SGLang in chat completions after installation. Once your server is running, API documentation is available at `http://localhost:30000/docs` (Swagger UI), `http://localhost:30000/redoc` (ReDoc), or `http://localhost:30000/openapi.json` (OpenAPI spec, useful for AI agents). Replace `30000` with your port if using a different one.
|
||||
|
||||
- For Vision Language Models, see [OpenAI APIs - Vision](./openai_api_vision).
|
||||
- For Embedding Models, see [OpenAI APIs - Embedding](./openai_api_embeddings) and [Encode (embedding model)](./native_api#encode-embedding-model).
|
||||
- For Reward Models, see [Classify (reward model)](./native_api#classify-reward-model).
|
||||
|
||||
|
||||
## Launch A Server
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
from sglang.test.doc_patch import launch_server_cmd
|
||||
from sglang.utils import wait_for_server, print_highlight, terminate_process
|
||||
|
||||
# This is equivalent to running the following command in your terminal
|
||||
# python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0
|
||||
|
||||
server_process, port = launch_server_cmd(
|
||||
"""
|
||||
python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct \
|
||||
--host 0.0.0.0 --log-level warning
|
||||
"""
|
||||
)
|
||||
|
||||
wait_for_server(f"http://localhost:{port}")
|
||||
```
|
||||
|
||||
## Using cURL
|
||||
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import subprocess, json
|
||||
|
||||
curl_command = f"""
|
||||
curl -s http://localhost:{port}/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{{"model": "qwen/qwen2.5-0.5b-instruct", "messages": [{{"role": "user", "content": "What is the capital of France?"}}]}}'
|
||||
"""
|
||||
|
||||
response = json.loads(subprocess.check_output(curl_command, shell=True))
|
||||
print_highlight(response)
|
||||
```
|
||||
|
||||
## Using Python Requests
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
url = f"http://localhost:{port}/v1/chat/completions"
|
||||
|
||||
data = {
|
||||
"model": "qwen/qwen2.5-0.5b-instruct",
|
||||
"messages": [{"role": "user", "content": "What is the capital of France?"}],
|
||||
}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
print_highlight(response.json())
|
||||
```
|
||||
|
||||
## Using OpenAI Python Client
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
|
||||
client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="qwen/qwen2.5-0.5b-instruct",
|
||||
messages=[
|
||||
{"role": "user", "content": "List 3 countries and their capitals."},
|
||||
],
|
||||
temperature=0,
|
||||
max_tokens=64,
|
||||
)
|
||||
print_highlight(response)
|
||||
```
|
||||
|
||||
### Streaming
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
|
||||
client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")
|
||||
|
||||
# Use stream=True for streaming responses
|
||||
response = client.chat.completions.create(
|
||||
model="qwen/qwen2.5-0.5b-instruct",
|
||||
messages=[
|
||||
{"role": "user", "content": "List 3 countries and their capitals."},
|
||||
],
|
||||
temperature=0,
|
||||
max_tokens=64,
|
||||
stream=True,
|
||||
)
|
||||
|
||||
# Handle the streaming output
|
||||
for chunk in response:
|
||||
if chunk.choices[0].delta.content:
|
||||
print(chunk.choices[0].delta.content, end="", flush=True)
|
||||
```
|
||||
|
||||
## Using Native Generation APIs
|
||||
|
||||
You can also use the native `/generate` endpoint with requests, which provides more flexibility. An API reference is available at [Sampling Parameters](./sampling_params).
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
response = requests.post(
|
||||
f"http://localhost:{port}/generate",
|
||||
json={
|
||||
"text": "The capital of France is",
|
||||
"sampling_params": {
|
||||
"temperature": 0,
|
||||
"max_new_tokens": 32,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
print_highlight(response.json())
|
||||
```
|
||||
### Streaming
|
||||
|
||||
|
||||
|
||||
```python Example
|
||||
import requests, json
|
||||
|
||||
response = requests.post(
|
||||
f"http://localhost:{port}/generate",
|
||||
json={
|
||||
"text": "The capital of France is",
|
||||
"sampling_params": {
|
||||
"temperature": 0,
|
||||
"max_new_tokens": 32,
|
||||
},
|
||||
"stream": True,
|
||||
},
|
||||
stream=True,
|
||||
)
|
||||
|
||||
prev = 0
|
||||
for chunk in response.iter_lines(decode_unicode=False):
|
||||
chunk = chunk.decode("utf-8")
|
||||
if chunk and chunk.startswith("data:"):
|
||||
if chunk == "data: [DONE]":
|
||||
break
|
||||
data = json.loads(chunk[5:].strip("\n"))
|
||||
output = data["text"]
|
||||
print(output[prev:], end="", flush=True)
|
||||
prev = len(output)
|
||||
```
|
||||
|
||||
```python Example
|
||||
terminate_process(server_process)
|
||||
```
|
||||
@@ -0,0 +1,266 @@
|
||||
---
|
||||
title: "Development Guide for JIT Kernels"
|
||||
sidebarTitle: "JIT Kernels"
|
||||
metatags:
|
||||
description: "SGLang JIT kernel development: clangd setup, TensorMatcher, LaunchKernel, add_constant example walkthrough."
|
||||
---
|
||||
## Environment Setup
|
||||
|
||||
We strongly recommend using `clangd` as the language server for JIT kernel development.
|
||||
For Ubuntu/Debian, you can download clangd from [apt.llvm.org](https://apt.llvm.org/).
|
||||
If you are using VS Code, we recommend installing the `clangd` extension for better IDE integration.
|
||||
|
||||
All JIT-related files are located in `python/sglang/jit_kernel`.
|
||||
Unlike `sgl-kernel`, which compiles CUDA/C++ binaries ahead of time (AOT), just-in-time (JIT) kernels are compiled at runtime.
|
||||
Consequently, a static `compile_commands.json` cannot be generated.
|
||||
To enable code completion with `clangd`, run `python -m sglang.jit_kernel` to generate a `.clangd` configuration file in your current directory.
|
||||
After generating the file, restart the clangd language server. It should now recognize all JIT kernel files.
|
||||
|
||||
## Code Structure
|
||||
|
||||
### C++ Implementation
|
||||
|
||||
C++ source code is located in `python/sglang/jit_kernel/csrc`.
|
||||
Reusable functions should be placed in `python/sglang/jit_kernel/include`.
|
||||
|
||||
We use [tvm-ffi](https://github.com/apache/tvm-ffi) for efficient foreign language bindings.
|
||||
Refer to the [documentation](https://tvm.apache.org/ffi/) for advanced usage, such as exporting C++ objects.
|
||||
Typically, `tvm::ffi::TensorView` is sufficient for passing PyTorch Tensors from Python.
|
||||
|
||||
### Python Interface
|
||||
|
||||
Python interfaces are defined in `python/sglang/jit_kernel`.
|
||||
The `load_jit` utility function in `python/sglang/jit_kernel/utils.py` loads and returns the compiled module.
|
||||
To export a C++ function (e.g., `cpp_func`), pass `cuda_wrappers=[("func", "cpp_func")]` to `load_jit`.
|
||||
The function can then be called in Python as `module.func`.
|
||||
|
||||
### C++ Utilities
|
||||
|
||||
The following C++ utilities are available:
|
||||
|
||||
#### Integer Range
|
||||
|
||||
Similar to PyTorch, we provide an `irange` function to represent an integer range.
|
||||
|
||||
```C++ Example
|
||||
#include <sgl_kernel/utils.h>
|
||||
|
||||
void test() {
|
||||
for (auto i : host::irange(100)) { // [0, 100)
|
||||
// do something
|
||||
}
|
||||
for (auto i : host::irange(0, 100)) { // [0, 100)
|
||||
// do something
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
#### Runtime Checking
|
||||
|
||||
`RuntimeCheck` validates conditions at runtime. It accepts optional arguments for error reporting.
|
||||
If the check fails, these arguments are output to aid debugging.
|
||||
`RuntimeDeviceCheck` verifies the status of the last kernel launch.
|
||||
|
||||
```C++ Example
|
||||
#include <sgl_kernel/utils.h>
|
||||
#include <sgl_kernel/utils.cuh>
|
||||
|
||||
void test() {
|
||||
host::RuntimeCheck(1 + 1 == 2, 1 + 1, " != ", 2);
|
||||
host::RuntimeDeviceCheck();
|
||||
// check the provided `cudaError_t`
|
||||
host::RuntimeDeviceCheck(cudaGetLastError());
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
#### Tensor Checking
|
||||
|
||||
`TensorMatcher` provides a readable way to validate and extract tensor shape information.
|
||||
|
||||
```cpp Example
|
||||
#include <sgl_kernel/tensor.h>
|
||||
|
||||
void test(const tvm::ffi::TensorView k_cache, const tvm::ffi::TensorView v_cache) {
|
||||
using namespace host;
|
||||
|
||||
auto D = SymbolicSize{"D"}; // cache dimension
|
||||
auto N = SymbolicSize{"N"}; // kvcache stride
|
||||
auto dtype = SymbolicDType{};
|
||||
auto device = SymbolicDevice{};
|
||||
|
||||
TensorMatcher({-1, D}) //
|
||||
.with_strides({N, 1})
|
||||
.with_dtype<int32_t, int64_t>(dtype)
|
||||
.with_device<kDLCUDA, kDLCPU>(device)
|
||||
.verify(k_cache)
|
||||
.verify(v_cache);
|
||||
}
|
||||
```
|
||||
|
||||
Configure the `TensorMatcher` with expected stride, dtype, and device properties before verification.
|
||||
- If `with_strides` is omitted, the tensor is expected to be contiguous.
|
||||
- Template arguments in `with_dtype` restrict the allowed data types.
|
||||
- Template arguments in `with_device` restrict the allowed devices.
|
||||
- Values passed to `with_xxx` methods enforce equality checks.
|
||||
- Passing `-1` for size or stride allows matching any value.
|
||||
|
||||
A `Symbolic` variable must resolve to the same value across all verifications.
|
||||
Use `.unwrap()` to retrieve the matched value after verification.
|
||||
|
||||
<Note>
|
||||
`TensorMatcher` is a temporary expression and should not be stored in a variable.
|
||||
</Note>
|
||||
|
||||
<Tip>
|
||||
Add `//` at the end of the `TensorMatcher` chain to enforce proper indentation.
|
||||
</Tip>
|
||||
|
||||
#### Kernel Launching
|
||||
|
||||
`LaunchKernel::resolve_device` retrieves the current `cudaStream` from PyTorch.
|
||||
Kernels can also be launched directly using `LaunchKernel`.
|
||||
|
||||
```cpp Example
|
||||
#include <sgl_kernel/utils.cuh>
|
||||
|
||||
#include <dlpack/dlpack.h>
|
||||
|
||||
__global__ void kernel() {}
|
||||
|
||||
void test() {
|
||||
const auto num_blocks = 1;
|
||||
const auto num_threads = 32;
|
||||
const auto dynamic_smem = 0;
|
||||
|
||||
DLDevice dev; // suppose this is initialized properly
|
||||
host::LaunchKernel(num_blocks, num_threads, dev)(kernel);
|
||||
|
||||
cudaStream_t stream = host::LaunchKernel::resolve_device(dev);
|
||||
host::LaunchKernel(num_blocks, num_threads, stream, dynamic_smem)(kernel);
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
## Add new kernels
|
||||
|
||||
This section walks through a complete, end-to-end example of adding a new JIT kernel to the system.
|
||||
We use a simple add_constant kernel as a running example, which adds a constant integer value to every element of an input tensor.
|
||||
|
||||
Conceptually, the Python interface looks like this:
|
||||
|
||||
```python Example
|
||||
def add_constant(src: torch.Tensor, c: int):
|
||||
return src + c
|
||||
```
|
||||
|
||||
### STEP 1: Write the C++ kernel
|
||||
|
||||
Write your CUDA kernel in [jit_kernel/csrc/add_constant.cuh](https://github.com/sgl-project/sglang/blob/main/python/sglang/jit_kernel/csrc/add_constant.cuh). For demonstration purposes, we pass the constant value as a template parameter.
|
||||
|
||||
```cpp Example
|
||||
#include <sgl_kernel/tensor.h> // For TensorMatcher, SymbolicSize, SymbolicDevice
|
||||
#include <sgl_kernel/utils.cuh> // For LaunchKernel
|
||||
#include <sgl_kernel/utils.h> // For div_ceil, RuntimeCheck
|
||||
|
||||
#include <dlpack/dlpack.h>
|
||||
#include <tvm/ffi/container/tensor.h>
|
||||
|
||||
#include <cstddef>
|
||||
#include <cstdint>
|
||||
|
||||
namespace {
|
||||
|
||||
template <int32_t kConstant>
|
||||
__global__ void add_constant_kernel(int32_t* dst, const int32_t* src, size_t length) {
|
||||
size_t idx = blockIdx.x * blockDim.x + threadIdx.x;
|
||||
if (idx < length) {
|
||||
dst[idx] = src[idx] + kConstant;
|
||||
}
|
||||
}
|
||||
|
||||
constexpr size_t kBlockSize = 256;
|
||||
|
||||
// You can also use struct with static method as an alternative
|
||||
template <int32_t kConstant>
|
||||
void add_constant(tvm::ffi::TensorView dst, tvm::ffi::TensorView src) {
|
||||
using namespace host;
|
||||
|
||||
// 1. Validate input tensors
|
||||
SymbolicSize N = {"num_elements"};
|
||||
SymbolicDevice device_;
|
||||
TensorMatcher({N}) // 1D tensor, must be contiguous
|
||||
.with_dtype<int32_t>() // must be int32
|
||||
.with_device<kDLCUDA>(device_) // must be on CUDA device
|
||||
.verify(dst) // check tensor dst
|
||||
.verify(src); // check tensor src
|
||||
|
||||
// 2. Extract required parameters, prepare for kernel launch
|
||||
const size_t num_elements = N.unwrap();
|
||||
const size_t grid_size = div_ceil(num_elements, kBlockSize);
|
||||
const DLDevice device = device_.unwrap();
|
||||
// some extra runtime checks using host::RuntimeCheck
|
||||
RuntimeCheck(num_elements > 0, "We only support non-empty tensors, got num_elements = ", num_elements);
|
||||
|
||||
// 3. Launch the kernel. Error code will be automatically checked.
|
||||
LaunchKernel(grid_size, kBlockSize, device /*, dynamic_smem*/)(
|
||||
// kernel function
|
||||
add_constant_kernel<kConstant>,
|
||||
// kernel arguments
|
||||
static_cast<int32_t*>(dst.data_ptr()),
|
||||
static_cast<int32_t*>(src.data_ptr()),
|
||||
num_elements);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
```
|
||||
|
||||
### STEP 2: Create Python Interfaces
|
||||
|
||||
Next, expose the kernel through a Python wrapper.
|
||||
Create a new file at [jit_kernel/add_constant.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/jit_kernel/add_constant.py) and expose the needed interfaces.
|
||||
|
||||
```python Example
|
||||
from __future__ import annotations
|
||||
|
||||
import functools
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.jit_kernel.utils import load_jit, make_cpp_args
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from tvm_ffi.module import Module
|
||||
|
||||
|
||||
@functools.cache
|
||||
def _jit_add_constant_module(constant: int) -> Module:
|
||||
args = make_cpp_args(constant) # pass all the template argument
|
||||
return load_jit(
|
||||
"add_constant",
|
||||
*args,
|
||||
cuda_files=["add_constant.cuh"],
|
||||
cuda_wrappers=[("add_constant", f"add_constant<{args}>")],
|
||||
)
|
||||
|
||||
|
||||
def add_constant(src: torch.Tensor, constant: int) -> torch.Tensor:
|
||||
dst = torch.empty_like(src)
|
||||
module = _jit_add_constant_module(constant)
|
||||
module.add_constant(dst, src)
|
||||
return dst
|
||||
|
||||
```
|
||||
|
||||
### STEP 3: Use your kernel
|
||||
|
||||
Finally, import and use the kernel like a regular Python function:
|
||||
|
||||
```python Example
|
||||
from sglang.jit_kernel.add_constant import add_constant
|
||||
```
|
||||
|
||||
For a complete, runnable example, refer to [test_add_constant.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/jit_kernel/test_add_constant.py).
|
||||
@@ -0,0 +1,358 @@
|
||||
---
|
||||
title: "Bench Serving Guide"
|
||||
metatags:
|
||||
description: "SGLang bench_serving: benchmark throughput, TTFT, ITL with random/sharegpt/image datasets. Multi-backend support."
|
||||
---
|
||||
This guide explains how to benchmark online serving throughput and latency using `python -m sglang.bench_serving`. It supports multiple inference backends via OpenAI-compatible and native endpoints, and produces both console metrics and optional JSONL outputs.
|
||||
|
||||
### What it does
|
||||
|
||||
- Generates synthetic or dataset-driven prompts and submits them to a target serving endpoint
|
||||
- Measures throughput, time-to-first-token (TTFT), inter-token latency (ITL), per-request end-to-end latency, and more
|
||||
- Supports streaming or non-streaming modes, rate control, and concurrency limits
|
||||
|
||||
### Supported backends and endpoints
|
||||
|
||||
- `sglang` / `sglang-native`: `POST /generate`
|
||||
- `sglang-oai`, `vllm`, `lmdeploy`: `POST /v1/completions`
|
||||
- `sglang-oai-chat`, `vllm-chat`, `lmdeploy-chat`: `POST /v1/chat/completions`
|
||||
- `trt` (TensorRT-LLM): `POST /v2/models/ensemble/generate_stream`
|
||||
- `gserver`: Custom server (Not Implemented yet in this script)
|
||||
- `truss`: `POST /v1/models/model:predict`
|
||||
|
||||
If `--base-url` is provided, requests are sent to it. Otherwise, `--host` and `--port` are used. When `--model` is not provided, the script will attempt to query `GET /v1/models` for an available model ID (OpenAI-compatible endpoints).
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Python 3.8+
|
||||
- Dependencies typically used by this script: `aiohttp`, `numpy`, `requests`, `tqdm`, `transformers`, and for some datasets `datasets`, `pillow`, `pybase64`. Install as needed.
|
||||
- An inference server running and reachable via the endpoints above
|
||||
- If your server requires authentication, set environment variable `OPENAI_API_KEY` (used as `Authorization: Bearer <key>`)
|
||||
|
||||
### Quick start
|
||||
|
||||
Run a basic benchmark against an sglang server exposing `/generate`:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct
|
||||
```
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 --port 30000 \
|
||||
--num-prompts 1000 \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct
|
||||
```
|
||||
|
||||
Or, using an OpenAI-compatible endpoint (completions):
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend vllm \
|
||||
--base-url http://127.0.0.1:8000 \
|
||||
--num-prompts 1000 \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct
|
||||
```
|
||||
|
||||
### Datasets
|
||||
|
||||
Select with `--dataset-name`:
|
||||
|
||||
- `sharegpt` (default): loads ShareGPT-style pairs; optionally restrict with `--sharegpt-context-len` and override outputs with `--sharegpt-output-len`
|
||||
- `random`: random text lengths; sampled from ShareGPT token space
|
||||
- `random-ids`: random token ids (can lead to gibberish)
|
||||
- `image`: generates images and wraps them in chat messages; supports custom resolutions, multiple formats, and different content types
|
||||
- `generated-shared-prefix`: synthetic dataset with shared long system prompts and short questions
|
||||
- `mmmu`: samples from MMMU (Math split) and includes images
|
||||
|
||||
Common dataset flags:
|
||||
|
||||
- `--num-prompts N`: number of requests
|
||||
- `--random-input-len`, `--random-output-len`, `--random-range-ratio`: for random/random-ids/image
|
||||
- `--image-count`: Number of images per request (for `image` dataset).
|
||||
|
||||
- `--apply-chat-template`: apply tokenizer chat template when constructing prompts
|
||||
- `--dataset-path PATH`: file path for ShareGPT json; if blank and missing, it will be downloaded and cached
|
||||
|
||||
Generated Shared Prefix flags (for `generated-shared-prefix`):
|
||||
|
||||
- `--gsp-num-groups`
|
||||
- `--gsp-prompts-per-group`
|
||||
- `--gsp-system-prompt-len`
|
||||
- `--gsp-question-len`
|
||||
- `--gsp-output-len`
|
||||
|
||||
Image dataset flags (for `image`):
|
||||
|
||||
- `--image-count`: Number of images per request
|
||||
- `--image-resolution`: Image resolution; supports presets (4k, 1080p, 720p, 360p) or custom 'heightxwidth' format (e.g., 1080x1920, 512x768)
|
||||
- `--image-format`: Image format (jpeg or png)
|
||||
- `--image-content`: Image content type (random or blank)
|
||||
|
||||
### Examples
|
||||
|
||||
1. To benchmark image dataset with 3 images per request, 500 prompts, 512 input length, and 512 output length, you can run:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-3B-Instruct --disable-radix-cache
|
||||
```
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang-oai-chat \
|
||||
--dataset-name image \
|
||||
--num-prompts 500 \
|
||||
--image-count 3 \
|
||||
--image-resolution 720p \
|
||||
--random-input-len 512 \
|
||||
--random-output-len 512
|
||||
```
|
||||
|
||||
2. To benchmark random dataset with 3000 prompts, 1024 input length, and 1024 output length, you can run:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server --model-path Qwen/Qwen2.5-3B-Instruct
|
||||
```
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--dataset-name random \
|
||||
--num-prompts 3000 \
|
||||
--random-input 1024 \
|
||||
--random-output 1024 \
|
||||
--random-range-ratio 0.5
|
||||
```
|
||||
|
||||
### Choosing model and tokenizer
|
||||
|
||||
- `--model` is required unless the backend exposes `GET /v1/models`, in which case the first model ID is auto-selected.
|
||||
- `--tokenizer` defaults to `--model`. Both can be HF model IDs or local paths.
|
||||
- For ModelScope workflows, setting `SGLANG_USE_MODELSCOPE=true` enables fetching via ModelScope (weights are skipped for speed).
|
||||
- If your tokenizer lacks a chat template, the script warns because token counting can be less robust for gibberish outputs.
|
||||
|
||||
### Rate, concurrency, and streaming
|
||||
|
||||
- `--request-rate`: requests per second. `inf` sends all immediately (burst). Non-infinite rate uses a Poisson process for arrival times.
|
||||
- `--max-concurrency`: caps concurrent in-flight requests regardless of arrival rate.
|
||||
- `--disable-stream`: switch to non-streaming mode when supported; TTFT then equals total latency for chat completions.
|
||||
|
||||
### Other key options
|
||||
|
||||
- `--output-file FILE.jsonl`: append JSONL results to file; auto-named if unspecified
|
||||
- `--output-details`: include per-request arrays (generated texts, errors, ttfts, itls, input/output lens)
|
||||
- `--extra-request-body '{"top_p":0.9,"temperature":0.6}'`: merged into payload (sampling params, etc.)
|
||||
- `--disable-ignore-eos`: pass through EOS behavior (varies by backend)
|
||||
- `--warmup-requests N`: run warmup requests with short output first (default 1)
|
||||
- `--flush-cache`: call `/flush_cache` (sglang) before main run
|
||||
- `--profile`: call `/start_profile` and `/stop_profile` (requires server to enable profiling, e.g., `SGLANG_TORCH_PROFILER_DIR`)
|
||||
- `--lora-name name1 name2 ...`: randomly pick one per request and pass to backend (e.g., `lora_path` for sglang)
|
||||
- `--tokenize-prompt`: send integer IDs instead of text (currently supports `--backend sglang` only)
|
||||
|
||||
### Authentication
|
||||
|
||||
If your target endpoint requires OpenAI-style auth, set:
|
||||
|
||||
```bash Command
|
||||
export OPENAI_API_KEY=sk-...yourkey...
|
||||
```
|
||||
|
||||
The script will add `Authorization: Bearer $OPENAI_API_KEY` automatically for OpenAI-compatible routes.
|
||||
|
||||
### Metrics explained
|
||||
|
||||
Printed after each run:
|
||||
|
||||
- Request throughput (req/s)
|
||||
- Input token throughput (tok/s) - includes both text and vision tokens
|
||||
- Output token throughput (tok/s)
|
||||
- Total token throughput (tok/s) - includes both text and vision tokens
|
||||
- Total input text tokens and Total input vision tokens - per-modality breakdown
|
||||
- Concurrency: aggregate time of all requests divided by wall time
|
||||
- End-to-End Latency (ms): mean/median/std/p99 per-request total latency
|
||||
- Time to First Token (TTFT, ms): mean/median/std/p99 for streaming mode
|
||||
- Inter-Token Latency (ITL, ms): mean/median/std/p95/p99/max between tokens
|
||||
- TPOT (ms): Token processing time after first token, i.e., `(latency - ttft)/(tokens-1)`
|
||||
- Accept length (sglang-only, if available): speculative decoding accept length
|
||||
|
||||
The script also retokenizes generated text with the configured tokenizer and reports "retokenized" counts.
|
||||
|
||||
### JSONL output format
|
||||
|
||||
When `--output-file` is set, one JSON object is appended per run. Base fields:
|
||||
|
||||
- Arguments summary: backend, dataset, request_rate, max_concurrency, etc.
|
||||
- Duration and totals: completed, total_input_tokens, total_output_tokens, retokenized totals
|
||||
- Throughputs and latency statistics as printed in the console
|
||||
- `accept_length` when available (sglang)
|
||||
|
||||
With `--output-details`, an extended object also includes arrays:
|
||||
|
||||
- `input_lens`, `output_lens`
|
||||
- `ttfts`, `itls` (per request: ITL arrays)
|
||||
- `generated_texts`, `errors`
|
||||
|
||||
### End-to-end examples
|
||||
|
||||
1) sglang native `/generate` (streaming):
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 --port 30000 \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 --random-output-len 1024 --random-range-ratio 0.5 \
|
||||
--num-prompts 2000 \
|
||||
--request-rate 100 \
|
||||
--max-concurrency 512 \
|
||||
--output-file sglang_random.jsonl --output-details
|
||||
```
|
||||
|
||||
2) OpenAI-compatible Completions (e.g., vLLM):
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend vllm \
|
||||
--base-url http://127.0.0.1:8000 \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--dataset-name sharegpt \
|
||||
--num-prompts 1000 \
|
||||
--sharegpt-output-len 256
|
||||
```
|
||||
|
||||
3) OpenAI-compatible Chat Completions (streaming):
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend vllm-chat \
|
||||
--base-url http://127.0.0.1:8000 \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--dataset-name random \
|
||||
--num-prompts 500 \
|
||||
--apply-chat-template
|
||||
```
|
||||
|
||||
4) Images (VLM) with chat template:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 --port 30000 \
|
||||
--model your-vlm-model \
|
||||
--dataset-name image \
|
||||
--image-count 2 \
|
||||
--image-resolution 720p \
|
||||
--random-input-len 128 --random-output-len 256 \
|
||||
--num-prompts 200 \
|
||||
--apply-chat-template
|
||||
```
|
||||
|
||||
4a) Images with custom resolution:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 --port 30000 \
|
||||
--model your-vlm-model \
|
||||
--dataset-name image \
|
||||
--image-count 1 \
|
||||
--image-resolution 512x768 \
|
||||
--random-input-len 64 --random-output-len 128 \
|
||||
--num-prompts 100 \
|
||||
--apply-chat-template
|
||||
```
|
||||
|
||||
4b) 1080p images with PNG format and blank content:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 --port 30000 \
|
||||
--model your-vlm-model \
|
||||
--dataset-name image \
|
||||
--image-count 1 \
|
||||
--image-resolution 1080p \
|
||||
--image-format png \
|
||||
--image-content blank \
|
||||
--random-input-len 64 --random-output-len 128 \
|
||||
--num-prompts 100 \
|
||||
--apply-chat-template
|
||||
```
|
||||
|
||||
5) Generated shared prefix (long system prompts + short questions):
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 --port 30000 \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--dataset-name generated-shared-prefix \
|
||||
--gsp-num-groups 64 --gsp-prompts-per-group 16 \
|
||||
--gsp-system-prompt-len 2048 --gsp-question-len 128 --gsp-output-len 256 \
|
||||
--num-prompts 1024
|
||||
```
|
||||
|
||||
6) Tokenized prompts (ids) for strict length control (sglang only):
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 --port 30000 \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--dataset-name random \
|
||||
--tokenize-prompt \
|
||||
--random-input-len 2048 --random-output-len 256 --random-range-ratio 0.2
|
||||
```
|
||||
|
||||
7) Profiling and cache flush (sglang):
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 --port 30000 \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--profile \
|
||||
--flush-cache
|
||||
```
|
||||
|
||||
8) TensorRT-LLM streaming endpoint:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend trt \
|
||||
--base-url http://127.0.0.1:8000 \
|
||||
--model your-trt-llm-model \
|
||||
--dataset-name random \
|
||||
--num-prompts 100 \
|
||||
--disable-ignore-eos
|
||||
```
|
||||
|
||||
9) Evaluating large-scale KVCache sharing with mooncake trace (sglang only):
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 --port 30000 \
|
||||
--model mode-name \
|
||||
--dataset-name mooncake \
|
||||
--mooncake-slowdown-factor 1.0 \
|
||||
--mooncake-num-rounds 1000 \
|
||||
--mooncake-workload conversation|mooncake|agent|synthetic
|
||||
--use-trace-timestamps true \
|
||||
--random-output-len 256
|
||||
```
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
- All requests failed: verify `--backend`, server URL/port, `--model`, and authentication. Check warmup errors printed by the script.
|
||||
- Throughput seems too low: adjust `--request-rate` and `--max-concurrency`; verify server batch size/scheduling; ensure streaming is enabled if appropriate.
|
||||
- Token counts look odd: prefer chat/instruct models with proper chat templates; otherwise tokenization of gibberish may be inconsistent.
|
||||
- Image/MMMU datasets: ensure you installed extra deps (`pillow`, `datasets`, `pybase64`).
|
||||
- Authentication errors (401/403): set `OPENAI_API_KEY` or disable auth on your server.
|
||||
|
||||
### Notes
|
||||
|
||||
- The script raises the file descriptor soft limit (`RLIMIT_NOFILE`) to help with many concurrent connections.
|
||||
- For sglang, `/get_server_info` is queried post-run to report speculative decoding accept length when available.
|
||||
@@ -0,0 +1,459 @@
|
||||
---
|
||||
title: "Benchmark and Profiling"
|
||||
metatags:
|
||||
description: "SGLang benchmarking and profiling: PyTorch Profiler, Nsight Systems, layerwise NVTX, PD disaggregation profiling."
|
||||
---
|
||||
## Benchmark
|
||||
|
||||
- Benchmark the latency of running a single static batch without a server. The arguments are the same as for `launch_server.py`.
|
||||
Note that this is a simplified test script without a dynamic batching server, so it may run out of memory for a batch size that a real server can handle. A real server truncates the prefill into several batches, while this simplified script does not.
|
||||
- Without a server (do not need to launch a server)
|
||||
```bash Command
|
||||
python -m sglang.bench_one_batch --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --batch 32 --input-len 256 --output-len 32
|
||||
```
|
||||
- With a server (please use `sglang.launch_server` to launch a server first and run the following command.)
|
||||
```bash Command
|
||||
python -m sglang.bench_one_batch_server --base-url http://127.0.0.1:30000 --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --batch-size 32 --input-len 256 --output-len 32
|
||||
```
|
||||
|
||||
|
||||
- Benchmark offline processing. This script will start an offline engine and run the benchmark.
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 10
|
||||
```
|
||||
|
||||
- Benchmark online serving. Please use `sglang.launch_server` to launch a server first and run the following command.
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving --backend sglang --num-prompt 10
|
||||
```
|
||||
|
||||
## Profile with PyTorch Profiler
|
||||
|
||||
[Pytorch Profiler](https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html) is a convenient basic tool to inspect kernel execution time, call stack, and kernel overlap and occupancy.
|
||||
|
||||
### Profile a server with `sglang.bench_serving`
|
||||
|
||||
```bash Command
|
||||
# set trace path
|
||||
export SGLANG_TORCH_PROFILER_DIR=/root/sglang/profile_log
|
||||
|
||||
# start server
|
||||
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct
|
||||
|
||||
# send profiling request from client
|
||||
python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --sharegpt-output-len 100 --profile
|
||||
```
|
||||
|
||||
Please make sure that the `SGLANG_TORCH_PROFILER_DIR` should be set at both server and client side, otherwise the trace file cannot be generated correctly . A secure way will be setting `SGLANG_TORCH_PROFILER_DIR` in the `.*rc` file of shell (e.g. `~/.bashrc` for bash shells).
|
||||
|
||||
For more details, please refer to [Bench Serving Guide](./bench_serving).
|
||||
|
||||
### Profile In PD Disaggregation Mode
|
||||
|
||||
When profiling in PD disaggregation mode, prefill and decode workers **must be profiled separately** due to torch profiler limitations. The `bench_serving` command provides dedicated options for this:
|
||||
|
||||
#### Profile Prefill Workers
|
||||
|
||||
```bash Command
|
||||
# set trace path
|
||||
export SGLANG_TORCH_PROFILER_DIR=/root/sglang/profile_log
|
||||
|
||||
# start prefill and decode servers (see PD disaggregation docs for setup)
|
||||
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --disaggregation-mode prefill
|
||||
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --disaggregation-mode decode --port 30001 --base-gpu-id 1
|
||||
|
||||
# start router
|
||||
python -m sglang_router.launch_router --pd-disaggregation --prefill http://127.0.0.1:30000 --decode http://127.0.0.1:30001 --host 0.0.0.0 --port 8000
|
||||
|
||||
# send profiling request targeting prefill workers
|
||||
python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --sharegpt-output-len 100 --profile --pd-separated --profile-prefill-url http://127.0.0.1:30000
|
||||
```
|
||||
|
||||
#### Profile Decode Workers
|
||||
|
||||
```bash Command
|
||||
# send profiling request targeting decode workers
|
||||
python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --sharegpt-output-len 100 --profile --pd-separated --profile-decode-url http://127.0.0.1:30001
|
||||
```
|
||||
|
||||
#### Important Notes
|
||||
|
||||
- `--profile-prefill-url` and `--profile-decode-url` are **mutually exclusive** - you cannot profile both at the same time
|
||||
- Both options support multiple worker URLs for multi-instance setups:
|
||||
```bash Command
|
||||
# Profile multiple prefill workers
|
||||
python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --profile --pd-separated --profile-prefill-url http://127.0.0.1:30000 http://127.0.0.1:30002
|
||||
|
||||
# Profile multiple decode workers
|
||||
python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --profile --pd-separated --profile-decode-url http://127.0.0.1:30001 http://127.0.0.1:30003
|
||||
```
|
||||
- Make sure `SGLANG_TORCH_PROFILER_DIR` is set on all worker nodes before starting the servers
|
||||
- For more details on setting up PD disaggregation, see [PD Disaggregation Guide](../advanced_features/pd_disaggregation)
|
||||
|
||||
### Profile a server with `sglang.bench_offline_throughput`
|
||||
```bash Command
|
||||
export SGLANG_TORCH_PROFILER_DIR=/root/sglang/profile_log
|
||||
|
||||
# profile one batch with bench_one_batch.py
|
||||
# batch size can be controlled with --batch argument
|
||||
python3 -m sglang.bench_one_batch --model-path meta-llama/Llama-3.1-8B-Instruct --batch 32 --input-len 1024 --output-len 10 --profile
|
||||
|
||||
# profile multiple batches with bench_offline_throughput.py
|
||||
python -m sglang.bench_offline_throughput --model-path meta-llama/Llama-3.1-8B-Instruct --dataset-name random --num-prompts 10 --profile --mem-frac=0.8
|
||||
```
|
||||
|
||||
### Profile a server with `sglang.profiler`
|
||||
|
||||
When the server is running (e.g., processing a decoding request), you can start live profiling immediately by sending a profile request to the server.
|
||||
|
||||
You can do this by running `python3 -m sglang.profiler`. For example:
|
||||
|
||||
```text Output
|
||||
# Terminal 1: Send a generation request
|
||||
python3 -m sglang.test.send_one
|
||||
|
||||
# Terminal 2: Before the above request finishes, quickly launch the following command in a separate terminal.
|
||||
# It will generate a profile of the above request for several decoding batches.
|
||||
python3 -m sglang.profiler
|
||||
```
|
||||
|
||||
You can also combine the above operations into a single command
|
||||
|
||||
```text Output
|
||||
python3 -m sglang.test.send_one --profile
|
||||
```
|
||||
|
||||
### Profile a server with HTTP API endpoints
|
||||
|
||||
SGLang provides HTTP API endpoints to control profiling on a running server. This allows you to start and stop profiling programmatically, which is useful for capturing specific workload patterns.
|
||||
|
||||
#### Using `/start_profile` endpoint
|
||||
|
||||
The `/start_profile` endpoint starts profiling on the server. You can control when profiling begins and how long it runs using the following parameters:
|
||||
|
||||
**Basic usage:**
|
||||
|
||||
```bash Command
|
||||
# Start profiling immediately for 10 steps
|
||||
curl -X POST http://127.0.0.1:30000/start_profile \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"num_steps": 10
|
||||
}'
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
|
||||
- `output_dir` (optional): Directory where profile traces will be saved. If not specified, uses `SGLANG_TORCH_PROFILER_DIR` environment variable, or `/tmp` as the default
|
||||
- `num_steps` (optional): Number of steps to profile. If not specified, profiling continues until manually stopped with `/end_profile`
|
||||
- `start_step` (optional): Step number at which to start profiling (inclusive). Useful for skipping warmup iterations
|
||||
- `activities` (optional): List of activities to profile, e.g., `["CPU", "GPU"]`. Default is `["CPU", "GPU"]`
|
||||
- `merge_profiles` (optional): Whether to merge distributed traces. Default is `false`
|
||||
|
||||
**Note on step ranges:** Profiling starts at `start_step` (inclusive) and continues for `num_steps` iterations. For example, with `start_step=3` and `num_steps=10`, profiling captures steps 3, 4, 5, 6, 7, 8, 9, 10, 11, and 12 (10 steps total, starting from step 3).
|
||||
|
||||
**Advanced usage with `start_step`:**
|
||||
|
||||
```bash Command
|
||||
# Wait 5 steps (warmup), then profile for 10 steps
|
||||
curl -X POST http://127.0.0.1:30000/start_profile \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"output_dir": "/tmp/profiles",
|
||||
"start_step": 5,
|
||||
"num_steps": 10,
|
||||
"activities": ["CPU", "GPU"]
|
||||
}'
|
||||
```
|
||||
|
||||
**Continuous profiling (manual stop):**
|
||||
|
||||
```bash Command
|
||||
# Start profiling without num_steps - must manually stop with /end_profile
|
||||
curl -X POST http://127.0.0.1:30000/start_profile
|
||||
```
|
||||
|
||||
#### Using `/end_profile` endpoint
|
||||
|
||||
The `/end_profile` endpoint stops an ongoing profiling session and saves the trace file.
|
||||
|
||||
```bash Command
|
||||
# Stop profiling and save traces
|
||||
curl -X POST http://127.0.0.1:30000/end_profile
|
||||
```
|
||||
|
||||
This is only needed when you start profiling without specifying `num_steps`. If `num_steps` is specified, profiling will automatically stop after that many steps.
|
||||
|
||||
#### Example workflow
|
||||
|
||||
```bash Command
|
||||
# Terminal 1: Start the server
|
||||
export SGLANG_TORCH_PROFILER_DIR=/tmp/profiles
|
||||
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct
|
||||
|
||||
# Terminal 2: Start continuous profiling
|
||||
curl -X POST http://127.0.0.1:30000/start_profile \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"start_step": 3
|
||||
}'
|
||||
|
||||
# Terminal 3: Send requests to generate load
|
||||
python -m sglang.bench_serving --backend sglang --num-prompts 100
|
||||
|
||||
# Terminal 2: Stop profiling when done
|
||||
curl -X POST http://127.0.0.1:30000/end_profile
|
||||
```
|
||||
|
||||
### Profiler Trace Merger for Distributed Traces
|
||||
|
||||
SGLang now supports automatic merging of profiling traces from distributed setups with multiple parallelism types (TP, DP, PP, EP). This feature is particularly useful for analyzing performance across distributed runs.
|
||||
|
||||
#### Multi-Node Profiling and Shared Storage Considerations
|
||||
|
||||
Single-node profiler output merging is completely supported. When profiling in distributed environments spanning multiple nodes, shared storage (e.g., NFS, Lustre) should be accessible by all nodes for the output directory to enable merging of trace files.
|
||||
|
||||
If there is no shared storage accessible across nodes, automatic merging of trace files during profiling is not supported directly as of now.
|
||||
|
||||
#### HTTP API Usage
|
||||
|
||||
```bash Command
|
||||
# Start profiling with automatic trace merging enabled
|
||||
curl -X POST <BASE_URL>/start_profile \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"output_dir": "/tmp/profiles", # where to store profile traces
|
||||
"num_steps": 10,
|
||||
"activities": ["CPU", "GPU"],
|
||||
"merge_profiles": true # optional argument to merge profile traces (default=False)
|
||||
}'
|
||||
```
|
||||
|
||||
#### Command Line Usage
|
||||
|
||||
```bash Command
|
||||
# Start profiling with merge enabled
|
||||
python -m sglang.profiler \
|
||||
--num-steps 10 \
|
||||
--cpu \
|
||||
--gpu \
|
||||
--output-dir /tmp/profiles \
|
||||
--merge-profiles # optional argument to merge profile traces (default=False)
|
||||
```
|
||||
|
||||
#### Output Files
|
||||
|
||||
The profile merger generates:
|
||||
- Individual rank trace files: `{profile_id}-TP-{tp}-DP-{dp}-PP-{pp}-EP-{ep}.trace.json.gz`
|
||||
- Merged trace file: `merged-{profile_id}.trace.json.gz`
|
||||
|
||||
### Possible PyTorch bugs
|
||||
If in any cases you encounter the following error (for example, using qwen 2.5 VL):
|
||||
```bash Command
|
||||
RuntimeError: !stack.empty() INTERNAL ASSERT FAILED at "/pytorch/torch/csrc/autograd/profiler_python.cpp":983, please report a bug to PyTorch. Python replay stack is empty.
|
||||
```
|
||||
This is likely a PyTorch Bug reported in [Bug: vLLM Profiler](https://github.com/vllm-project/vllm/issues/18240) and [Bug: torch.profiler.profile](https://github.com/pytorch/pytorch/issues/101632). As a workaround, you may disable `with_stack` with an environment variable such as follows:
|
||||
```bash Command
|
||||
export SGLANG_PROFILE_WITH_STACK=False
|
||||
python -m sglang.bench_offline_throughput --model-path meta-llama/Llama-3.1-8B-Instruct --dataset-name random --num-prompts 10 --profile --mem-frac=0.8
|
||||
```
|
||||
|
||||
### View traces
|
||||
|
||||
Trace files can be loaded and visualized from:
|
||||
|
||||
1. https://ui.perfetto.dev/ (any browser)
|
||||
2. chrome://tracing (Chrome browser only)
|
||||
|
||||
If browser cannot open trace file due to its large size,
|
||||
client can generate a small trace file (<100MB) by controlling number of prompts and lengths of prompt outputs.
|
||||
For example, when profiling a server,
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 2 --sharegpt-output-len 100 --profile
|
||||
```
|
||||
|
||||
This command sets the number of prompts to 2 with `--num-prompts` argument and limits the length of output sequences to 100 with `--sharegpt-output-len` argument, which can generate a small trace file for browser to open smoothly.
|
||||
|
||||
Additionally, if you want to locate the SGLang Python source code through the cuda kernel in Trace, you need to disable CUDA Graph when starting the service. This can be done by using the `--disable-cuda-graph` parameter in the command to start the service.
|
||||
|
||||
## Profile with Nsight
|
||||
|
||||
[Nsight systems](https://docs.nvidia.com/nsight-systems/) is an advanced tool that exposes more profiling details, such as register and shared memory usage, annotated code regions and low-level CUDA APIs and events.
|
||||
|
||||
1. Prerequisite:
|
||||
|
||||
Install using apt, or run inside a [NVIDIA Docker container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch/tags) or [SGLang Docker container](https://github.com/sgl-project/sglang/tree/main/docker).
|
||||
|
||||
```bash Command
|
||||
# install nsys
|
||||
# https://docs.nvidia.com/nsight-systems/InstallationGuide/index.html
|
||||
apt update
|
||||
apt install -y --no-install-recommends gnupg
|
||||
echo "deb http://developer.download.nvidia.com/devtools/repos/ubuntu$(source /etc/lsb-release; echo "$DISTRIB_RELEASE" | tr -d .)/$(dpkg --print-architecture) /" | tee /etc/apt/sources.list.d/nvidia-devtools.list
|
||||
apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
|
||||
apt update
|
||||
apt install nsight-systems-cli
|
||||
```
|
||||
|
||||
2. To profile a single batch, use
|
||||
|
||||
```bash Command
|
||||
nsys profile --trace-fork-before-exec=true --cuda-graph-trace=node python3 -m sglang.bench_one_batch --model meta-llama/Meta-Llama-3-8B --batch-size 64 --input-len 512
|
||||
```
|
||||
|
||||
3. To profile a server, e.g.
|
||||
|
||||
```bash Command
|
||||
# launch the server, set the delay and duration times according to needs
|
||||
# after the duration time has been used up, server will be killed by nsys
|
||||
|
||||
nsys profile --trace-fork-before-exec=true --cuda-graph-trace=node -o sglang.out --delay 60 --duration 70 python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --disable-radix-cache
|
||||
|
||||
# client
|
||||
python3 -m sglang.bench_serving --backend sglang --num-prompts 1000 --dataset-name random --random-input 1024 --random-output 512
|
||||
```
|
||||
|
||||
In practice, we recommend users to set `--duration` argument to a large value. Whenever user wants the server to stop profiling. Firstly run:
|
||||
|
||||
```bash Command
|
||||
nsys sessions list
|
||||
```
|
||||
|
||||
to get the session id in the form of `profile-XXXXX`, then run:
|
||||
|
||||
```bash Command
|
||||
nsys stop --session=profile-XXXXX
|
||||
```
|
||||
|
||||
to manually kill the profiler and generate `nsys-rep` files instantly.
|
||||
|
||||
4. Use NVTX to annotate code regions, e.g. to see their execution time.
|
||||
|
||||
```bash Command
|
||||
# install nvtx
|
||||
pip install nvtx
|
||||
```
|
||||
|
||||
```python Example
|
||||
# code snippets
|
||||
import nvtx
|
||||
with nvtx.annotate("description", color="color"):
|
||||
# some critical code
|
||||
```
|
||||
|
||||
### Layer-wise NVTX Profiling with Nsight Systems
|
||||
|
||||
SGLang provides built-in layerwise NVTX annotations that can be combined with the CUDA Profiler for detailed per-layer profiling in Nsight Systems. This is particularly useful for identifying performance bottlenecks at the layer level.
|
||||
|
||||
#### Using `--enable-layerwise-nvtx-marker` with Nsight Systems and `/start_profile`
|
||||
|
||||
The `--enable-layerwise-nvtx-marker` flag automatically adds NVTX markers to every layer in your model. This is particularly powerful when combined with Nsight Systems profiling to see detailed per-layer performance.
|
||||
|
||||
**Method 1: Using `/start_profile` with CUDA_PROFILER (for programmatic control)**
|
||||
|
||||
This method allows you to control exactly when profiling starts/stops via HTTP API while Nsight Systems is running.
|
||||
|
||||
1. Launch the server with layerwise NVTX enabled under Nsight Systems:
|
||||
|
||||
```bash Command
|
||||
# Terminal 1: Start server with nsys and capture-range option
|
||||
nsys profile --trace-fork-before-exec=true \
|
||||
--cuda-graph-trace=node \
|
||||
--capture-range=cudaProfilerApi \
|
||||
--capture-range-end=stop \
|
||||
-o layerwise_profile \
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
||||
--enable-layerwise-nvtx-marker \
|
||||
--disable-cuda-graph
|
||||
```
|
||||
|
||||
Note: NVTX markers are not emitted for kernel launches captured by CUDA graphs. Use `--disable-cuda-graph` to ensure all layerwise NVTX markers are emitted in the trace.
|
||||
|
||||
2. In another terminal, control profiling via `/start_profile` with `CUDA_PROFILER` activity:
|
||||
|
||||
```bash Command
|
||||
# Terminal 2: Wait for server to be ready, then start CUDA profiling
|
||||
# Wait 3 steps for warmup, then profile for 10 steps
|
||||
curl -X POST http://127.0.0.1:30000/start_profile \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"start_step": 3,
|
||||
"num_steps": 10,
|
||||
"activities": ["CUDA_PROFILER"]
|
||||
}'
|
||||
```
|
||||
|
||||
3. Send requests to generate load:
|
||||
|
||||
```bash Command
|
||||
# Terminal 3: Generate workload
|
||||
python -m sglang.bench_serving --backend sglang --num-prompts 100
|
||||
```
|
||||
|
||||
4. Profiling will automatically stop after 10 steps (due to `num_steps: 10`). If you hadn't specified `num_steps`, you would need to manually stop it:
|
||||
|
||||
```bash Command
|
||||
# Terminal 2: Only needed if num_steps was not specified
|
||||
curl -X POST http://127.0.0.1:30000/end_profile
|
||||
```
|
||||
|
||||
The `--capture-range=cudaProfilerApi` option tells Nsight Systems to only capture data between `cudaProfilerStart()` and `cudaProfilerStop()` calls (triggered by `/start_profile` and `/end_profile`), reducing overhead and file size. The `start_step` parameter skips the first 3 steps to avoid capturing warmup overhead.
|
||||
|
||||
**Method 2: Simpler approach without `/start_profile` API**
|
||||
|
||||
For simpler use cases where you don't need fine-grained control over profiling start/stop, you can profile with Nsight Systems capturing the entire workload:
|
||||
|
||||
```bash Command
|
||||
# Terminal 1: Start server with layerwise NVTX
|
||||
# Note: --disable-cuda-graph ensures all NVTX markers are emitted
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
||||
--enable-layerwise-nvtx-marker \
|
||||
--disable-cuda-graph
|
||||
|
||||
# Terminal 2: Profile the benchmarking client
|
||||
nsys profile --trace-fork-before-exec=true \
|
||||
--cuda-graph-trace=node \
|
||||
-o layerwise_profile \
|
||||
python -m sglang.bench_serving --backend sglang --num-prompts 10
|
||||
```
|
||||
|
||||
This approach profiles the entire client execution, including all server interactions. The layerwise NVTX markers will be visible in the Nsight Systems timeline.
|
||||
|
||||
**Viewing the profiling results:**
|
||||
|
||||
Open the generated `.qdrep` file with Nsight Systems:
|
||||
|
||||
```bash Command
|
||||
nsys-ui layerwise_profile.qdrep
|
||||
```
|
||||
|
||||
In the Nsight Systems GUI, you'll see:
|
||||
- **NVTX ranges**: Each layer appears as a labeled range in the timeline with detailed information in the marker metadata
|
||||
- **CUDA kernels**: All GPU kernels are shown alongside the layer annotations
|
||||
- **Layer hierarchy**: The full module path (e.g., `meta-llama/Meta-Llama-3.1-8B-Instruct.model.layers.0.self_attn.qkv_proj`) helps identify specific layers. The prefix uses the full model path from `--model-path`.
|
||||
- **Tensor shapes**: Input/output dimensions and parameter shapes are included in the NVTX marker data
|
||||
|
||||
**Benefits of layerwise NVTX profiling:**
|
||||
|
||||
- **Granular visibility**: See exactly which layers are taking the most time
|
||||
- **Memory tracking**: Identify layers with large memory allocations
|
||||
- **Bottleneck identification**: Quickly locate inefficient operations
|
||||
- **Communication overhead**: In multi-GPU setups, see per-layer communication costs
|
||||
- **Development debugging**: Validate that model architecture changes have the expected performance impact
|
||||
|
||||
## Other tips
|
||||
|
||||
1. You can benchmark a model using dummy weights by only providing the config.json file. This allows for quick testing of model variants without training. To do so, add `--load-format dummy` to the above commands and then you only need a correct `config.json` under the checkpoint folder.
|
||||
2. You can benchmark a model with modified configs (e.g., less layers) by using `--json-model-override-args`. For example, you can benchmark a model with only 2 layers and 2 kv heads using:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_one_batch --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --batch 32 --input-len 256 --output-len 32 --load-format dummy --json-model-override-args '{"num_hidden_layers": 1, "num_key_value_heads": 1}'
|
||||
```
|
||||
|
||||
3. You can use `--python-backtrace=cuda` to see python call stack for all CUDA kernels, as in PyTorch Profiler. (Caveat: this can cause inaccurately long kernel runtimes for CUDA event based timing)
|
||||
4. For more arguments see [Nsight Systems User Guide](https://docs.nvidia.com/nsight-systems/UserGuide/index.html).
|
||||
@@ -0,0 +1,149 @@
|
||||
---
|
||||
title: "Contribution Guide"
|
||||
mode: wide
|
||||
metatags:
|
||||
description: "SGLang contribution guide: source install, pre-commit, unit tests, CI triggers, code style, sgl-kernel updates."
|
||||
---
|
||||
Welcome to **SGLang**! We appreciate your interest in contributing. This guide provides a concise overview of how to set up your environment, run tests, build documentation, and open a Pull Request (PR). Whether you're fixing a small bug or developing a major feature, we encourage following these steps for a smooth contribution process.
|
||||
|
||||
## Install SGLang from Source
|
||||
|
||||
### Fork and clone the repository
|
||||
|
||||
**Note**: New contributors do **not** have the write permission to push to the official SGLang repo. Please fork the repository under your GitHub account, then clone your fork locally.
|
||||
|
||||
```bash
|
||||
git clone https://github.com/<your_user_name>/sglang.git
|
||||
```
|
||||
|
||||
### Build from source
|
||||
|
||||
Refer to [Install SGLang from Source](../get-started/installation).
|
||||
|
||||
## Format code with pre-commit
|
||||
|
||||
We use [pre-commit](https://pre-commit.com/) to maintain consistent code style checks. Before pushing your changes, please run:
|
||||
|
||||
```bash
|
||||
pip3 install pre-commit
|
||||
pre-commit install
|
||||
pre-commit run --all-files
|
||||
```
|
||||
|
||||
- **`pre-commit run --all-files`** manually runs all configured checks, applying fixes if possible. If it fails the first time, re-run it to ensure lint errors are fully resolved. Make sure your code passes all checks **before** creating a Pull Request.
|
||||
- **Do not commit** directly to the `main` branch. Always create a new branch (e.g., `feature/my-new-feature`), push your changes, and open a PR from that branch.
|
||||
|
||||
## Run and add unit tests
|
||||
|
||||
If you add a new feature or fix a bug, please add corresponding unit tests to ensure coverage and prevent regression.
|
||||
SGLang uses Python's built-in [unittest](https://docs.python.org/3/library/unittest.html) framework.
|
||||
For detailed instructions on running tests and integrating them into CI, refer to [test/README](https://github.com/sgl-project/sglang/tree/main/test/README).
|
||||
|
||||
## Write documentations
|
||||
|
||||
We recommend new contributors start from writing documentation, which helps you quickly understand SGLang codebase.
|
||||
For more details, please refer to [docs/README](https://github.com/sgl-project/sglang/blob/main/docs/README.md).
|
||||
|
||||
## Test the accuracy
|
||||
If your code changes the model output, please run the accuracy tests. A quick sanity check is the few-shot GSM8K.
|
||||
|
||||
```text Output
|
||||
# Launch a server
|
||||
python3 -m sglang.launch_server --model Qwen/Qwen2-7B-Instruct
|
||||
|
||||
# Evaluate
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
|
||||
```
|
||||
|
||||
Please note that the above script is primarily a sanity check, not a rigorous accuracy or speed test.
|
||||
This test can have significant variance (1%-5%) in accuracy due to batching and the non-deterministic nature of the inference engine.
|
||||
Also, do not rely on the "Latency/Output throughput" from this script, as it is not a proper speed test.
|
||||
|
||||
GSM8K is too easy for state-of-the-art models nowadays. Please try your own more challenging accuracy tests.
|
||||
You can find additional accuracy eval examples in:
|
||||
- [test_eval_accuracy_large.py](https://github.com/sgl-project/sglang/blob/main/test/srt/test_eval_accuracy_large.py)
|
||||
- [test_gpt_oss_1gpu.py](https://github.com/sgl-project/sglang/blob/main/test/srt/test_gpt_oss_1gpu.py)
|
||||
|
||||
## Benchmark the speed
|
||||
Refer to [Benchmark and Profiling](../developer_guide/benchmark_and_profiling).
|
||||
|
||||
## Requesting a review for merge
|
||||
You can follow the pull request merge process described in [MAINTAINER](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER).
|
||||
You will need to work with the Merge Oncall, Codeowner, and other reviewers to get their approvals.
|
||||
Then your PR can be merged.
|
||||
|
||||
## How to Trigger CI Tests
|
||||
|
||||
We have a lot of open PRs but limited CI machines, so only top and trusted contributors have permission to trigger CI tests.
|
||||
Users with permission are listed in the [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json)
|
||||
|
||||
For CI to run on a pull request, it must have the "run-ci" label. Authorized users can add the label or rerun failed tests by commenting on the PR with one of these commands:
|
||||
|
||||
- `/tag-run-ci-label`: Adds the "run-ci" label. Every future commit will trigger CI.
|
||||
- `/rerun-failed-ci`: Reruns the failed or flaky tests from the most recent commit.
|
||||
- `/tag-and-rerun-ci`: A single command that performs both `/tag-run-ci-label` and `/rerun-failed-ci`.
|
||||
- `/rerun-stage <stage-name>`: Reruns a specific test stage without waiting for its dependencies. This is useful when you want to quickly validate a fix for a specific test failure instead of waiting ~30 minutes for preceding stages to complete.
|
||||
|
||||
If you have permission, the [Slash Command Handler](https://github.com/sgl-project/sglang/actions/workflows/slash-command-handler.yml) will run your command and react with a +1 to your comment. It may take up to a few minutes for the reaction to appear. Here's a usage [example](https://github.com/sgl-project/sglang/pull/14253#issuecomment-3599509302).
|
||||
|
||||
To avoid spamming a PR with too many `/rerun-failed-ci` comments, you can also trigger the command by editing an existing comment and adding any suffix (e.g., `/rerun-failed-ci try again`).
|
||||
|
||||
Example of rerunning a single test stage: `/rerun-stage unit-test-backend-4-gpu`.
|
||||
|
||||
If you don't have permission, please ask maintainers to trigger CI for you.
|
||||
|
||||
### CI rate limits
|
||||
|
||||
Due to CI scheduling and limited resources, higher-priority PRs may preempt running jobs. In such cases, you may need to rerun the tests.
|
||||
|
||||
We apply CI rate limits to prevent abuse and ensure fair usage of our CI resources.
|
||||
|
||||
Each CI workflow has a default limit defined in its workflow configuration file. For example, in [pr-gate.yml](https://github.com/sgl-project/sglang/blob/main/.github/workflows/pr-gate.yml), the default cooldown period is 120 minutes, and each workflow can override it via the `cool-down-minutes` input parameter:
|
||||
|
||||
```yaml Config
|
||||
cool-down-minutes:
|
||||
description: "Default cooldown period in minutes; 0 disables rate limiting"
|
||||
type: number
|
||||
default: 120
|
||||
```
|
||||
|
||||
Users listed in [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json) may have a per-user cooldown interval. In practice, we use the minimum of the workflow's default window and the user-specific interval.
|
||||
|
||||
|
||||
## Code style guidance
|
||||
- Avoid code duplication. If the same code snippet (more than five lines) appears multiple times, extract it into a shared function.
|
||||
- Minimize device synchronization. Reduce expensive CPU-GPU synchronization operations, such as `tensor.item()` or `tensor.cpu()`, whenever possible. Use vectorized code.
|
||||
- Prioritize extreme efficiency. SGLang is a runtime, and most of your code runs on the critical path for every request. Optimize all minor overheads as much as possible, especially in the model forward code.
|
||||
- A common pattern is some runtime checks in the model forward pass (e.g., [this](https://github.com/sgl-project/sglang/blob/f1b0eda55c2c4838e8ab90a0fac7fb1e3d7064ab/python/sglang/srt/models/deepseek_v2.py#L486-L491)). These are very likely the same for every layer. Please cache the result as a single boolean value whenever possible.
|
||||
- Make functions as pure as possible. Avoid in-place modification of arguments.
|
||||
- Keep files concise. If a file exceeds 2,000 lines of code, split it into multiple smaller files. (e.g., `scheduler.py`, `scheduler_output_processor_mixin.py`)
|
||||
- Keep tests run fast.
|
||||
- If a single test file run longer than 500 seconds, split it into multiple smaller files (e.g., `test_eagle_infer_a.py`, `test_eagle_infer_b.py`).
|
||||
- If a single job in a github workflow runs longer than 30 mins, split it into smaller jobs/steps.
|
||||
- Reuse server launches in your unit tests to make tests run faster.
|
||||
- When supporting new hardware or features, follow these guidelines:
|
||||
- Do not drastically change existing code.
|
||||
- Always prefer new files to introduce specific components for your new hardware (e.g., `allocator_ascend.py`).
|
||||
- If you write multiple if/else blocks for new features, ensure the common path (e.g., NVIDIA hardware or the existing code path) is the first branch.
|
||||
|
||||
## How to update sgl-kernel
|
||||
Since sglang and sgl-kernel are separate Python packages, our current GitHub CI infrastructure does not support updating a kernel and using it immediately within the same pull request (PR).
|
||||
To add a new kernel or modify an existing one in the sgl-kernel package, you must use multiple PRs.
|
||||
|
||||
Follow these steps:
|
||||
|
||||
1. Submit a PR to update the sgl-kernel source code without using it in sglang python package (e.g., [#8884](https://github.com/sgl-project/sglang/pull/8884/files)).
|
||||
2. Bump the version of sgl-kernel (e.g., [#9220](https://github.com/sgl-project/sglang/pull/9220/files)).
|
||||
- Once merged, this will trigger an automatic release of the sgl-kernel wheel to PyPI.
|
||||
- If not urgent, you can wait for other people to release the wheel. A new version will typically be released within one week.
|
||||
3. Apply the changes:
|
||||
- Update the sgl-kernel version in `sglang/python/pyproject.toml` to use the modified kernels.
|
||||
- Update the related caller code in the sglang to use the new kernel.
|
||||
|
||||
## Tips for newcomers
|
||||
|
||||
If you want to contribute but don't have a specific idea in mind, pick issues labeled ["good first issue" or "help wanted"](https://github.com/sgl-project/sglang/issues?q=is%3Aissue+label%3A%22good+first+issue%22%2C%22help+wanted%22). These tasks typically have lower complexity and provide an excellent introduction to the codebase. Also check out this [code walk-through](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/tree/main/sglang/code-walk-through) for a deeper look into SGLang's workflow.
|
||||
|
||||
If you have any questions or want to start a discussion, please feel free to ask in our [Slack channel](https://slack.sglang.io).
|
||||
|
||||
Thank you for your interest in SGLang. Happy coding!
|
||||
@@ -0,0 +1,119 @@
|
||||
---
|
||||
title: "Development Guide Using Docker"
|
||||
sidebarTitle: "Using Docker"
|
||||
metatags:
|
||||
description: "SGLang Docker development: VSCode dev container, remote tunnels, debugger setup, nsys profiling."
|
||||
---
|
||||
## Setup VSCode on a Remote Host
|
||||
(Optional - you can skip this step if you plan to run sglang dev container locally)
|
||||
|
||||
1. In the remote host, download `code` from [Https://code.visualstudio.com/docs/?dv=linux64cli](https://code.visualstudio.com/download) and run `code tunnel` in a shell.
|
||||
|
||||
Example
|
||||
```bash Command
|
||||
wget https://vscode.download.prss.microsoft.com/dbazure/download/stable/fabdb6a30b49f79a7aba0f2ad9df9b399473380f/vscode_cli_alpine_x64_cli.tar.gz
|
||||
tar xf vscode_cli_alpine_x64_cli.tar.gz
|
||||
|
||||
# https://code.visualstudio.com/docs/remote/tunnels
|
||||
./code tunnel
|
||||
```
|
||||
|
||||
2. In your local machine, press F1 in VSCode and choose "Remote Tunnels: Connect to Tunnel".
|
||||
|
||||
## Setup Docker Container
|
||||
|
||||
### Option 1. Use the default dev container automatically from VSCode
|
||||
There is a `.devcontainer` folder in the sglang repository root folder to allow VSCode to automatically start up within dev container. You can read more about this VSCode extension in VSCode official document [Developing inside a Container](https://code.visualstudio.com/docs/devcontainers/containers).
|
||||
<Frame>
|
||||
<img src="https://github.com/user-attachments/assets/6a245da8-2d4d-4ea8-8db1-5a05b3a66f6d" alt="VSCode Dev Container Architecture" />
|
||||
</Frame>
|
||||
|
||||
*Figure 1: Diagram from VSCode official documentation [Developing inside a Container](https://code.visualstudio.com/docs/devcontainers/containers).*
|
||||
|
||||
To enable this, you only need to:
|
||||
1. Start Visual Studio Code and install [VSCode dev container extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers).
|
||||
2. Press F1, type and choose "Dev Container: Open Folder in Container.
|
||||
3. Input the `sglang` local repo path in your machine and press enter.
|
||||
|
||||
The first time you open it in dev container might take longer due to docker pull and build. Once it's successful, you should set on your status bar at the bottom left displaying that you are in a dev container:
|
||||
|
||||
<Frame>
|
||||
<img src="https://github.com/user-attachments/assets/650bba0b-c023-455f-91f9-ab357340106b" alt="VSCode Dev Container Status Bar" />
|
||||
</Frame>
|
||||
|
||||
Now when you run `sglang.launch_server` in the VSCode terminal or start debugging using F5, sglang server will be started in the dev container with all your local changes applied automatically:
|
||||
|
||||
<Frame>
|
||||
<img src="https://github.com/user-attachments/assets/748c85ba-7f8c-465e-8599-2bf7a8dde895" alt="SGLang Server Running in Dev Container" />
|
||||
</Frame>
|
||||
|
||||
|
||||
### Option 2. Start up containers manually (advanced)
|
||||
|
||||
The following startup command is an example for internal development by the SGLang team. You can **modify or add directory mappings as needed**, especially for model weight downloads, to prevent repeated downloads by different Docker containers.
|
||||
|
||||
❗️ **Note on RDMA**
|
||||
|
||||
1. `--network host` and `--privileged` are required by RDMA. If you don't need RDMA, you can remove them but keeping them there does not harm. Thus, we enable these two flags by default in the commands below.
|
||||
2. You may need to set `NCCL_IB_GID_INDEX` if you are using RoCE, for example: `export NCCL_IB_GID_INDEX=3`.
|
||||
|
||||
```bash Command
|
||||
# Change the name to yours
|
||||
docker run -itd --shm-size 32g --gpus all -v <volumes-to-mount> --ipc=host --network=host --privileged --name sglang_dev lmsysorg/sglang:dev /bin/zsh
|
||||
docker exec -it sglang_dev /bin/zsh
|
||||
```
|
||||
Some useful volumes to mount are:
|
||||
1. **Huggingface model cache**: mounting model cache can avoid re-download every time docker restarts. Default location on Linux is `~/.cache/huggingface/`.
|
||||
2. **SGLang repository**: code changes in the SGLang local repository will be automatically synced to the .devcontainer.
|
||||
|
||||
Example 1: Monting local cache folder `/opt/dlami/nvme/.cache` but not the SGLang repo. Use this when you prefer to manually transfer local code changes to the devcontainer.
|
||||
```bash Command
|
||||
docker run -itd --shm-size 32g --gpus all -v /opt/dlami/nvme/.cache:/root/.cache --ipc=host --network=host --privileged --name sglang_zhyncs lmsysorg/sglang:dev /bin/zsh
|
||||
docker exec -it sglang_zhyncs /bin/zsh
|
||||
```
|
||||
Example 2: Mounting both HuggingFace cache and local SGLang repo. Local code changes are automatically synced to the devcontainer as the SGLang is installed in editable mode in the dev image.
|
||||
```bash Command
|
||||
docker run -itd --shm-size 32g --gpus all -v $HOME/.cache/huggingface/:/root/.cache/huggingface -v $HOME/src/sglang:/sgl-workspace/sglang --ipc=host --network=host --privileged --name sglang_zhyncs lmsysorg/sglang:dev /bin/zsh
|
||||
docker exec -it sglang_zhyncs /bin/zsh
|
||||
```
|
||||
## Debug SGLang with VSCode Debugger
|
||||
1. (Create if not exist) open `launch.json` in VSCode.
|
||||
2. Add the following config and save. Please note that you can edit the script as needed to apply different parameters or debug a different program (e.g. benchmark script).
|
||||
```JSON Config
|
||||
{
|
||||
"version": "0.2.0",
|
||||
"configurations": [
|
||||
{
|
||||
"name": "Python Debugger: launch_server",
|
||||
"type": "debugpy",
|
||||
"request": "launch",
|
||||
"module": "sglang.launch_server",
|
||||
"console": "integratedTerminal",
|
||||
"args": [
|
||||
"--model-path", "meta-llama/Llama-3.2-1B",
|
||||
"--host", "0.0.0.0",
|
||||
"--port", "30000",
|
||||
"--trust-remote-code",
|
||||
],
|
||||
"justMyCode": false
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
3. Press "F5" to start. VSCode debugger will ensure that the program will pause at the breakpoints even if the program is running at remote SSH/Tunnel host + dev container.
|
||||
|
||||
## Profile
|
||||
|
||||
```bash Command
|
||||
# Change batch size, input, output and add `disable-cuda-graph` (for easier analysis)
|
||||
# e.g. DeepSeek V3
|
||||
nsys profile -o deepseek_v3 python3 -m sglang.bench_one_batch --batch-size 1 --input 128 --output 256 --model deepseek-ai/DeepSeek-V3 --trust-remote-code --tp 8 --disable-cuda-graph
|
||||
```
|
||||
|
||||
## Evaluation
|
||||
|
||||
```bash Command
|
||||
# e.g. gsm8k 8 shot
|
||||
python3 benchmark/gsm8k/bench_sglang.py --num-questions 2000 --parallel 2000 --num-shots 8
|
||||
```
|
||||
@@ -0,0 +1,149 @@
|
||||
---
|
||||
title: "Evaluating New Models with SGLang"
|
||||
metatags:
|
||||
description: "SGLang model evaluation: MMLU, GSM8K, GPQA, HumanEval, MMMU benchmarks. Latency and throughput testing commands."
|
||||
---
|
||||
This document provides commands for evaluating models' accuracy and performance. Before open-sourcing new models, we strongly suggest running these commands to verify whether the score matches your internal benchmark results.
|
||||
|
||||
**For cross verification, please submit commands for installation, server launching, and benchmark running with all the scores and hardware requirements when open-sourcing your models.**
|
||||
|
||||
[Reference: MiniMax M2](https://github.com/sgl-project/sglang/pull/12129)
|
||||
|
||||
## Accuracy
|
||||
|
||||
### LLMs
|
||||
|
||||
SGLang provides built-in scripts to evaluate common benchmarks.
|
||||
|
||||
**MMLU**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.test.run_eval \
|
||||
--eval-name mmlu \
|
||||
--port 30000 \
|
||||
--num-examples 1000 \
|
||||
--max-tokens 8192
|
||||
```
|
||||
|
||||
**GSM8K**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.test.few_shot_gsm8k \
|
||||
--host http://127.0.0.1 \
|
||||
--port 30000 \
|
||||
--num-questions 200 \
|
||||
--num-shots 5
|
||||
```
|
||||
|
||||
**HellaSwag**
|
||||
|
||||
```bash Command
|
||||
python benchmark/hellaswag/bench_sglang.py \
|
||||
--host http://127.0.0.1 \
|
||||
--port 30000 \
|
||||
--num-questions 200 \
|
||||
--num-shots 20
|
||||
```
|
||||
|
||||
**GPQA**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.test.run_eval \
|
||||
--eval-name gpqa \
|
||||
--port 30000 \
|
||||
--num-examples 198 \
|
||||
--max-tokens 120000 \
|
||||
--repeat 8
|
||||
```
|
||||
|
||||
<Tip>
|
||||
For reasoning models, add `--thinking-mode <mode>` (e.g., `qwen3`, `deepseek-r1`, `deepseek-v3`). You may skip it if the model has forced thinking enabled.
|
||||
</Tip>
|
||||
|
||||
**HumanEval**
|
||||
|
||||
```bash Command
|
||||
pip install human_eval
|
||||
|
||||
python -m sglang.test.run_eval \
|
||||
--eval-name humaneval \
|
||||
--num-examples 10 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### VLMs
|
||||
|
||||
**MMMU**
|
||||
|
||||
```bash Command
|
||||
python benchmark/mmmu/bench_sglang.py \
|
||||
--port 30000 \
|
||||
--concurrency 64
|
||||
```
|
||||
|
||||
<Tip>
|
||||
You can set max tokens by passing `--extra-request-body '{"max_tokens": 4096}'`.
|
||||
</Tip>
|
||||
|
||||
For models capable of processing video, we recommend extending the evaluation to include `VideoMME`, `MVBench`, and other relevant benchmarks.
|
||||
|
||||
## Performance
|
||||
|
||||
Performance benchmarks measure **Latency** (Time To First Token - TTFT) and **Throughput** (tokens/second).
|
||||
|
||||
### LLMs
|
||||
|
||||
**Latency-Sensitive Benchmark**
|
||||
|
||||
This simulates a scenario with low concurrency (e.g., single user) to measure latency.
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--dataset-name random \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
**Throughput-Sensitive Benchmark**
|
||||
|
||||
This simulates a high-traffic scenario to measure maximum system throughput.
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--dataset-name random \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
**Single Batch Performance**
|
||||
|
||||
You can also benchmark the performance of processing a single batch offline.
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_one_batch_server \
|
||||
--model <model-path> \
|
||||
--batch-size 8 \
|
||||
--input-len 1024 \
|
||||
--output-len 1024
|
||||
```
|
||||
|
||||
You can run more granular benchmarks:
|
||||
|
||||
- **Low Concurrency**: `--num-prompts 10 --max-concurrency 1`
|
||||
- **Medium Concurrency**: `--num-prompts 80 --max-concurrency 16`
|
||||
- **High Concurrency**: `--num-prompts 500 --max-concurrency 100`
|
||||
|
||||
## Reporting Results
|
||||
|
||||
For each evaluation, please report:
|
||||
|
||||
1. **Metric Score**: Accuracy % (LLMs and VLMs); Latency (ms) and Throughput (tok/s) (LLMs only).
|
||||
2. **Environment settings**: GPU type/count, SGLang commit hash.
|
||||
3. **Launch configuration**: Model path, TP size, and any special flags.
|
||||
4. **Evaluation parameters**: Number of shots, examples, max tokens.
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
title: Developer Guide
|
||||
description: Contributing to SGLang — development setup, benchmarking, and evaluation.
|
||||
---
|
||||
|
||||
- [Contribution Guide](./contribution_guide)
|
||||
- [Development Guide (Docker)](./development_guide_using_docker)
|
||||
- [JIT Kernels](./JIT_kernels)
|
||||
- [Benchmark and Profiling](./benchmark_and_profiling)
|
||||
- [Bench Serving](./bench_serving)
|
||||
- [Evaluating New Models](./evaluating_new_models)
|
||||
@@ -0,0 +1,21 @@
|
||||
---
|
||||
title: "PyPI Package Release Process"
|
||||
metatags:
|
||||
description: "SGLang PyPI release: version update, upload_pypi.sh script, GitHub release creation."
|
||||
---
|
||||
## Update the version in code
|
||||
Update the package version in `python/pyproject.toml` and `python/sglang/__init__.py`.
|
||||
|
||||
## Upload the PyPI package
|
||||
|
||||
```text Output
|
||||
pip install build twine
|
||||
```
|
||||
|
||||
```text Output
|
||||
cd python
|
||||
bash upload_pypi.sh
|
||||
```
|
||||
|
||||
## Make a release in GitHub
|
||||
Make a new release https://github.com/sgl-project/sglang/releases/new.
|
||||
@@ -0,0 +1,54 @@
|
||||
---
|
||||
title: "Set Up Self-Hosted Runners for GitHub Action"
|
||||
metatags:
|
||||
description: "SGLang GitHub Actions self-hosted runner: Docker setup for NVIDIA/AMD GPUs, config.sh and run.sh."
|
||||
---
|
||||
## Add a Runner
|
||||
|
||||
### Step 1: Start a docker container.
|
||||
|
||||
**You can mount a folder for the shared huggingface model weights cache. **
|
||||
The command below uses `/tmp/huggingface` as an example.
|
||||
|
||||
```text Output
|
||||
docker pull nvidia/cuda:12.9.1-devel-ubuntu22.04
|
||||
# Nvidia
|
||||
docker run --shm-size 128g -it -v /tmp/huggingface:/hf_home --gpus all nvidia/cuda:12.9.1-devel-ubuntu22.04 /bin/bash
|
||||
# AMD
|
||||
docker run --rm --device=/dev/kfd --device=/dev/dri --group-add video --shm-size 128g -it -v /tmp/huggingface:/hf_home lmsysorg/sglang:v0.5.0rc1-rocm630 /bin/bash
|
||||
# AMD just the last 2 GPUs
|
||||
docker run --rm --device=/dev/kfd --device=/dev/dri/renderD176 --device=/dev/dri/renderD184 --group-add video --shm-size 128g -it -v /tmp/huggingface:/hf_home lmsysorg/sglang:v0.5.0rc1-rocm630 /bin/bash
|
||||
```
|
||||
|
||||
### Step 2: Configure the runner by `config.sh`
|
||||
|
||||
Run these commands inside the container.
|
||||
|
||||
```text Output
|
||||
apt update && apt install -y curl python3-pip git
|
||||
pip install --upgrade pip
|
||||
export RUNNER_ALLOW_RUNASROOT=1
|
||||
```
|
||||
|
||||
Then follow https://github.com/sgl-project/sglang/settings/actions/runners/new?arch=x64&os=linux to run `config.sh`
|
||||
|
||||
**Notes**
|
||||
- Do not need to specify the runner group
|
||||
- Give it a name (e.g., `test-sgl-gpu-0`) and some labels (e.g., `1-gpu-runner`). The labels can be edited later in Github Settings.
|
||||
- Do not need to change the work folder.
|
||||
|
||||
### Step 3: Run the runner by `run.sh`
|
||||
|
||||
- Set up environment variables
|
||||
```text Output
|
||||
export HF_HOME=/hf_home
|
||||
export SGLANG_IS_IN_CI=true
|
||||
export HF_TOKEN=hf_xxx
|
||||
export OPENAI_API_KEY=sk-xxx
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
```
|
||||
|
||||
- Run it forever
|
||||
```text Output
|
||||
while true; do ./run.sh; echo "Restarting..."; sleep 2; done
|
||||
```
|
||||
@@ -0,0 +1,256 @@
|
||||
---
|
||||
title: Installation
|
||||
description: Install SGLang with pip/uv, source, Docker, Kubernetes, and cloud deployment options.
|
||||
keywords:
|
||||
- installation
|
||||
- sglang
|
||||
- pip
|
||||
- docker
|
||||
---
|
||||
|
||||
You can install SGLang using one of the methods below.
|
||||
This page primarily applies to common NVIDIA GPU platforms.
|
||||
For other or newer platforms, please refer to the dedicated pages for [AMD GPUs](../hardware-platforms/amd-gpus), [Intel Xeon CPUs](../hardware-platforms/cpu-server), [Google TPU](../hardware-platforms/tpu), [NVIDIA DGX Spark](https://lmsys.org/blog/2025-11-03-gpt-oss-on-nvidia-dgx-spark/), [NVIDIA Jetson](../hardware-platforms/nvidia), [Ascend NPUs](../hardware-platforms/ascend-npus/SGLang-installation-with-NPUs-support), and [Intel XPU](../hardware-platforms/xpu).
|
||||
|
||||
<a id="install-methods"></a>
|
||||
## Install methods
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Pip or uv">
|
||||
It is recommended to use <Tooltip tip="A fast Python package manager.">uv</Tooltip> for faster installation:
|
||||
|
||||
```bash
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
uv pip install "sglang"
|
||||
```
|
||||
|
||||
### Quick fixes to common problems
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Wrong torch version">
|
||||
In some cases (for example, GB200), the command above might install a wrong torch version (for example, the CPU version) due to dependency resolution. Reinstall the correct [PyTorch](https://pytorch.org/get-started/locally/) with the following:
|
||||
|
||||
```bash
|
||||
uv pip install "torch" "torchvision" --extra-index-url https://download.pytorch.org/whl/cu129 --force-reinstall
|
||||
```
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="CUDA 13 without Docker">
|
||||
If you do not have Docker access, install the matching `sgl_kernel` wheel from [the sgl-project whl releases](https://github.com/sgl-project/whl/releases) after installing SGLang. Replace `X.Y.Z` with the `sgl_kernel` version required by your SGLang (you can find this by running `uv pip show sgl_kernel`).
|
||||
|
||||
**x86_64**
|
||||
|
||||
```bash
|
||||
uv pip install "https://github.com/sgl-project/whl/releases/download/vX.Y.Z/sgl_kernel-X.Y.Z+cu130-cp310-abi3-manylinux2014_x86_64.whl"
|
||||
```
|
||||
|
||||
**aarch64**
|
||||
|
||||
```bash
|
||||
uv pip install "https://github.com/sgl-project/whl/releases/download/vX.Y.Z/sgl_kernel-X.Y.Z+cu130-cp310-abi3-manylinux2014_aarch64.whl"
|
||||
```
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="CUDA_HOME not set">
|
||||
Choose one of the following solutions:
|
||||
|
||||
1. Set `CUDA_HOME` to your CUDA install root:
|
||||
|
||||
```bash
|
||||
export CUDA_HOME=/usr/local/cuda-<your-cuda-version>
|
||||
```
|
||||
|
||||
2. Install FlashInfer first following the [FlashInfer installation doc](https://docs.flashinfer.ai/installation.html), then install SGLang as described above.
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
</Tab>
|
||||
|
||||
<Tab title="From source">
|
||||
```bash
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
pip install --upgrade pip
|
||||
pip install -e "python"
|
||||
```
|
||||
|
||||
### Quick fixes to common problems
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Development setup">
|
||||
If you want to develop SGLang, try the dev docker image. Refer to [setup docker container](../developer_guide/development_guide_using_docker#setup-docker-container). The docker image is `lmsysorg/sglang:dev`.
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
The docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from [Dockerfile](https://github.com/sgl-project/sglang/tree/main/docker).
|
||||
Replace `<secret>` below with your huggingface hub [token](https://huggingface.co/docs/hub/en/security-tokens).
|
||||
|
||||
**Standard image**
|
||||
|
||||
```bash
|
||||
docker run --gpus all \
|
||||
--shm-size 32g \
|
||||
-p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<secret>" \
|
||||
--ipc=host \
|
||||
lmsysorg/sglang:latest \
|
||||
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000
|
||||
```
|
||||
|
||||
**Runtime image for production**
|
||||
|
||||
```bash
|
||||
docker run --gpus all \
|
||||
--shm-size 32g \
|
||||
-p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<secret>" \
|
||||
--ipc=host \
|
||||
lmsysorg/sglang:latest-runtime \
|
||||
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000
|
||||
```
|
||||
|
||||
You can also find the nightly docker images [here](https://hub.docker.com/r/lmsysorg/sglang/tags?name=nightly).
|
||||
|
||||
<Note>
|
||||
On B300/GB300 (SM103) or CUDA 13 environment, use the nightly image at `lmsysorg/sglang:dev-cu13` or stable image at `lmsysorg/sglang:latest-cu130-runtime`. Do not re-install the project as editable inside the docker image, since it will override the version of libraries specified by the cu13 docker image.
|
||||
</Note>
|
||||
</Tab>
|
||||
|
||||
<Tab title="Kubernetes">
|
||||
Please check out [OME](https://github.com/sgl-project/ome), a Kubernetes operator for enterprise-grade management and serving of large language models (LLMs).
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Single node serving">
|
||||
For models that fit into GPUs on one node, create the deployment and service with llama-31-8b as example.
|
||||
|
||||
```bash
|
||||
kubectl apply -f docker/k8s-sglang-service.yaml
|
||||
```
|
||||
</Tab>
|
||||
|
||||
<Tab title="Multi-node serving">
|
||||
For larger models (for example, `DeepSeek-R1`), modify the model path and arguments, then create the statefulset and service.
|
||||
|
||||
```bash
|
||||
kubectl apply -f docker/k8s-sglang-distributed-sts.yaml
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker Compose">
|
||||
<Note>
|
||||
This method is recommended if you plan to serve it as a service. A better approach is to use the [k8s-sglang-service.yaml](https://github.com/sgl-project/sglang/blob/main/docker/k8s-sglang-service.yaml).
|
||||
</Note>
|
||||
|
||||
1. Copy the [compose.yml](https://github.com/sgl-project/sglang/blob/main/docker/compose.yaml) to your local machine.
|
||||
2. Start the service:
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
</Tab>
|
||||
|
||||
<Tab title="SkyPilot">
|
||||
To deploy on Kubernetes or 12+ clouds, you can use [SkyPilot](https://github.com/skypilot-org/skypilot).
|
||||
|
||||
1. Install SkyPilot and set up Kubernetes cluster or cloud access. See [SkyPilot's documentation](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html).
|
||||
2. Deploy on your own infra with a single command and get the HTTP API endpoint:
|
||||
|
||||
**SkyPilot YAML: `sglang.yaml`**
|
||||
|
||||
```yaml Config
|
||||
# sglang.yaml
|
||||
envs:
|
||||
HF_TOKEN: null
|
||||
|
||||
resources:
|
||||
image_id: docker:lmsysorg/sglang:latest
|
||||
accelerators: A100
|
||||
ports: 30000
|
||||
|
||||
run: |
|
||||
conda deactivate
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
```bash
|
||||
# Deploy on any cloud or Kubernetes cluster. Use --cloud <cloud> to select a specific cloud provider.
|
||||
HF_TOKEN=<secret> sky launch -c sglang --env HF_TOKEN sglang.yaml
|
||||
|
||||
# Get the HTTP API endpoint
|
||||
sky status --endpoint 30000 sglang
|
||||
```
|
||||
|
||||
3. To scale with autoscaling and failure recovery, check out the [SkyServe + SGLang guide](https://github.com/skypilot-org/skypilot/tree/master/llm/sglang#serving-llama-2-with-sglang-for-more-traffic-using-skyserve).
|
||||
</Tab>
|
||||
|
||||
<Tab title="AWS SageMaker">
|
||||
To deploy on SGLang on AWS SageMaker, check out [AWS SageMaker Inference](https://aws.amazon.com/sagemaker/ai/deploy).
|
||||
|
||||
Amazon Web Services provide supports for SGLang containers along with routine security patching. For available SGLang containers, check out [AWS SGLang DLCs](https://github.com/aws/deep-learning-containers/blob/master/available_images.md#sglang-containers).
|
||||
|
||||
To host a model with your own container, follow the following steps:
|
||||
|
||||
1. Build a docker container with [sagemaker.Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/sagemaker.Dockerfile) alongside the [serve](https://github.com/sgl-project/sglang/blob/main/docker/serve) script, then push it to AWS ECR.
|
||||
|
||||
**Dockerfile build script: `build-and-push.sh`**
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
AWS_ACCOUNT="<YOUR_AWS_ACCOUNT>"
|
||||
AWS_REGION="<YOUR_AWS_REGION>"
|
||||
REPOSITORY_NAME="<YOUR_REPOSITORY_NAME>"
|
||||
IMAGE_TAG="<YOUR_IMAGE_TAG>"
|
||||
|
||||
ECR_REGISTRY="${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com"
|
||||
IMAGE_URI="${ECR_REGISTRY}/${REPOSITORY_NAME}:${IMAGE_TAG}"
|
||||
|
||||
echo "Starting build and push process..."
|
||||
|
||||
# Login to ECR
|
||||
echo "Logging into ECR..."
|
||||
aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${ECR_REGISTRY}
|
||||
|
||||
# Build the image
|
||||
echo "Building Docker image..."
|
||||
docker build -t ${IMAGE_URI} -f sagemaker.Dockerfile .
|
||||
|
||||
echo "Pushing ${IMAGE_URI}"
|
||||
docker push ${IMAGE_URI}
|
||||
|
||||
echo "Build and push completed successfully!"
|
||||
```
|
||||
|
||||
2. Deploy a model for serving on AWS Sagemaker. Refer to [deploy_and_serve_endpoint.py](https://github.com/sgl-project/sglang/blob/main/examples/sagemaker/deploy_and_serve_endpoint.py). For more information, check out [sagemaker-python-sdk](https://github.com/aws/sagemaker-python-sdk).
|
||||
|
||||
**Default command**
|
||||
|
||||
The model server on SageMaker runs: `python3 -m sglang.launch_server --model-path opt/ml/model --host 0.0.0.0 --port 8080`.
|
||||
|
||||
**Custom arguments**
|
||||
|
||||
The [serve](https://github.com/sgl-project/sglang/blob/main/docker/serve) script exposes all options in `python3 -m sglang.launch_server --help` through environment variables prefixed with `SM_SGLANG_`.
|
||||
|
||||
**Environment variable mapping**
|
||||
|
||||
The serve script converts variables with prefix `SM_SGLANG_` from `SM_SGLANG_INPUT_ARGUMENT` into `--input-argument` for the `python3 -m sglang.launch_server` CLI.
|
||||
|
||||
**Example**
|
||||
|
||||
To run [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) with reasoning parser, add `SM_SGLANG_MODEL_PATH=Qwen/Qwen3-0.6B` and `SM_SGLANG_REASONING_PARSER=qwen3`.
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
## Common notes
|
||||
|
||||
- [FlashInfer](https://github.com/flashinfer-ai/flashinfer) is the default attention kernel backend. It only supports sm75 and above. If you encounter any FlashInfer-related issues on sm75+ devices (for example, T4, A10, A100, L4, L40S, H100), switch to other kernels by adding `--attention-backend triton --sampling-backend pytorch` and open an issue on GitHub.
|
||||
- To reinstall flashinfer locally, use the following command: `pip3 install --upgrade flashinfer-python --force-reinstall --no-deps` and then delete the cache with `rm -rf ~/.cache/flashinfer`.
|
||||
- When encountering `ptxas fatal : Value 'sm_103a' is not defined for option 'gpu-name'` on B300/GB300, fix it with `export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas`.
|
||||
@@ -0,0 +1,332 @@
|
||||
---
|
||||
title: "Quickstart"
|
||||
description: "Get up and running with SGLang in minutes: install, launch a server, and send your first request."
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This guide walks you through the entire flow of getting started with SGLang:
|
||||
|
||||
1. **Install** SGLang
|
||||
2. **Launch** an inference server
|
||||
3. **Send requests** using cURL, OpenAI Python client, Python `requests`, or the native SGLang API
|
||||
|
||||
By the end, you'll have a working SGLang server responding to your prompts.
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Python**: 3.9 or higher
|
||||
- **GPU**: NVIDIA GPU with CUDA support (sm75 and above, e.g., T4, A10, A100, L4, L40S, H100)
|
||||
- **OS**: Linux (recommended)
|
||||
|
||||
<Note>
|
||||
For other platforms, see the dedicated guides for [AMD GPUs](../hardware-platforms/amd-gpus), [Intel Xeon CPUs](../hardware-platforms/cpu-server), [Google TPUs](../hardware-platforms/tpu), [NVIDIA Jetson](../hardware-platforms/nvidia), [Ascend NPUs](../hardware-platforms/ascend-npus/SGLang-installation-with-NPUs-support), and [Intel XPU](../hardware-platforms/xpu).
|
||||
</Note>
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Pip / uv (Recommended)">
|
||||
We recommend using **uv** for faster installation:
|
||||
|
||||
```bash
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
uv pip install sglang
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="From Source">
|
||||
```bash
|
||||
# Clone and install from source
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
pip install --upgrade pip
|
||||
pip install -e "python"
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Docker">
|
||||
The Docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags).
|
||||
|
||||
Replace `<secret>` with your [Hugging Face token](https://huggingface.co/docs/hub/en/security-tokens):
|
||||
|
||||
```bash
|
||||
docker run --gpus all \
|
||||
--shm-size 32g \
|
||||
-p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<secret>" \
|
||||
--ipc=host \
|
||||
lmsysorg/sglang:latest \
|
||||
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000
|
||||
```
|
||||
|
||||
For production deployments, use the smaller **runtime** variant (~40% size reduction):
|
||||
|
||||
```bash
|
||||
docker run --gpus all \
|
||||
--shm-size 32g \
|
||||
-p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<secret>" \
|
||||
--ipc=host \
|
||||
lmsysorg/sglang:latest-runtime \
|
||||
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
<Tip>
|
||||
If you encounter `OSError: CUDA_HOME environment variable is not set`, set it with:
|
||||
```bash
|
||||
export CUDA_HOME=/usr/local/cuda-<your-cuda-version>
|
||||
```
|
||||
</Tip>
|
||||
|
||||
---
|
||||
|
||||
## Launch a Server
|
||||
|
||||
Start the SGLang server with a model. Here we use `qwen/qwen2.5-0.5b-instruct` as a lightweight example:
|
||||
|
||||
```bash
|
||||
python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --port 30000
|
||||
```
|
||||
|
||||
Wait until you see `The server is fired up and ready to roll!` in the terminal output.
|
||||
|
||||
<Note>
|
||||
Once the server is running, API documentation is available at:
|
||||
- **Swagger UI**: `http://localhost:30000/docs`
|
||||
- **ReDoc**: `http://localhost:30000/redoc`
|
||||
- **OpenAPI Spec**: `http://localhost:30000/openapi.json`
|
||||
</Note>
|
||||
|
||||
<Info>
|
||||
The server automatically applies the chat template from the Hugging Face tokenizer. You can override it with `--chat-template` when launching.
|
||||
</Info>
|
||||
|
||||
---
|
||||
|
||||
## Send Requests
|
||||
|
||||
SGLang is fully **OpenAI API-compatible**, so you can use the same tools and libraries you already know.
|
||||
|
||||
### Using cURL
|
||||
|
||||
```bash
|
||||
curl http://localhost:30000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "qwen/qwen2.5-0.5b-instruct",
|
||||
"messages": [
|
||||
{"role": "user", "content": "What is the capital of France?"}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
### Using OpenAI Python Client
|
||||
|
||||
Install the OpenAI Python library if you haven't:
|
||||
|
||||
```bash
|
||||
pip install openai
|
||||
```
|
||||
|
||||
Then send a request:
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="qwen/qwen2.5-0.5b-instruct",
|
||||
messages=[
|
||||
{"role": "user", "content": "List 3 countries and their capitals."},
|
||||
],
|
||||
temperature=0,
|
||||
max_tokens=64,
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
#### Streaming
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="qwen/qwen2.5-0.5b-instruct",
|
||||
messages=[
|
||||
{"role": "user", "content": "List 3 countries and their capitals."},
|
||||
],
|
||||
temperature=0,
|
||||
max_tokens=64,
|
||||
stream=True,
|
||||
)
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices[0].delta.content:
|
||||
print(chunk.choices[0].delta.content, end="", flush=True)
|
||||
```
|
||||
|
||||
### Using Python Requests
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
url = "http://localhost:30000/v1/chat/completions"
|
||||
|
||||
data = {
|
||||
"model": "qwen/qwen2.5-0.5b-instruct",
|
||||
"messages": [{"role": "user", "content": "What is the capital of France?"}],
|
||||
}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
print(response.json())
|
||||
```
|
||||
|
||||
### Using the Native `/generate` API
|
||||
|
||||
SGLang also provides a native `/generate` endpoint for more flexibility.
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
response = requests.post(
|
||||
"http://localhost:30000/generate",
|
||||
json={
|
||||
"text": "The capital of France is",
|
||||
"sampling_params": {
|
||||
"temperature": 0,
|
||||
"max_new_tokens": 32,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
print(response.json())
|
||||
```
|
||||
|
||||
#### Streaming with `/generate`
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
import json
|
||||
|
||||
response = requests.post(
|
||||
"http://localhost:30000/generate",
|
||||
json={
|
||||
"text": "The capital of France is",
|
||||
"sampling_params": {
|
||||
"temperature": 0,
|
||||
"max_new_tokens": 32,
|
||||
},
|
||||
"stream": True,
|
||||
},
|
||||
stream=True,
|
||||
)
|
||||
|
||||
prev = 0
|
||||
for chunk in response.iter_lines(decode_unicode=False):
|
||||
chunk = chunk.decode("utf-8")
|
||||
if chunk and chunk.startswith("data:"):
|
||||
if chunk == "data: [DONE]":
|
||||
break
|
||||
data = json.loads(chunk[5:].strip("\n"))
|
||||
output = data["text"]
|
||||
print(output[prev:], end="", flush=True)
|
||||
prev = len(output)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Offline Batch Inference (No Server)
|
||||
|
||||
SGLang also supports offline batch inference using the `Engine` class directly -- no HTTP server required.
|
||||
|
||||
```python Example
|
||||
import sglang as sgl
|
||||
|
||||
llm = sgl.Engine(model_path="qwen/qwen2.5-0.5b-instruct")
|
||||
|
||||
prompts = [
|
||||
"Hello, my name is",
|
||||
"The president of the United States is",
|
||||
"The capital of France is",
|
||||
"The future of AI is",
|
||||
]
|
||||
|
||||
sampling_params = {"temperature": 0.8, "top_p": 0.95}
|
||||
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
|
||||
for prompt, output in zip(prompts, outputs):
|
||||
print(f"Prompt: {prompt}\nGenerated text: {output['text']}\n")
|
||||
|
||||
llm.shutdown()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Troubleshooting
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="CUDA_HOME not set">
|
||||
Set the `CUDA_HOME` environment variable to your CUDA install root:
|
||||
```bash
|
||||
export CUDA_HOME=/usr/local/cuda-<your-cuda-version>
|
||||
```
|
||||
</Accordion>
|
||||
<Accordion title="FlashInfer issues on sm75+ devices">
|
||||
Switch to alternative backends by adding these flags when launching the server:
|
||||
```bash
|
||||
--attention-backend triton --sampling-backend pytorch
|
||||
```
|
||||
</Accordion>
|
||||
<Accordion title="Reinstalling FlashInfer">
|
||||
```bash
|
||||
pip3 install --upgrade flashinfer-python --force-reinstall --no-deps
|
||||
rm -rf ~/.cache/flashinfer
|
||||
```
|
||||
</Accordion>
|
||||
<Accordion title="ptxas error on B300/GB300 (sm_103a)">
|
||||
```bash
|
||||
export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas
|
||||
```
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
---
|
||||
|
||||
{/*
|
||||
WIP, TBD linked later
|
||||
## What's Next?
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="OpenAI-Compatible APIs" href="/basic_usage/openai_api_completions">
|
||||
Explore the full Chat Completions and Completions APIs, including multi-turn conversations.
|
||||
</Card>
|
||||
<Card title="Vision Language Models" href="/basic_usage/openai_api_vision">
|
||||
Send image inputs alongside text using OpenAI-compatible vision APIs.
|
||||
</Card>
|
||||
<Card title="Sampling Parameters" href="/basic_usage/sampling_params">
|
||||
Fine-tune generation with temperature, top-p, frequency penalty, and more.
|
||||
</Card>
|
||||
<Card title="Server Arguments" href="/advanced_features/server_arguments">
|
||||
Customize server behavior with advanced launch arguments like tensor parallelism.
|
||||
</Card>
|
||||
<Card title="Structured Outputs" href="/advanced_features/structured_outputs">
|
||||
Constrain model output to JSON, regex, or EBNF grammars.
|
||||
</Card>
|
||||
<Card title="Ollama-Compatible API" href="/basic_usage/ollama_api">
|
||||
Use the familiar Ollama CLI and Python library with SGLang as the backend.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
*/}
|
||||
@@ -0,0 +1,194 @@
|
||||
---
|
||||
title: "AMD GPUs"
|
||||
---
|
||||
This document describes how run SGLang on AMD GPUs. If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
|
||||
|
||||
## System Configuration
|
||||
|
||||
When using AMD GPUs (such as MI300X), certain system-level optimizations help ensure stable performance. Here we take MI300X as an example. AMD provides official documentation for MI300X optimization and system tuning:
|
||||
|
||||
* [AMD MI300X Tuning Guides](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/index.html)
|
||||
* [LLM inference performance validation on AMD Instinct MI300X](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference/vllm-benchmark.html)
|
||||
* [AMD Instinct MI300X System Optimization](https://rocm.docs.amd.com/en/latest/how-to/system-optimization/mi300x.html)
|
||||
* [AMD Instinct MI300X Workload Optimization](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/workload.html)
|
||||
* [Supercharge DeepSeek-R1 Inference on AMD Instinct MI300X](https://rocm.blogs.amd.com/artificial-intelligence/DeepSeekR1-Part2/README.html)
|
||||
|
||||
<Note>
|
||||
We strongly recommend reading these docs and guides entirely to fully utilize your system.
|
||||
</Note>
|
||||
|
||||
Below are a few key settings to confirm or enable for SGLang:
|
||||
|
||||
### Update GRUB Settings
|
||||
|
||||
In `/etc/default/grub`, append the following to `GRUB_CMDLINE_LINUX`:
|
||||
|
||||
<CodeGroup>
|
||||
```text GRUB Configuration
|
||||
pci=realloc=off iommu=pt
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Afterward, run `sudo update-grub` (or your distro's equivalent) and reboot.
|
||||
|
||||
### Disable NUMA Auto-Balancing
|
||||
|
||||
<CodeGroup>
|
||||
```bash Disable NUMA
|
||||
sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
You can automate or verify this change using [this helpful script](https://github.com/ROCm/triton/blob/rocm_env/scripts/amd/env_check.sh).
|
||||
|
||||
Again, please go through the entire documentation to confirm your system is using the recommended configuration.
|
||||
|
||||
## Install SGLang
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Docker (Recommended)">
|
||||
The docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from [rocm.Dockerfile](https://github.com/sgl-project/sglang/tree/main/docker).
|
||||
|
||||
1. **Build the docker image**
|
||||
If you use pre-built images, you can skip this step and replace `sglang_image` with the pre-built image names in the steps below.
|
||||
|
||||
<CodeGroup>
|
||||
```bash Build Image
|
||||
docker build -t sglang_image -f rocm.Dockerfile .
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
2. **Create a convenient alias**
|
||||
<CodeGroup>
|
||||
```bash Create Alias
|
||||
alias drun='docker run -it --rm --network=host --privileged --device=/dev/kfd --device=/dev/dri \
|
||||
--ipc=host --shm-size 16G --group-add video --cap-add=SYS_PTRACE \
|
||||
--security-opt seccomp=unconfined \
|
||||
-v $HOME/dockerx:/dockerx \
|
||||
-v /data:/data'
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
If you are using RDMA, please note that:
|
||||
|
||||
* `--network host` and `--privileged` are required by RDMA. If you don't need RDMA, you can remove them.
|
||||
* You may need to set `NCCL_IB_GID_INDEX` if you are using RoCE, for example: `export NCCL_IB_GID_INDEX=3`.
|
||||
|
||||
3. **Launch the server**
|
||||
<Note>
|
||||
Replace `<secret>` below with your [huggingface hub token](https://huggingface.co/docs/hub/en/security-tokens).
|
||||
</Note>
|
||||
|
||||
<CodeGroup>
|
||||
```bash Launch Server
|
||||
drun -p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<secret>" \
|
||||
sglang_image \
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path NousResearch/Meta-Llama-3.1-8B \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
4. **Verify the installation**
|
||||
You can run a benchmark in another terminal or refer to [other docs](../basic_usage/openai_api_completions) to send requests to the engine.
|
||||
|
||||
<CodeGroup>
|
||||
```bash Run Benchmark
|
||||
drun sglang_image \
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--dataset-name random \
|
||||
--num-prompts 4000 \
|
||||
--random-input 128 \
|
||||
--random-output 128
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
With your AMD system properly configured and SGLang installed, you can now fully leverage AMD hardware to power SGLang's machine learning capabilities.
|
||||
</Tab>
|
||||
|
||||
<Tab title="From Source">
|
||||
1. **Clone the repository**
|
||||
Clone the SGLang repository.
|
||||
|
||||
<CodeGroup>
|
||||
```bash
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
2. **Compile sgl-kernel**
|
||||
Upgrade pip and compile the sgl-kernel for ROCm support.
|
||||
|
||||
<CodeGroup>
|
||||
```bash
|
||||
pip install --upgrade pip
|
||||
cd sgl-kernel
|
||||
python setup_rocm.py install
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
3. **Install sglang package**
|
||||
Install the SGLang Python package with HIP and diffusion support.
|
||||
|
||||
<CodeGroup>
|
||||
```bash
|
||||
cd ..
|
||||
rm -rf python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
|
||||
pip install -e "python[all_hip]"
|
||||
```
|
||||
</CodeGroup>
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
## Examples
|
||||
|
||||
### Running DeepSeek-V3
|
||||
|
||||
The only difference when running DeepSeek-V3 is in how you start the server.
|
||||
|
||||
<CodeGroup>
|
||||
```bash DeepSeek-V3
|
||||
drun -p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--ipc=host \
|
||||
--env "HF_TOKEN=<secret>" \
|
||||
sglang_image \
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3 \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
[Running DeepSeek-R1 on a single NDv5 MI300X VM](https://techcommunity.microsoft.com/blog/azurehighperformancecomputingblog/running-deepseek-r1-on-a-single-ndv5-mi300x-vm/4372726) could also be a good reference.
|
||||
|
||||
### Running Llama3.1
|
||||
|
||||
Running Llama3.1 is nearly identical to running DeepSeek-V3. The only difference is in the model specified when starting the server.
|
||||
|
||||
<CodeGroup>
|
||||
```bash Llama3.1
|
||||
drun -p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--ipc=host \
|
||||
--env "HF_TOKEN=<secret>" \
|
||||
sglang_image \
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Warmup Step
|
||||
|
||||
When the server displays `The server is fired up and ready to roll!`, it means the startup is successful.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,309 @@
|
||||
## Running DeepSeek-V3
|
||||
|
||||
### Running DeepSeek in PD mixed mode on 1 x Atlas 800I A3
|
||||
|
||||
W4A8 Model weights could be found [here](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8).
|
||||
|
||||
<CodeGroup>
|
||||
```shell Launch Server
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
|
||||
#Deepep communication settings
|
||||
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
|
||||
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
|
||||
export HCCL_BUFFSIZE=1600
|
||||
|
||||
#spec overlap
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
|
||||
#npu acceleration operator
|
||||
export SGLANG_NPU_USE_MLAPO=1
|
||||
export SGLANG_USE_FIA_NZ=1
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path ${MODEL_PATH} \
|
||||
--tp 16 \
|
||||
--trust-remote-code \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--quantization modelslim \
|
||||
--watchdog-timeout 9000 \
|
||||
--cuda-graph-bs 8 16 24 28 32 \
|
||||
--mem-fraction-static 0.68 \
|
||||
--max-running-requests 128 \
|
||||
--context-length 8188 \
|
||||
--disable-radix-cache \
|
||||
--chunked-prefill-size -1 \
|
||||
--max-prefill-tokens 16384 \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode auto \
|
||||
--enable-dp-attention \
|
||||
--dp-size 4 \
|
||||
--enable-dp-lm-head \
|
||||
--speculative-algorithm NEXTN \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--dtype bfloat16
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Running DeepSeek with PD disaggregation mode on 2 x Atlas 800I A3
|
||||
|
||||
W4A8 Model weights could be found [here](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8).
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Prefill">
|
||||
```shell Command
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
|
||||
#memfabric config store
|
||||
export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:<PORT>"
|
||||
|
||||
#Deepep communication settings
|
||||
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
|
||||
export HCCL_BUFFSIZE=1536
|
||||
|
||||
#npu acceleration operator
|
||||
export SGLANG_NPU_USE_MLAPO=1
|
||||
export SGLANG_USE_FIA_NZ=1
|
||||
export TASK_QUEUE_ENABLE=2
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--model-path ${MODEL_PATH} \
|
||||
--host $PREFILL_HOST_IP \
|
||||
--port 8000 \
|
||||
--disaggregation-mode prefill \
|
||||
--disaggregation-bootstrap-port 8996 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--trust-remote-code \
|
||||
--nnodes 1 \
|
||||
--node-rank 0 \
|
||||
--tp-size 16 \
|
||||
--mem-fraction-static 0.6 \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--quantization modelslim \
|
||||
--load-balance-method round_robin \
|
||||
--max-running-requests 8 \
|
||||
--context-length 8192 \
|
||||
--disable-radix-cache \
|
||||
--chunked-prefill-size -1 \
|
||||
--max-prefill-tokens 28680 \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode normal \
|
||||
--speculative-algorithm NEXTN \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--dp-size 2 \
|
||||
--enable-dp-attention \
|
||||
--disable-shared-experts-fusion \
|
||||
--dtype bfloat16
|
||||
```
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Decode">
|
||||
```shell Command
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
|
||||
#memfabric config store
|
||||
export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:<PORT>"
|
||||
|
||||
#Deepep communication settings
|
||||
export HCCL_BUFFSIZE=720
|
||||
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=88
|
||||
|
||||
#spec overlap
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
|
||||
#npu acceleration operator
|
||||
unset TASK_QUEUE_ENABLE
|
||||
export SGLANG_NPU_USE_MLAPO=1
|
||||
export SGLANG_USE_FIA_NZ=1
|
||||
export ENABLE_MOE_NZ=1
|
||||
|
||||
# suggest max-running-requests <= max-cuda-graph-bs * dp_size, Because when this value is exceeded, performance will significantly degrade.
|
||||
python -m sglang.launch_server \
|
||||
--model-path ${MODEL_PATH} \
|
||||
--disaggregation-mode decode \
|
||||
--host $DECODE_HOST_IP \
|
||||
--port 8001 \
|
||||
--trust-remote-code \
|
||||
--nnodes 1 \
|
||||
--node-rank 0 \
|
||||
--tp-size 16 \
|
||||
--dp-size 16 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--max-running-requests 352 \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--quantization modelslim \
|
||||
--prefill-round-robin-balance \
|
||||
--moe-a2a-backend deepep \
|
||||
--enable-dp-attention \
|
||||
--deepep-mode low_latency \
|
||||
--enable-dp-lm-head \
|
||||
--cuda-graph-bs 8 10 12 14 16 18 20 22 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--watchdog-timeout 9000 \
|
||||
--context-length 8192 \
|
||||
--speculative-algorithm NEXTN \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--disable-shared-experts-fusion \
|
||||
--dtype bfloat16 \
|
||||
--tokenizer-worker-num 4
|
||||
```
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Router">
|
||||
```shell Command
|
||||
python -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--policy cache_aware \
|
||||
--prefill http://<PREFILL_HOST_IP>:8000 8996 \
|
||||
--decode http://<DECODE_HOST_IP>:8001 \
|
||||
--host 127.0.0.1 \
|
||||
--port 6688
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
### Running DeepSeek with PD disaggregation on 4 x Atlas 800I A3
|
||||
|
||||
W8A8 Model weights could be found [here](https://modelers.cn/models/State_Cloud/Deepseek-R1-bf16-hfd-w8a8).
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Prefill & Decode">
|
||||
```shell Command
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
|
||||
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
|
||||
export ASCEND_MF_STORE_URL="tcp://your prefill ip1:24669"
|
||||
|
||||
P_IP=('your prefill ip1' 'your prefill ip2')
|
||||
|
||||
D_IP=('your decode ip1' 'your decode ip2')
|
||||
|
||||
MODEL_PATH=xxx
|
||||
|
||||
export SGLANG_NPU_USE_MLAPO=1
|
||||
export SGLANG_USE_FIA_NZ=1
|
||||
|
||||
LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
|
||||
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
|
||||
echo "${LOCAL_HOST1}"
|
||||
echo "${LOCAL_HOST2}"
|
||||
# prefill
|
||||
for i in "${!P_IP[@]}";
|
||||
do
|
||||
if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
|
||||
then
|
||||
echo "${P_IP[$i]}"
|
||||
export HCCL_BUFFSIZE=1536
|
||||
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
|
||||
export TASK_QUEUE_ENABLE=2
|
||||
|
||||
export HCCL_SOCKET_IFNAME=lo
|
||||
export GLOO_SOCKET_IFNAME=lo
|
||||
python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \
|
||||
--port 8000 --disaggregation-bootstrap-port $((8998+$i)) --trust-remote-code --nnodes 1 --node-rank 0 \
|
||||
--tp-size 16 --mem-fraction-static 0.81 --attention-backend ascend --device npu --quantization modelslim \
|
||||
--disaggregation-transfer-backend ascend --max-running-requests 8 --context-length 8192 --disable-radix-cache \
|
||||
--chunked-prefill-size -1 --max-prefill-tokens 28680 --moe-a2a-backend deepep --deepep-mode normal \
|
||||
--speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \
|
||||
--dp-size 2 --enable-dp-attention --disable-shared-experts-fusion --dtype bfloat16 --enable-attn-tp-input-scattered
|
||||
NODE_RANK=$i
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
# decode
|
||||
for i in "${!D_IP[@]}";
|
||||
do
|
||||
if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
|
||||
then
|
||||
echo "${D_IP[$i]}"
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export HCCL_BUFFSIZE=650
|
||||
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=78
|
||||
export TASK_QUEUE_ENABLE=1
|
||||
export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1
|
||||
export HCCL_SOCKET_IFNAME=xxx
|
||||
export GLOO_SOCKET_IFNAME=xxx
|
||||
python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \
|
||||
--port 8001 --trust-remote-code --dist-init-addr ${D_IP[0]}:5000 --nnodes 2 --node-rank $i --tp-size 32 --dp-size 32 \
|
||||
--mem-fraction-static 0.815 --max-running-requests 832 --attention-backend ascend --device npu --quantization modelslim \
|
||||
--moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency --enable-dp-lm-head --moe-dense-tp 1 \
|
||||
--cuda-graph-bs 12 14 16 18 20 22 24 26 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \
|
||||
--speculative-algorithm NEXTN --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3 \
|
||||
--tokenizer-worker-num 4 --prefill-round-robin-balance --disable-shared-experts-fusion --dtype bfloat16 \
|
||||
--load-balance-method decode_round_robin
|
||||
NODE_RANK=$i
|
||||
break
|
||||
fi
|
||||
done
|
||||
```
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Router">
|
||||
```shell Command
|
||||
export SGLANG_DP_ROUND_ROBIN=1
|
||||
python -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--policy cache_aware \
|
||||
--prefill http://P_IP:8000 8998 \
|
||||
--prefill http://P_IP:8000 8999 \
|
||||
--decode http://D_IP:8001 \
|
||||
--host 127.0.0.1 \
|
||||
--port 6688 \
|
||||
--mini-lb
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
### Test GSM8K
|
||||
|
||||
<CodeGroup>
|
||||
```python Test GSM8K
|
||||
from types import SimpleNamespace
|
||||
from sglang.test.few_shot_gsm8k import run_eval
|
||||
|
||||
def gsm8k():
|
||||
args = SimpleNamespace(
|
||||
num_shots=5,
|
||||
data_path=None,
|
||||
num_questions=200,
|
||||
max_new_tokens=512,
|
||||
parallel=32,
|
||||
host=f"http://127.0.0.1",
|
||||
port=6688,
|
||||
)
|
||||
metrics = run_eval(args)
|
||||
print(f"{metrics=}")
|
||||
print(f"{metrics['accuracy']=}")
|
||||
if __name__ == "__main__":
|
||||
gsm8k()
|
||||
```
|
||||
</CodeGroup>
|
||||
@@ -0,0 +1,200 @@
|
||||
## Introduction
|
||||
|
||||
The GLM (General Language Model) series is an open-source bilingual large language model family jointly developed by the KEG Laboratory of Tsinghua University and Zhipu AI. This series of models has performed outstandingly in the field of Chinese NLP with its unique unified pre-training framework and bilingual capabilities. [GLM-5](https://huggingface.co/zai-org/GLM-5) adopts the DeepSeek-V3/V3.2 architecture, including the sparse attention (DSA) and multi-token prediction (MTP). Ascend supports GLM-5 with 0Day based on the SGLang inference framework, achieving low-code seamless enablement and compatibility with the mainstream distributed parallel capabilities within the current SGLang framework. We welcome developers to download and experience it.
|
||||
|
||||
## Environment Preparation
|
||||
|
||||
### Model Weight
|
||||
|
||||
- `GLM-5.0`(BF16 version): [Download model weight](https://www.modelscope.cn/models/ZhipuAI/GLM-5).
|
||||
- `GLM-5.0-w4a8`(Quantized version without mtp): [Download model weight](https://modelers.cn/models/Eco-Tech/GLM-5-w4a8).
|
||||
- You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantify the model naively.
|
||||
|
||||
|
||||
### Installation
|
||||
|
||||
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the quay.io platform. You can directly pull it.
|
||||
|
||||
<CodeGroup>
|
||||
```bash Pull and Start Container
|
||||
#Atlas 800 A3
|
||||
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann8.5.0-a3-glm5
|
||||
#Atlas 800 A2
|
||||
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann8.5.0-910b-glm5
|
||||
|
||||
#start container
|
||||
docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
|
||||
--privileged=true --net=host \
|
||||
-v /var/queue_schedule:/var/queue_schedule \
|
||||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
-v /usr/local/sbin:/usr/local/sbin \
|
||||
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
|
||||
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
|
||||
--device=/dev/davinci0:/dev/davinci0 \
|
||||
--device=/dev/davinci1:/dev/avinci1 \
|
||||
--device=/dev/davinci2:/dev/davinci2 \
|
||||
--device=/dev/davinci3:/dev/davinci3 \
|
||||
--device=/dev/davinci4:/dev/davinci4 \
|
||||
--device=/dev/davinci5:/dev/davinci5 \
|
||||
--device=/dev/davinci6:/dev/davinci6 \
|
||||
--device=/dev/davinci7:/dev/davinci7 \
|
||||
--device=/dev/davinci8:/dev/davinci8 \
|
||||
--device=/dev/davinci9:/dev/davinci9 \
|
||||
--device=/dev/davinci10:/dev/davinci10 \
|
||||
--device=/dev/davinci11:/dev/davinci11 \
|
||||
--device=/dev/davinci12:/dev/davinci12 \
|
||||
--device=/dev/davinci13:/dev/davinci13 \
|
||||
--device=/dev/davinci14:/dev/davinci14 \
|
||||
--device=/dev/davinci15:/dev/davinci15 \
|
||||
--device=/dev/davinci_manager:/dev/davinci_manager \
|
||||
--device=/dev/hisi_hdc:/dev/hisi_hdc \
|
||||
--entrypoint=bash \
|
||||
swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:${TAG}
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Note: Using this image, you need to update transformers to main branch
|
||||
<CodeGroup>
|
||||
```shell Update Transformers
|
||||
# reinstall transformers
|
||||
pip install git+https://github.com/huggingface/transformers.git
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
## Deployment
|
||||
|
||||
### Single-node Deployment
|
||||
|
||||
- Quantized model `glm5_w4a8` can be deployed on 1 Atlas 800 A3 (64G × 16) .
|
||||
|
||||
Run the following script to execute online inference.
|
||||
|
||||
<CodeGroup>
|
||||
```shell Launch Server
|
||||
# high performance cpu
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
# bind cpu
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
unset https_proxy
|
||||
unset http_proxy
|
||||
unset HTTPS_PROXY
|
||||
unset HTTP_PROXY
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
# cann
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
export SGLANG_NPU_USE_MULTI_STREAM=1
|
||||
export HCCL_BUFFSIZE=1000
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
export HCCL_SOCKET_IFNAME=lo
|
||||
export GLOO_SOCKET_IFNAME=lo
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path $MODEL_PATH \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--tp-size 16 --nnodes 1 --node-rank 0 \
|
||||
--chunked-prefill-size 16384 --max-prefill-tokens 280000 \
|
||||
--trust-remote-code \
|
||||
--host 127.0.0.1 \
|
||||
--mem-fraction-static 0.7 \
|
||||
--port 8000 \
|
||||
--served-model-name glm-5 \
|
||||
--cuda-graph-bs 16 \
|
||||
--quantization modelslim \
|
||||
--moe-a2a-backend deepep --deepep-mode auto
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Multi-node Deployment
|
||||
|
||||
- `GLM-5-bf16`: require at least 2 Atlas 800 A3 (64G × 16).
|
||||
|
||||
**A3 series**
|
||||
|
||||
Modify the IP of 2 nodes, then run the same scripts on two nodes.
|
||||
|
||||
**node 0/1**
|
||||
|
||||
<CodeGroup>
|
||||
```shell Launch Multi-node Server
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
# bind cpu
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
unset https_proxy
|
||||
unset http_proxy
|
||||
unset HTTPS_PROXY
|
||||
unset HTTP_PROXY
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
# cann
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
export SGLANG_NPU_USE_MULTI_STREAM=1
|
||||
export HCCL_BUFFSIZE=1000
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
|
||||
# Run command ifconfig on two nodes, find out which inet addr has same IP with your node IP. That is your public interface, which should be added here
|
||||
export HCCL_SOCKET_IFNAME=lo
|
||||
export GLOO_SOCKET_IFNAME=lo
|
||||
|
||||
|
||||
P_IP=('your ip1' 'your ip2')
|
||||
P_MASTER="${P_IP[0]}:your port"
|
||||
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
|
||||
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
|
||||
LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
|
||||
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
|
||||
for i in "${!P_IP[@]}";
|
||||
do
|
||||
if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
|
||||
then
|
||||
echo "${P_IP[$i]}"
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path $MODEL_PATH \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--tp-size 32 --nnodes 2 --node-rank $i --dist-init-addr $P_MASTER \
|
||||
--chunked-prefill-size 16384 --max-prefill-tokens 131072 \
|
||||
--trust-remote-code \
|
||||
--host 127.0.0.1 \
|
||||
--mem-fraction-static 0.8\
|
||||
--port 8000 \
|
||||
--served-model-name glm-5 \
|
||||
--cuda-graph-max-bs 16 \
|
||||
--disable-radix-cache
|
||||
NODE_RANK=$i
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Prefill-Decode Disaggregation
|
||||
|
||||
Not test yet.
|
||||
|
||||
### Using Benchmark
|
||||
|
||||
Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling) for details.
|
||||
@@ -0,0 +1,169 @@
|
||||
## Introduction
|
||||
|
||||
MindSpore is a high-performance AI framework optimized for Ascend NPUs. This doc guides users to run MindSpore models in SGLang.
|
||||
|
||||
## Requirements
|
||||
|
||||
MindSpore currently only supports Ascend NPU devices. Users need to first install Ascend CANN software packages.
|
||||
The CANN software packages can be downloaded from the [Ascend Official Website](https://www.hiascend.com). The recommended version is 8.3.RC2.
|
||||
|
||||
## Supported Models
|
||||
|
||||
Currently, the following models are supported:
|
||||
|
||||
- **Qwen3**: Dense and MoE models
|
||||
- **DeepSeek V3/R1**
|
||||
- *More models coming soon...*
|
||||
|
||||
## Installation
|
||||
|
||||
<Note>
|
||||
Currently, MindSpore models are provided by an independent package `sgl-mindspore`. Support for MindSpore is built upon current SGLang support for Ascend NPU platform. Please first [install SGLang for Ascend NPU](./SGLang-installation-with-NPUs-support) and then install `sgl-mindspore`:
|
||||
</Note>
|
||||
|
||||
<CodeGroup>
|
||||
```shell Install
|
||||
git clone https://github.com/mindspore-lab/sgl-mindspore.git
|
||||
cd sgl-mindspore
|
||||
pip install -e .
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
|
||||
## Run Model
|
||||
|
||||
Current SGLang-MindSpore supports Qwen3 and DeepSeek V3/R1 models. This doc uses Qwen3-8B as an example.
|
||||
|
||||
### Offline infer
|
||||
|
||||
Use the following script for offline infer:
|
||||
|
||||
<CodeGroup>
|
||||
```python Offline Inference
|
||||
import sglang as sgl
|
||||
|
||||
# Initialize the engine with MindSpore backend
|
||||
llm = sgl.Engine(
|
||||
model_path="/path/to/your/model", # Local model path
|
||||
device="npu", # Use NPU device
|
||||
model_impl="mindspore", # MindSpore implementation
|
||||
attention_backend="ascend", # Attention backend
|
||||
tp_size=1, # Tensor parallelism size
|
||||
dp_size=1 # Data parallelism size
|
||||
)
|
||||
|
||||
# Generate text
|
||||
prompts = [
|
||||
"Hello, my name is",
|
||||
"The capital of France is",
|
||||
"The future of AI is"
|
||||
]
|
||||
|
||||
sampling_params = {"temperature": 0, "top_p": 0.9}
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
|
||||
for prompt, output in zip(prompts, outputs):
|
||||
print(f"Prompt: {prompt}")
|
||||
print(f"Generated: {output['text']}")
|
||||
print("---")
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Start server
|
||||
|
||||
Launch a server with MindSpore backend:
|
||||
|
||||
<CodeGroup>
|
||||
```bash Launch Server
|
||||
# Basic server startup
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path /path/to/your/model \
|
||||
--host 0.0.0.0 \
|
||||
--device npu \
|
||||
--model-impl mindspore \
|
||||
--attention-backend ascend \
|
||||
--tp-size 1 \
|
||||
--dp-size 1
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
For distributed server with multiple nodes:
|
||||
|
||||
<CodeGroup>
|
||||
```bash Multi-node Distributed
|
||||
# Multi-node distributed server
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path /path/to/your/model \
|
||||
--host 0.0.0.0 \
|
||||
--device npu \
|
||||
--model-impl mindspore \
|
||||
--attention-backend ascend \
|
||||
--dist-init-addr 127.0.0.1:29500 \
|
||||
--nnodes 2 \
|
||||
--node-rank 0 \
|
||||
--tp-size 4 \
|
||||
--dp-size 2
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
#### Debug Mode
|
||||
|
||||
Enable sglang debug logging by log-level argument.
|
||||
|
||||
<CodeGroup>
|
||||
```bash Debug Mode
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path /path/to/your/model \
|
||||
--host 0.0.0.0 \
|
||||
--device npu \
|
||||
--model-impl mindspore \
|
||||
--attention-backend ascend \
|
||||
--log-level DEBUG
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Enable mindspore info and debug logging by setting environments.
|
||||
|
||||
<CodeGroup>
|
||||
```bash Set Log Level
|
||||
export GLOG_v=1 # INFO
|
||||
export GLOG_v=0 # DEBUG
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
#### Explicitly select devices
|
||||
|
||||
Use the following environment variable to explicitly select the devices to use.
|
||||
|
||||
<CodeGroup>
|
||||
```shell Select Devices
|
||||
export ASCEND_RT_VISIBLE_DEVICES=4,5,6,7 # to set device
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
#### Some communication environment issues
|
||||
|
||||
In case of some environment with special communication environment, users need set some environment variables.
|
||||
|
||||
<CodeGroup>
|
||||
```shell Disable LCCL
|
||||
export MS_ENABLE_LCCL=off # current not support LCCL communication mode in SGLang-MindSpore
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
#### Some dependencies of protobuf
|
||||
|
||||
In case of some environment with special protobuf version, users need set some environment variables to avoid binary version mismatch.
|
||||
|
||||
<CodeGroup>
|
||||
```shell Fix Protobuf
|
||||
export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python # to avoid protobuf binary version mismatch
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
## Support
|
||||
For MindSpore-specific issues:
|
||||
|
||||
- Refer to the [MindSpore documentation](https://www.mindspore.cn/)
|
||||
@@ -0,0 +1,124 @@
|
||||
## Running Qwen3
|
||||
|
||||
### Running Qwen3-32B on 1 x Atlas 800I A3
|
||||
|
||||
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-32B)
|
||||
|
||||
<CodeGroup>
|
||||
```shell Launch Server
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1536
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--device npu \
|
||||
--attention-backend ascend \
|
||||
--trust-remote-code \
|
||||
--tp-size 4 \
|
||||
--model-path Qwen/Qwen3-32B \
|
||||
--mem-fraction-static 0.8
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Running Qwen3-32B on 1 x Atlas 800I A3 with Qwen3-32B-Eagle3
|
||||
|
||||
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-32B)
|
||||
|
||||
Speculative model weights could be found [here](https://huggingface.co/Zhihu-ai/Zhi-Create-Qwen3-32B-Eagle3)
|
||||
|
||||
<CodeGroup>
|
||||
```shell Launch Server with Eagle3
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--device npu \
|
||||
--attention-backend ascend \
|
||||
--trust-remote-code \
|
||||
--tp-size 4 \
|
||||
--model-path Qwen/Qwen3-32B \
|
||||
--mem-fraction-static 0.8 \
|
||||
--speculative-algorithm EAGLE3 \
|
||||
--speculative-draft-model-path Qwen/Qwen3-32B-Eagle3 \
|
||||
--speculative-num-steps 1 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 2
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Running Qwen3-30B-A3B MOE on 1 x Atlas 800I A3
|
||||
|
||||
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-30B-A3B)
|
||||
|
||||
<CodeGroup>
|
||||
```shell Launch Server
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1536
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
|
||||
export SGLANG_DEEPEP_BF16_DISPATCH=1
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--device npu \
|
||||
--attention-backend ascend \
|
||||
--trust-remote-code \
|
||||
--tp-size 4 \
|
||||
--model-path Qwen/Qwen3-30B-A3B \
|
||||
--mem-fraction-static 0.8
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Running Qwen3-235B-A22B-Instruct-2507 MOE on 1 x Atlas 800I A3
|
||||
|
||||
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507)
|
||||
|
||||
<CodeGroup>
|
||||
```shell Launch Server
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1536
|
||||
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
|
||||
export SGLANG_DEEPEP_BF16_DISPATCH=1
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-235B-A22B-Instruct-2507 \
|
||||
--tp-size 16 \
|
||||
--trust-remote-code \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--watchdog-timeout 9000 \
|
||||
--mem-fraction-static 0.8
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Running Qwen3-VL-8B-Instruct on 1 x Atlas 800I A3
|
||||
|
||||
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct)
|
||||
|
||||
<CodeGroup>
|
||||
```shell Launch Server
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1536
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--enable-multimodal \
|
||||
--attention-backend ascend \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--trust-remote-code \
|
||||
--tp-size 4 \
|
||||
--model-path Qwen/Qwen3-VL-8B-Instruct \
|
||||
--mem-fraction-static 0.8
|
||||
```
|
||||
</CodeGroup>
|
||||
@@ -0,0 +1,106 @@
|
||||
## Environment Preparation
|
||||
|
||||
### Installation
|
||||
|
||||
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the quay.io platform. You can directly pull it.
|
||||
|
||||
<CodeGroup>
|
||||
```bash Pull and Start Container
|
||||
#Atlas 800 A3
|
||||
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann8.5.0-a3-qwen3.5
|
||||
#Atlas 800 A2
|
||||
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann8.5.0-910b-qwen3.5
|
||||
|
||||
#start container
|
||||
docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
|
||||
--privileged=true --net=host \
|
||||
-v /var/queue_schedule:/var/queue_schedule \
|
||||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
-v /usr/local/sbin:/usr/local/sbin \
|
||||
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
|
||||
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
|
||||
--device=/dev/davinci0:/dev/davinci0 \
|
||||
--device=/dev/davinci1:/dev/davinci1 \
|
||||
--device=/dev/davinci2:/dev/davinci2 \
|
||||
--device=/dev/davinci3:/dev/davinci3 \
|
||||
--device=/dev/davinci4:/dev/davinci4 \
|
||||
--device=/dev/davinci5:/dev/davinci5 \
|
||||
--device=/dev/davinci6:/dev/davinci6 \
|
||||
--device=/dev/davinci7:/dev/davinci7 \
|
||||
--device=/dev/davinci8:/dev/davinci8 \
|
||||
--device=/dev/davinci9:/dev/davinci9 \
|
||||
--device=/dev/davinci10:/dev/davinci10 \
|
||||
--device=/dev/davinci11:/dev/davinci11 \
|
||||
--device=/dev/davinci12:/dev/davinci12 \
|
||||
--device=/dev/davinci13:/dev/davinci13 \
|
||||
--device=/dev/davinci14:/dev/davinci14 \
|
||||
--device=/dev/davinci15:/dev/davinci15 \
|
||||
--device=/dev/davinci_manager:/dev/davinci_manager \
|
||||
--device=/dev/hisi_hdc:/dev/hisi_hdc \
|
||||
--entrypoint=bash \
|
||||
swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:${TAG}
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
## Deployment
|
||||
|
||||
### Single-node Deployment
|
||||
|
||||
- Quantized model `qwen35_w8a8` can be deployed on 1 Atlas 800 A3 (64G × 16) .
|
||||
|
||||
Run the following script to execute online inference.
|
||||
|
||||
<CodeGroup>
|
||||
```shell Launch Server
|
||||
# high performance cpu
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
# bind cpu
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
unset https_proxy
|
||||
unset http_proxy
|
||||
unset HTTPS_PROXY
|
||||
unset HTTP_PROXY
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
# cann
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
export SGLANG_NPU_USE_MULTI_STREAM=1
|
||||
export HCCL_BUFFSIZE=1000
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
export HCCL_SOCKET_IFNAME=lo
|
||||
export GLOO_SOCKET_IFNAME=lo
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path $MODEL_PATH \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--tp-size 16 --nnodes 1 --node-rank 0 \
|
||||
--chunked-prefill-size 16384 --max-prefill-tokens 280000 \
|
||||
--trust-remote-code \
|
||||
--host 127.0.0.1 \
|
||||
--mem-fraction-static 0.7 \
|
||||
--port 8000 \
|
||||
--cuda-graph-bs 16 \
|
||||
--quantization modelslim \
|
||||
--enable-multimodal \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--dtype bfloat16
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Prefill-Decode Disaggregation
|
||||
|
||||
Not test yet.
|
||||
|
||||
### Using Benchmark
|
||||
|
||||
Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling) for details.
|
||||
+318
@@ -0,0 +1,318 @@
|
||||
---
|
||||
title: SGLang installation with NPUs support
|
||||
---
|
||||
|
||||
You can install SGLang using any of the methods below. Please go through `System Settings` section to ensure the clusters are roaring at max performance. Feel free to leave an issue [here at sglang](https://github.com/sgl-project/sglang/issues) if you encounter any issues or have any problems.
|
||||
|
||||
## Component Version Mapping For SGLang
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "34%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Component</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Version</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Obtain Way</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>HDK</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>25.3.RC1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[<Icon icon="download" />](https://hiascend.com/hardware/firmware-drivers/commercial?product=7\&model=33)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>CANN</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8.5.0</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[Obtain Images](#obtain-cann-image)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Pytorch Adapter</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>7.3.0</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[<Icon icon="external-link" />](https://gitcode.com/Ascend/pytorch/releases)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MemFabric</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1.0.5</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`pip install memfabric-hybrid==1.0.5`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Triton</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>3.2.0</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`pip install triton-ascend`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Bisheng</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>20251121</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[<Icon icon="external-link" />](https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/sglang/triton_ascend/Ascend-BiSheng-toolkit_aarch64_20251121.run)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>SGLang NPU Kernel</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>NA</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[<Icon icon="external-link" />](https://github.com/sgl-project/sgl-kernel-npu/releases)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<Accordion title="Obtain CANN Image" defaultOpen="true">
|
||||
|
||||
You can obtain the dependency of a specified version of CANN through an image.
|
||||
|
||||
```bash
|
||||
# for Atlas 800I A3 and Ubuntu OS
|
||||
docker pull quay.io/ascend/cann:8.5.0-a3-ubuntu22.04-py3.11
|
||||
# for Atlas 800I A2 and Ubuntu OS
|
||||
docker pull quay.io/ascend/cann:8.5.0-910b-ubuntu22.04-py3.11
|
||||
```
|
||||
</Accordion>
|
||||
|
||||
## Preparing the Running Environment
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Source">
|
||||
<AccordionGroup>
|
||||
<Accordion title="Python Version">
|
||||
Only `python==3.11` is supported currently. If you don't want to break system pre-installed python, try installing with [conda](https://github.com/conda/conda).
|
||||
|
||||
```bash
|
||||
conda create --name sglang_npu python=3.11
|
||||
conda activate sglang_npu
|
||||
```
|
||||
</Accordion>
|
||||
<Accordion title="CANN">
|
||||
Prior to start work with SGLang on Ascend you need to install CANN Toolkit, Kernels operator package and NNAL version 8.3.RC2 or higher, check the [installation guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/83RC1/softwareinst/instg/instg_0008.html?Mode=PmIns\&InstallType=local\&OS=openEuler\&Software=cannToolKit)
|
||||
</Accordion>
|
||||
<Accordion title="MemFabric-Hybrid">
|
||||
If you want to use PD disaggregation mode, you need to install MemFabric-Hybrid. MemFabric-Hybrid is a drop-in replacement of Mooncake Transfer Engine that enables KV cache transfer on Ascend NPU clusters.
|
||||
|
||||
```bash
|
||||
pip install memfabric-hybrid==1.0.5
|
||||
```
|
||||
</Accordion>
|
||||
<Accordion title="Pytorch and Pytorch Framework Adaptor on Ascend">
|
||||
```bash
|
||||
PYTORCH_VERSION=2.8.0
|
||||
TORCHVISION_VERSION=0.23.0
|
||||
TORCH_NPU_VERSION=2.8.0
|
||||
pip install torch==$PYTORCH_VERSION torchvision==$TORCHVISION_VERSION --index-url https://download.pytorch.org/whl/cpu
|
||||
pip install torch_npu==$TORCH_NPU_VERSION
|
||||
```
|
||||
|
||||
If you are using other versions of `torch` and install `torch_npu`, check [installation guide](https://github.com/Ascend/pytorch/blob/master/README)
|
||||
</Accordion>
|
||||
<Accordion title="Triton on Ascend">
|
||||
We provide our own implementation of Triton for Ascend.
|
||||
|
||||
```bash
|
||||
BISHENG_NAME="Ascend-BiSheng-toolkit_aarch64_20251121.run"
|
||||
BISHENG_URL="https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/sglang/triton_ascend/${BISHENG_NAME}"
|
||||
wget -O "${BISHENG_NAME}" "${BISHENG_URL}" && chmod a+x "${BISHENG_NAME}" && "./${BISHENG_NAME}" --install && rm "${BISHENG_NAME}"
|
||||
```
|
||||
|
||||
```bash
|
||||
pip install triton-ascend
|
||||
```
|
||||
|
||||
For installation of Triton on Ascend nightly builds or from sources, follow [installation guide](https://gitcode.com/Ascend/triton-ascend/blob/master/docs/sources/getting-started/installation)
|
||||
|
||||
</Accordion>
|
||||
<Accordion title="SGLang Kernels NPU">
|
||||
We provide SGL kernels for Ascend NPU, check [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/sgl_kernel_npu/README).
|
||||
</Accordion>
|
||||
<Accordion title="DeepEP-compatible Library">
|
||||
We provide a DeepEP-compatible Library as a drop-in replacement of deepseek-ai's DeepEP library, check the [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/deep_ep/README).
|
||||
</Accordion>
|
||||
<Accordion title="Installing SGLang from source">
|
||||
```bash
|
||||
# Use the last release branch
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
mv python/pyproject_npu.toml python/pyproject.toml
|
||||
pip install -e python[all_npu]
|
||||
```
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
### Obtain Image
|
||||
|
||||
You can download the SGLang image or build an image based on Dockerfile to obtain the Ascend NPU image.
|
||||
|
||||
1. **Download SGLang image**
|
||||
|
||||
```bash
|
||||
dockerhub: docker.io/lmsysorg/sglang:$tag
|
||||
# Main-based tag, change main to specific version like v0.5.6,
|
||||
# you can get image for specific version
|
||||
Atlas 800I A3 : {main}-cann8.5.0-a3
|
||||
Atlas 800I A2: {main}-cann8.5.0-910b
|
||||
```
|
||||
|
||||
2. **Build an image based on Dockerfile**
|
||||
|
||||
```bash
|
||||
# Clone the SGLang repository
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang/docker
|
||||
|
||||
# Build the docker image
|
||||
# If there are network errors, please modify the Dockerfile to use offline dependencies or use a proxy
|
||||
docker build -t <image_name> -f npu.Dockerfile .
|
||||
```
|
||||
|
||||
### Create Docker
|
||||
|
||||
<Info>`--privileged` and `--network=host` are required by RDMA, which is typically needed by Ascend NPU clusters.</Info>
|
||||
<Note>The following docker command is based on Atlas 800I A3 machines. If you are using Atlas 800I A2, make sure only `davinci[0-7]` are mapped into container.</Note>
|
||||
|
||||
```bash
|
||||
alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
|
||||
--device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
|
||||
--device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
|
||||
--device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11 \
|
||||
--device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15 \
|
||||
--device=/dev/davinci_manager --device=/dev/hisi_hdc \
|
||||
--volume /usr/local/sbin:/usr/local/sbin --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
|
||||
--volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
|
||||
--volume /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
--volume /var/queue_schedule:/var/queue_schedule --volume ~/.cache/:/root/.cache/'
|
||||
|
||||
# Add HF_TOKEN env for download model by SGLang.
|
||||
drun --env "HF_TOKEN=<secret>" \
|
||||
<image_name> \
|
||||
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend
|
||||
```
|
||||
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
## System Settings
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="CPU performance power scheme" defaultOpen="true">
|
||||
The default power scheme on Ascend hardware is `ondemand` which could affect performance, changing it to `performance` is recommended.
|
||||
|
||||
```bash
|
||||
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
|
||||
# Make sure changes are applied successfully
|
||||
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor # shows performance
|
||||
```
|
||||
</Accordion>
|
||||
<Accordion title="Disable NUMA balancing" defaultOpen="true">
|
||||
```bash
|
||||
sudo sysctl -w kernel.numa_balancing=0
|
||||
# Check
|
||||
cat /proc/sys/kernel/numa_balancing # shows 0
|
||||
```
|
||||
</Accordion>
|
||||
<Accordion title="Prevent swapping out system memory" defaultOpen="true">
|
||||
```bash
|
||||
sudo sysctl -w vm.swappiness=10
|
||||
|
||||
# Check
|
||||
cat /proc/sys/vm/swappiness # shows 10
|
||||
```
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
## Running SGLang Service
|
||||
|
||||
<Tabs>
|
||||
<Tab title="For Large Language Models">
|
||||
|
||||
### PD Mixed Scene
|
||||
|
||||
```bash
|
||||
# Enabling CPU Affinity
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend
|
||||
```
|
||||
|
||||
### PD Separation Scene
|
||||
|
||||
1. **Launch Prefill Server**
|
||||
|
||||
```bash
|
||||
# Enabling CPU Affinity
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
# PIP: recommended to config first Prefill Server IP
|
||||
# PORT: one free port
|
||||
# all sglang servers need to be config the same PIP and PORT,
|
||||
export ASCEND_MF_STORE_URL="tcp://PIP:PORT"
|
||||
# if you are Atlas 800I A2 hardware and use rdma for kv cache transfer, add this parameter
|
||||
export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
||||
--disaggregation-mode prefill \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--disaggregation-bootstrap-port 8995 \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--base-gpu-id 0 \
|
||||
--tp-size 1 \
|
||||
--host 127.0.0.1 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
2. **Launch Decode Server**
|
||||
|
||||
```bash
|
||||
# PIP: recommended to config first Prefill Server IP
|
||||
# PORT: one free port
|
||||
# all sglang servers need to be config the same PIP and PORT,
|
||||
export ASCEND_MF_STORE_URL="tcp://PIP:PORT"
|
||||
# if you are Atlas 800I A2 hardware and use rdma for kv cache transfer, add this parameter
|
||||
export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
||||
--disaggregation-mode decode \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--base-gpu-id 1 \
|
||||
--tp-size 1 \
|
||||
--host 127.0.0.1 \
|
||||
--port 8001
|
||||
```
|
||||
|
||||
3. **Launch Router**
|
||||
|
||||
```bash
|
||||
python3 -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--policy cache_aware \
|
||||
--prefill http://127.0.0.1:8000 8995 \
|
||||
--decode http://127.0.0.1:8001 \
|
||||
--host 127.0.0.1 \
|
||||
--port 6688
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="For Multimodal Language Models">
|
||||
|
||||
### PD Mixed Scene
|
||||
|
||||
```bash
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path Qwen3-VL-30B-A3B-Instruct \
|
||||
--host 127.0.0.1 \
|
||||
--port 8000 \
|
||||
--tp 4 \
|
||||
--device npu \
|
||||
--attention-backend ascend \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--disable-radix-cache \
|
||||
--trust-remote-code \
|
||||
--enable-multimodal \
|
||||
--sampling-backend ascend
|
||||
```
|
||||
|
||||
</Tab>
|
||||
</Tabs>
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,610 @@
|
||||
This section describes the models supported on the Ascend NPU, including Large Language Models, Multimodal Language
|
||||
Models, Embedding Models, Reward Models and Rerank Models. Mainstream DeepSeek/Qwen/GLM series are included.
|
||||
You are welcome to enable various models based on your business requirements.
|
||||
|
||||
<Accordion title="Large Language Models">
|
||||
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Models</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Model Family</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek V3/V3.1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>vllm-ascend/DeepSeek-V3.2-Exp-W8A8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>vllm-ascend/DeepSeek-R1-0528-W8A8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>vllm-ascend/DeepSeek-V2-Lite-W8A8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-30B-A3B-Instruct-2507</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-32B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-0.6B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>vllm-ascend/Qwen3-235B-A22B-W8A8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-Next-80B-A3B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3-Coder-480B-A35B-Instruct-w8a8-QuaRot</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen2.5-7B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>vllm-ascend/QWQ-32B-W8A8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>meta-llama/Llama-4-Scout-17B-16E-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AI-ModelScope/Llama-3.1-8B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>LLM-Research/llama-2-7b</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>LLM-Research/Llama-3.2-1B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>mistralai/Mistral-7B-Instruct-v0.2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Mistral</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>google/gemma-3-4b-it</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Gemma</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>microsoft/Phi-4-multimodal-instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Phi</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>allenai/OLMoE-1B-7B-0924</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>OLMoE</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>stabilityai/stablelm-2-1_6b</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>StableLM</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>CohereForAI/c4ai-command-r-v01</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Command-R</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>huihui-ai/grok-2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Grok</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ZhipuAI/chatglm2-6b</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>ChatGLM</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Shanghai_AI_Laboratory/internlm2-7b</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>InternLM 2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>ExaONE 3</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>xverse/XVERSE-MoE-A36B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>XVERSE</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>HuggingFaceTB/SmolLM-1.7B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>SmolLM</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ZhipuAI/glm-4-9b-chat</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GLM-4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>XiaomiMiMo/MiMo-7B-RL</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MiMo</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>arcee-ai/AFM-4.5B-Base</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Arcee AFM-4.5B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Howeee/persimmon-8b-chat</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Persimmon</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>inclusionAI/Ling-lite</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Ling</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ibm-granite/granite-3.1-8b-instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Granite</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ibm-granite/granite-3.0-3b-a800m-instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Granite MoE</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AI-ModelScope/dbrx-instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DBRX (Databricks)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>baichuan-inc/Baichuan2-13B-Chat</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Baichuan 2 (7B, 13B)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>baidu/ERNIE-4.5-21B-A3B-PT</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>ERNIE-4.5 (4.5, 4.5MoE series)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>OpenBMB/MiniCPM3-4B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MiniCPM (v3, 4B)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Kimi/Kimi-K2-Thinking</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Kimi</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>openai/gpt-oss-120b</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GPTOSS</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>allenai/OLMo-2-1124-7B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>OLMo</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>minimax/MiniMax-M2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MiniMax-M2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>upstage/SOLAR-10.7B-Instruct-v1.0</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Solar</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>bigcode/starcoder2-7b</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>StarCoder2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>arcee-ai/Trinity-Mini</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Trinity (Nano, Mini)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
</Accordion>
|
||||
|
||||
|
||||
<Accordion title="Multimodal Language Models">
|
||||
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Models</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Model Family (Variants)</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen2.5-VL-3B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen-VL</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen2.5-VL-72B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen-VL</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-VL-30B-A3B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen-VL</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-VL-8B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen-VL</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-VL-4B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen-VL</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-VL-235B-A22B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen-VL</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>deepseek-ai/deepseek-vl2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek-VL2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>deepseek-ai/Janus-Pro-1B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Janus-Pro (1B, 7B)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>deepseek-ai/Janus-Pro-7B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Janus-Pro (1B, 7B)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>openbmb/MiniCPM-V-2_6</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MiniCPM-V / MiniCPM-o</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>openbmb/MiniCPM-o-2_6</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MiniCPM-V / MiniCPM-o</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>google/gemma-3-4b-it</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Gemma 3 (Multimodal)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>mistralai/Mistral-Small-3.1-24B-Instruct-2503</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Mistral-Small-3.1-24B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>microsoft/Phi-4-multimodal-instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Phi-4-multimodal-instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>XiaomiMiMo/MiMo-VL-7B-RL</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MiMo-VL (7B)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AI-ModelScope/llava-v1.6-34b</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>LLaVA (v1.5 & v1.6)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>lmms-lab/llava-next-72b</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>LLaVA-NeXT (8B, 72B)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>lmms-lab/llava-onevision-qwen2-7b-ov</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>LLaVA-OneVision</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Kimi/Kimi-VL-A3B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Kimi-VL (A3B)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ZhipuAI/GLM-4.5V</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GLM-4.5V (106B)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>LLM-Research/Llama-3.2-11B-Vision-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama 3.2 Vision (11B)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>rednote-hilab/dots.ocr</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DotsVLM-OCR</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
</Accordion>
|
||||
|
||||
|
||||
<Accordion title="Embedding Models">
|
||||
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Models</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Model Family</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>intfloat/e5-mistral-7b-instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>E5 (Llama/Mistral based)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>iic/gte_Qwen2-1.5B-instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GTE-Qwen2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-Embedding-8B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3-Embedding</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Alibaba-NLP/gme-Qwen2-VL-2B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GME (Multimodal)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AI-ModelScope/clip-vit-large-patch14-336</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>CLIP</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>BAAI/bge-large-en-v1.5</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>BGE</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
</Accordion>
|
||||
|
||||
|
||||
<Accordion title="Reward Models">
|
||||
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Models</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Model Family</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Skywork/Skywork-Reward-Llama-3.1-8B-v0.2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama3.1 Reward</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Shanghai_AI_Laboratory/internlm2-7b-reward</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>InternLM 2 Reward</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen2.5-Math-RM-72B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen2.5 Reward - Math</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Howeee/Qwen2.5-1.5B-apeach</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen2.5 Reward - Sequence</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AI-ModelScope/Skywork-Reward-Gemma-2-27B-v0.2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Gemma 2-27B Reward</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
</Accordion>
|
||||
|
||||
|
||||
<Accordion title="Rerank Models">
|
||||
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Models</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Model Family</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>BAAI/bge-reranker-v2-m3</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>BGE-Reranker</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</Accordion>
|
||||
@@ -0,0 +1,355 @@
|
||||
---
|
||||
title: "CPU Servers"
|
||||
---
|
||||
|
||||
The document addresses how to set up the [SGLang](https://github.com/sgl-project/sglang) environment and run LLM inference on CPU servers.
|
||||
SGLang is enabled and optimized on the CPUs equipped with Intel® AMX® Instructions,
|
||||
which are 4th generation or newer Intel® Xeon® Scalable Processors.
|
||||
|
||||
## Optimized Model List
|
||||
|
||||
A list of popular LLMs are optimized and run efficiently on CPU,
|
||||
including the most notable open-source models like Llama series, Qwen series,
|
||||
and DeepSeek series like DeepSeek-R1 and DeepSeek-V3.1-Terminus.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "22%"}} />
|
||||
<col style={{width: "26%"}} />
|
||||
<col style={{width: "30%"}} />
|
||||
<col style={{width: "22%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model Name</th>
|
||||
<th style={{textAlign: "center", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16</th>
|
||||
<th style={{textAlign: "center", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>W8A8_INT8</th>
|
||||
<th style={{textAlign: "center", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>FP8</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-R1</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/meituan/DeepSeek-R1-Channel-INT8">DeepSeek-R1-Channel-INT8</a></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/deepseek-ai/DeepSeek-R1">DeepSeek-R1</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-V3.1-Terminus</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8">DeepSeek-V3.1-Terminus-Channel-int8</a></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/deepseek-ai/DeepSeek-V3.1-Terminus">DeepSeek-V3.1-Terminus</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-3.2-3B</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct">Llama-3.2-3B-Instruct</a></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/Llama-3.2-3B-Instruct-quantized.w8a8">Llama-3.2-3B-quantized.w8a8</a></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-3.1-8B</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct">Llama-3.1-8B-Instruct</a></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8">Llama-3.1-8B-quantized.w8a8</a></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>QwQ-32B</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/QwQ-32B-quantized.w8a8">QwQ-32B-quantized.w8a8</a></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-Distilled-Llama</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8">DeepSeek-R1-Distill-Llama-70B-quantized.w8a8</a></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3-235B</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/Qwen/Qwen3-235B-A22B-FP8">Qwen3-235B-A22B-FP8</a></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
> **Note:** The model identifiers listed in the table above have been verified on 6th Gen Intel® Xeon® P-core platforms.
|
||||
|
||||
## Installation
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Docker (Recommended)">
|
||||
It is recommended to use Docker for setting up the SGLang environment.
|
||||
A [Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile) is provided to facilitate the installation.
|
||||
|
||||
> **Note:** Replace `<secret>` below with your [HuggingFace access token](https://huggingface.co/docs/hub/en/security-tokens).
|
||||
|
||||
<CodeGroup>
|
||||
```bash Clone, Build and Run
|
||||
# Clone the SGLang repository
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang/docker
|
||||
|
||||
# Build the docker image
|
||||
docker build -t sglang-cpu:latest -f xeon.Dockerfile .
|
||||
|
||||
# Initiate a docker container
|
||||
docker run \
|
||||
-it \
|
||||
--privileged \
|
||||
--ipc=host \
|
||||
--network=host \
|
||||
-v /dev/shm:/dev/shm \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
-p 30000:30000 \
|
||||
-e "HF_TOKEN=<secret>" \
|
||||
sglang-cpu:latest /bin/bash
|
||||
```
|
||||
</CodeGroup>
|
||||
</Tab>
|
||||
|
||||
<Tab title="From Source">
|
||||
If you prefer to install SGLang in a bare metal environment, the setup process is as follows.
|
||||
|
||||
Please install the required packages and libraries beforehand if they are not already present on your system.
|
||||
You can refer to the Ubuntu-based installation commands in
|
||||
[the Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile#L11) for guidance.
|
||||
|
||||
1. **Install uv and create a virtual environment**
|
||||
<CodeGroup>
|
||||
```bash Create Virtual Environment
|
||||
# Taking '/opt' as the example uv env folder, feel free to change it as needed
|
||||
cd /opt
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
source $HOME/.local/bin/env
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
2. **Create a config file for torch package indexes**
|
||||
Create the `uv.toml` config file:
|
||||
|
||||
<CodeGroup>
|
||||
```bash Open Config File
|
||||
vim .venv/uv.toml
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Press `a` to enter insert mode in `vim`, then paste the following content:
|
||||
|
||||
<CodeGroup>
|
||||
```toml
|
||||
[[index]]
|
||||
name = "torch"
|
||||
url = "https://download.pytorch.org/whl/cpu"
|
||||
|
||||
[[index]]
|
||||
name = "torchvision"
|
||||
url = "https://download.pytorch.org/whl/cpu"
|
||||
|
||||
[[index]]
|
||||
name = "torchaudio"
|
||||
url = "https://download.pytorch.org/whl/cpu"
|
||||
|
||||
[[index]]
|
||||
name = "triton"
|
||||
url = "https://download.pytorch.org/whl/cpu"
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Save the file (press `Esc`, then type `:x` and hit `Enter`), then set it as the default `uv` config:
|
||||
|
||||
<CodeGroup>
|
||||
```bash Set Config Path
|
||||
export UV_CONFIG_FILE=/opt/.venv/uv.toml
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
3. **Clone SGLang and build packages**
|
||||
<CodeGroup>
|
||||
```bash Build SGLang
|
||||
# Clone the SGLang code
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
git checkout <YOUR-DESIRED-VERSION>
|
||||
|
||||
# Use dedicated toml file
|
||||
cd python
|
||||
cp pyproject_cpu.toml pyproject.toml
|
||||
# Install SGLang dependent libs, and build SGLang main package
|
||||
uv pip install --upgrade pip setuptools
|
||||
uv pip install .
|
||||
|
||||
# Build the CPU backend kernels
|
||||
cd ../sgl-kernel
|
||||
cp pyproject_cpu.toml pyproject.toml
|
||||
uv pip install .
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
4. **Set required environment variables**
|
||||
<CodeGroup>
|
||||
```bash Set Environment Variables
|
||||
export SGLANG_USE_CPU_ENGINE=1
|
||||
|
||||
# Set 'LD_LIBRARY_PATH' and 'LD_PRELOAD' to ensure the libs can be loaded by sglang processes
|
||||
export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu
|
||||
export LD_PRELOAD=${LD_PRELOAD}:/opt/.venv/lib/libiomp5.so:${LD_LIBRARY_PATH}/libtcmalloc.so.4:${LD_LIBRARY_PATH}/libtbbmalloc.so.2
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
> **Note:** The environment variable `SGLANG_USE_CPU_ENGINE=1` is required to enable the SGLang service with the CPU engine.
|
||||
|
||||
> **Note:** If you encounter code compilation issues during the `sgl-kernel` building process, please check your `gcc` and `g++` versions and upgrade them if they are outdated. It is recommended to use `gcc-13` and `g++-13` as they have been verified in the official Docker container.
|
||||
|
||||
> **Note:** The system library path is typically located in one of the following directories: `~/.local/lib/`, `/usr/local/lib/`, `/usr/local/lib64/`, `/usr/lib/`, `/usr/lib64/`, and `/usr/lib/x86_64-linux-gnu/`. In the above example commands, `/usr/lib/x86_64-linux-gnu` is used. Please adjust the path according to your server configuration.
|
||||
|
||||
It is recommended to add the following to your `~/.bashrc` file to avoid setting these variables every time you open a new terminal:
|
||||
|
||||
<CodeGroup>
|
||||
```bash Persist in ~/.bashrc
|
||||
source .venv/bin/activate
|
||||
export SGLANG_USE_CPU_ENGINE=1
|
||||
export LD_LIBRARY_PATH=<YOUR-SYSTEM-LIBRARY-FOLDER>
|
||||
export LD_PRELOAD=<YOUR-LIBS-PATHS>
|
||||
```
|
||||
</CodeGroup>
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
## Launch of the Serving Engine
|
||||
|
||||
Example command to launch SGLang serving:
|
||||
|
||||
<CodeGroup>
|
||||
```bash Launch Server
|
||||
python -m sglang.launch_server \
|
||||
--model <MODEL_ID_OR_PATH> \
|
||||
--trust-remote-code \
|
||||
--disable-overlap-schedule \
|
||||
--device cpu \
|
||||
--host 0.0.0.0 \
|
||||
--tp 6
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
> **Note:** For running W8A8 quantized models, please add the flag `--quantization w8a8_int8`.
|
||||
|
||||
> **Note:** The flag `--tp 6` specifies that tensor parallelism will be applied using 6 ranks (TP6). On a CPU platform, a TP rank means a sub-NUMA cluster (SNC). You can get the SNC count using `lscpu`. If the specified TP rank number differs from the total SNC count, the system will automatically utilize the first `n` SNCs — but `n` cannot exceed the total SNC number.
|
||||
>
|
||||
> To specify the cores to be used, set the environment variable `SGLANG_CPU_OMP_THREADS_BIND`. For example, to use the first 40 cores of each SNC on a Xeon® 6980P server (which has 43-43-42 cores on the 3 SNCs of a socket):
|
||||
|
||||
<CodeGroup>
|
||||
```bash Set Thread Binding
|
||||
export SGLANG_CPU_OMP_THREADS_BIND="0-39|43-82|86-125|128-167|171-210|214-253"
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
> Please beware that with `SGLANG_CPU_OMP_THREADS_BIND` set, the available memory amounts of the ranks may not be determined in advance. You may need to set `--max-total-tokens` to avoid out-of-memory errors.
|
||||
|
||||
> **Note:** For optimizing decoding with `torch.compile`, add the flag `--enable-torch-compile`. To specify the maximum batch size, set `--torch-compile-max-bs`. For example, `--enable-torch-compile --torch-compile-max-bs 4` uses `torch.compile` with a maximum batch size of 4. The maximum applicable batch size is 16.
|
||||
|
||||
> **Note:** A warmup step is automatically triggered when the service is started. The server is ready when you see the log `The server is fired up and ready to roll!`.
|
||||
|
||||
## Benchmarking with Requests
|
||||
|
||||
You can benchmark the performance via the `bench_serving` script.
|
||||
Run the command in another terminal. An example command would be:
|
||||
|
||||
<CodeGroup>
|
||||
```bash Run Benchmark
|
||||
python -m sglang.bench_serving \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1 \
|
||||
--request-rate inf \
|
||||
--random-range-ratio 1.0
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Detailed parameter descriptions are available via the command:
|
||||
|
||||
<CodeGroup>
|
||||
```bash Benchmark Help
|
||||
python -m sglang.bench_serving -h
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Additionally, requests can be formatted using
|
||||
[the OpenAI Completions API](../basic_usage/openai_api_completions)
|
||||
and sent via the command line (e.g., using `curl`) or through your own scripts.
|
||||
|
||||
## Example Usage Commands
|
||||
|
||||
Large Language Models can range from fewer than 1 billion to several hundred billion parameters.
|
||||
Dense models larger than 20B are expected to run on flagship 6th Gen Intel® Xeon® processors
|
||||
with dual sockets and a total of 6 sub-NUMA clusters. Dense models of approximately 10B parameters or fewer,
|
||||
or MoE (Mixture of Experts) models with fewer than 10B activated parameters, can run on more common
|
||||
4th generation or newer Intel® Xeon® processors, or utilize a single socket of the flagship 6th Gen Intel® Xeon® processors.
|
||||
|
||||
### Example: Running DeepSeek-V3.1-Terminus
|
||||
|
||||
<CodeGroup>
|
||||
```bash W8A8_INT8
|
||||
python -m sglang.launch_server \
|
||||
--model IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8 \
|
||||
--trust-remote-code \
|
||||
--disable-overlap-schedule \
|
||||
--device cpu \
|
||||
--quantization w8a8_int8 \
|
||||
--host 0.0.0.0 \
|
||||
--enable-torch-compile \
|
||||
--torch-compile-max-bs 4 \
|
||||
--tp 6
|
||||
```
|
||||
|
||||
```bash FP8
|
||||
python -m sglang.launch_server \
|
||||
--model deepseek-ai/DeepSeek-V3.1-Terminus \
|
||||
--trust-remote-code \
|
||||
--disable-overlap-schedule \
|
||||
--device cpu \
|
||||
--host 0.0.0.0 \
|
||||
--enable-torch-compile \
|
||||
--torch-compile-max-bs 4 \
|
||||
--tp 6
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
> **Note:** Please set `--torch-compile-max-bs` to the maximum desired batch size for your deployment, which can be up to 16. The value `4` in the examples is illustrative.
|
||||
|
||||
### Example: Running Llama-3.2-3B
|
||||
|
||||
<CodeGroup>
|
||||
```bash BF16
|
||||
python -m sglang.launch_server \
|
||||
--model meta-llama/Llama-3.2-3B-Instruct \
|
||||
--trust-remote-code \
|
||||
--disable-overlap-schedule \
|
||||
--device cpu \
|
||||
--host 0.0.0.0 \
|
||||
--enable-torch-compile \
|
||||
--torch-compile-max-bs 16 \
|
||||
--tp 2
|
||||
```
|
||||
|
||||
```bash W8A8_INT8
|
||||
python -m sglang.launch_server \
|
||||
--model RedHatAI/Llama-3.2-3B-quantized.w8a8 \
|
||||
--trust-remote-code \
|
||||
--disable-overlap-schedule \
|
||||
--device cpu \
|
||||
--quantization w8a8_int8 \
|
||||
--host 0.0.0.0 \
|
||||
--enable-torch-compile \
|
||||
--torch-compile-max-bs 16 \
|
||||
--tp 2
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
> **Note:** The `--torch-compile-max-bs` and `--tp` settings are examples that should be adjusted for your setup. For instance, use `--tp 3` to utilize 1 socket with 3 sub-NUMA clusters on an Intel® Xeon® 6980P server.
|
||||
|
||||
Once the server has been launched, you can test it using the `bench_serving` command or create
|
||||
your own commands or scripts following [the benchmarking example](#benchmarking-with-requests).
|
||||
@@ -0,0 +1,5 @@
|
||||
---
|
||||
title: NVIDIA GPUs
|
||||
---
|
||||
|
||||
Please refer to the [Installation Guide](/docs/get-started/installation) to get started with SGLang on NVIDIA GPUs.
|
||||
@@ -0,0 +1,102 @@
|
||||
---
|
||||
title: NVIDIA Jetson Orin
|
||||
description: Guide for installing and running SGLang on NVIDIA Jetson Orin devices.
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before starting, ensure the following:
|
||||
|
||||
- [NVIDIA Jetson AGX Orin Devkit](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/) is set up with JetPack 6.1 or later.
|
||||
- CUDA Toolkit and cuDNN are installed.
|
||||
- Verify that the Jetson AGX Orin is in high-performance mode:
|
||||
|
||||
<CodeGroup>
|
||||
```bash
|
||||
sudo nvpmodel -m 0
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
## Installing and Running SGLang with Jetson Containers
|
||||
|
||||
1. **Clone the jetson-containers repository**
|
||||
|
||||
```bash
|
||||
git clone https://github.com/dusty-nv/jetson-containers.git
|
||||
```
|
||||
|
||||
2. **Run the installation script**
|
||||
|
||||
```bash
|
||||
bash jetson-containers/install.sh
|
||||
```
|
||||
|
||||
3. **Build the container image**
|
||||
|
||||
```bash
|
||||
jetson-containers build sglang
|
||||
```
|
||||
|
||||
4. **Run the container**
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Using jetson-containers">
|
||||
```bash
|
||||
jetson-containers run $(autotag sglang)
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Using Docker manually">
|
||||
```bash
|
||||
docker run --runtime nvidia -it --rm --network=host IMAGE_NAME
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
## Running Inference
|
||||
|
||||
Launch the server:
|
||||
|
||||
<CodeGroup>
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-R1-Distill-Llama-8B \
|
||||
--device cuda \
|
||||
--dtype half \
|
||||
--attention-backend flashinfer \
|
||||
--mem-fraction-static 0.8 \
|
||||
--context-length 8192
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
The quantization and limited context length (`--dtype half` `--context-length 8192`) are due to the limited computational resources in [Nvidia jetson kit](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/). A detailed explanation can be found in [Server Arguments](../advanced_features/server_arguments).
|
||||
|
||||
After launching the engine, refer to [Chat completions](../basic_usage/openai_api_completions#Usage) to test the usability.
|
||||
|
||||
## Running Quantization with TorchAO
|
||||
|
||||
TorchAO is suggested to NVIDIA Jetson Orin.
|
||||
|
||||
<CodeGroup>
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--device cuda \
|
||||
--dtype bfloat16 \
|
||||
--attention-backend flashinfer \
|
||||
--mem-fraction-static 0.8 \
|
||||
--context-length 8192 \
|
||||
--torchao-config int4wo-128
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
This enables TorchAO's int4 weight-only quantization with a 128-group size. The usage of `--torchao-config int4wo-128` is also for memory efficiency.
|
||||
|
||||
## Structured Output with XGrammar
|
||||
|
||||
Please refer to [SGLang doc structured output](../advanced_features/structured_outputs).
|
||||
|
||||
Thanks to the support from [Nurgaliyev Shakhizat](https://github.com/shahizat), [Dustin Franklin](https://github.com/dusty-nv) and [Johnny Núñez Cano](https://github.com/johnnynunez).
|
||||
|
||||
## References
|
||||
|
||||
- [NVIDIA Jetson AGX Orin Documentation](https://developer.nvidia.com/embedded/jetson-agx-orin)
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
title: Hardware Platforms
|
||||
description: Platform-specific guides for running SGLang on GPUs, TPUs, NPUs, CPUs, and more.
|
||||
---
|
||||
|
||||
- [NVIDIA GPUs](./nvidia-gpus)
|
||||
- [AMD GPUs](./amd-gpus)
|
||||
- [Ascend NPUs](./ascend-npus/SGLang-installation-with-NPUs-support)
|
||||
- [CPU Server](./cpu-server)
|
||||
- [NVIDIA (Edge & Embedded)](./nvidia)
|
||||
- [TPU](./tpu)
|
||||
- [XPU](./xpu)
|
||||
@@ -0,0 +1,673 @@
|
||||
---
|
||||
title: "TPU"
|
||||
description: "SGLang supports high-performance TPU inference through the SGLang-JAX backend, which is specifically optimized for Google Cloud TPUs. The JAX-based implementation delivers exceptional throughput and low latency for Large Language Model (LLM) serving workloads on TPU hardware."
|
||||
---
|
||||
|
||||
<Note>
|
||||
SGLang TPU support is implemented via the SGLang-JAX backend, a dedicated JAX-based inference engine maintained as a separate repository at [sgl-project/sglang-jax](https://github.com/sgl-project/sglang-jax).
|
||||
</Note>
|
||||
|
||||
For TPU-specific issues or feature requests, please visit the [sglang-jax GitHub issues page](https://github.com/sgl-project/sglang-jax/issues).
|
||||
|
||||
---
|
||||
|
||||
# System Requirements
|
||||
|
||||
---
|
||||
|
||||
## Supported TPU Hardware
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="TPU v6e" icon="server">
|
||||
**32 GB** HBM Memory — Available on Google Cloud
|
||||
</Card>
|
||||
<Card title="TPU v7" icon="server">
|
||||
**96 GB per core** HBM Memory — Available on Google Cloud
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
---
|
||||
|
||||
## Software Requirements
|
||||
|
||||
<CardGroup cols={3}>
|
||||
<Card title="Python" icon="python">
|
||||
Version **3.12 or higher**
|
||||
</Card>
|
||||
<Card title="JAX" icon="code">
|
||||
**Latest version** with TPU support
|
||||
</Card>
|
||||
<Card title="Environment" icon="cloud">
|
||||
Google Cloud TPU VM or compatible TPU runtime. **Optional:** SkyPilot for simplified cloud deployment.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
---
|
||||
|
||||
# Feature Support Matrix
|
||||
|
||||
SGLang-JAX provides comprehensive TPU-optimized features for production LLM serving:
|
||||
|
||||
### Supported Features
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "34%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Feature**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>**Support Status**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Description**</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>High-Throughput Continuous Batching</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Dynamic request batching for maximum TPU utilization</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Radix Tree KV Cache</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Memory-efficient prefix sharing between requests</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>FlashAttention Backend</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>TPU-optimized attention kernel for long sequences</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Tensor Parallelism</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Distribute models across multiple TPU cores</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Paged Attention</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Flexible KV cache management with paging</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Speculative Decoding (EAGLE/EAGLE3)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>20-40% throughput improvement for compatible models</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Chunked Prefill</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Mixed prefill-decode batching</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>OpenAI-Compatible API</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Drop-in replacement for OpenAI API</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Data Parallel Attention</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>🚧</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>In development — Attention computation with data parallelism</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Quantization</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>🚧</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>In development — Model quantization for reduced memory usage</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Multi-LoRA</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>🚧</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>In development — Serve multiple LoRA adapters simultaneously</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
---
|
||||
|
||||
### Attention Backend Comparison
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "20%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Backend**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>**Paged Attention**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Spec Decoding**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>**MLA**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Sliding Window**</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>FlashAttention (fa)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Native</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<Note>
|
||||
FlashAttention backend is recommended for production workloads due to superior memory efficiency and performance.
|
||||
</Note>
|
||||
|
||||
---
|
||||
|
||||
# Optimized Model List
|
||||
|
||||
The following models have been tested and optimized for TPU deployment:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "50%"}} />
|
||||
<col style={{width: "50%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Model Family**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>**Performance Status**</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen 3</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>⭐ Recommended for production</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen 3 MoE</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>⭐ Best performance</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen 2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Needs improvement</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen 2 MoE</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Needs improvement</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen 1.5</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Needs improvement</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Llama/LLaMA</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Needs improvement</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Grok-2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Needs improvement</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Gemma 2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Verified on TPU</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Bailing MoE</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Needs improvement</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
---
|
||||
|
||||
# Installation
|
||||
|
||||
<Tabs>
|
||||
<Tab title="PyPI (Recommended)">
|
||||
```bash
|
||||
pip install sglang-jax
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="From Source">
|
||||
```bash
|
||||
git clone https://github.com/sgl-project/sglang-jax
|
||||
cd sglang-jax
|
||||
uv venv --python 3.12 && source .venv/bin/activate
|
||||
uv pip install -e "python[all]"
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Docker">
|
||||
<Warning>
|
||||
Docker support for TPU is currently under development. Please use PyPI or source installation methods.
|
||||
</Warning>
|
||||
</Tab>
|
||||
<Tab title="SkyPilot (Cloud TPU)">
|
||||
SkyPilot provides simplified deployment on Google Cloud TPU:
|
||||
|
||||
1. **Install SkyPilot and configure GCP access**
|
||||
|
||||
See the [SkyPilot documentation](https://docs.skypilot.co/en/latest/) for setup instructions.
|
||||
|
||||
2. **Create a SkyPilot configuration file**
|
||||
|
||||
Create a SkyPilot YAML file: `sglang-jax.sky.yaml`
|
||||
|
||||
3. **Launch your TPU cluster**
|
||||
|
||||
```bash
|
||||
# Standard deployment
|
||||
sky launch -c sglang-jax sglang-jax.sky.yaml --infra=gcp
|
||||
|
||||
# With spot instances for cost savings
|
||||
sky launch -c sglang-jax sglang-jax.sky.yaml --infra=gcp --use-spot
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
---
|
||||
|
||||
# Launch the Serving Engine
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Basic: Qwen-7B">
|
||||
```bash
|
||||
JAX_COMPILATION_CACHE_DIR=/tmp/jit_cache python3 -u -m sgl_jax.launch_server \
|
||||
--model-path Qwen/Qwen-7B-Chat \
|
||||
--trust-remote-code \
|
||||
--dist-init-addr=0.0.0.0:10011 \
|
||||
--nnodes=1 \
|
||||
--tp-size=4 \
|
||||
--device=tpu \
|
||||
--random-seed=3 \
|
||||
--node-rank=0 \
|
||||
--mem-fraction-static=0.8 \
|
||||
--max-prefill-tokens=8192 \
|
||||
--download-dir=/tmp \
|
||||
--dtype=bfloat16 \
|
||||
--skip-server-warmup \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Key Parameters Explained" icon="sliders">
|
||||
<ParamField path="JAX_COMPILATION_CACHE_DIR" type="string">
|
||||
Enables JIT compilation caching to accelerate server startup on subsequent runs. Recommended: `/tmp/jit_cache`
|
||||
</ParamField>
|
||||
<ParamField path="--tp-size" type="integer" default="1">
|
||||
Tensor parallelism size; match this to your TPU core count (typically `1`, `4`, or `8`).
|
||||
</ParamField>
|
||||
<ParamField path="--device" type="string" default="tpu">
|
||||
Specifies TPU device. This is the default for `sglang-jax`.
|
||||
</ParamField>
|
||||
<ParamField path="--dtype" type="string" default="bfloat16">
|
||||
Uses bfloat16 precision, which TPUs are optimized for.
|
||||
</ParamField>
|
||||
<ParamField path="--mem-fraction-static" type="float" default="0.8">
|
||||
Allocates this fraction of TPU HBM for static memory. Adjustable from `0.2` to `0.9`.
|
||||
</ParamField>
|
||||
<ParamField path="--max-prefill-tokens" type="integer" default="8192">
|
||||
Maximum number of tokens processed in the prefill phase.
|
||||
</ParamField>
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
</Tab>
|
||||
<Tab title="High-Performance: Qwen3-8B">
|
||||
For production workloads with optimal throughput:
|
||||
|
||||
```bash
|
||||
python3 -u -m sgl_jax.launch_server \
|
||||
--model-path Qwen/Qwen3-8B \
|
||||
--trust-remote-code \
|
||||
--tp-size=4 \
|
||||
--device=tpu \
|
||||
--mem-fraction-static=0.8 \
|
||||
--chunked-prefill-size=2048 \
|
||||
--dtype=bfloat16 \
|
||||
--max-running-requests=256 \
|
||||
--page-size=128 \
|
||||
--attention-backend=fa
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Speculative Decoding (EAGLE3)">
|
||||
Speculative decoding can improve throughput by 20-40% for compatible models:
|
||||
|
||||
```bash
|
||||
python3 -u -m sgl_jax.launch_server \
|
||||
--model-path Qwen/Qwen3-32B \
|
||||
--trust-remote-code \
|
||||
--device=tpu \
|
||||
--tp-size=4 \
|
||||
--mem-fraction-static=0.8 \
|
||||
--max-prefill-tokens=4096 \
|
||||
--attention-backend=fa \
|
||||
--dtype=bfloat16 \
|
||||
--port=30000 \
|
||||
--host=0.0.0.0 \
|
||||
--disable-overlap-schedule \
|
||||
--speculative-algorithm=EAGLE3 \
|
||||
--speculative-draft-model-path=AngelSlim/Qwen3-32B_eagle3 \
|
||||
--page-size=64 \
|
||||
--speculative-eagle-topk=1 \
|
||||
--speculative-num-steps=3 \
|
||||
--speculative-num-draft-tokens=4
|
||||
```
|
||||
|
||||
<Note>
|
||||
Speculative decoding is currently supported for Qwen3 and LLaMA model families. See the [Speculative Decoding documentation](../advanced_features/speculative_decoding) for detailed configuration guidance.
|
||||
</Note>
|
||||
</Tab>
|
||||
<Tab title="Multi-Node Distributed">
|
||||
For large models requiring multiple TPU VMs:
|
||||
|
||||
```bash
|
||||
# Node 0 (coordinator)
|
||||
python3 -m sgl_jax.launch_server \
|
||||
--model-path MODEL_PATH \
|
||||
--dist-init-addr=NODE0_IP:10011 \
|
||||
--nnodes=2 \
|
||||
--node-rank=0 \
|
||||
--tp-size=8 \
|
||||
[other parameters...]
|
||||
|
||||
# Node 1 (worker)
|
||||
python3 -m sgl_jax.launch_server \
|
||||
--model-path MODEL_PATH \
|
||||
--dist-init-addr=NODE0_IP:10011 \
|
||||
--nnodes=2 \
|
||||
--node-rank=1 \
|
||||
--tp-size=8 \
|
||||
[other parameters...]
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
---
|
||||
|
||||
# Benchmarking with Requests
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Throughput Testing">
|
||||
Basic throughput benchmark:
|
||||
|
||||
```bash
|
||||
python3 -m sgl_jax.bench_serving \
|
||||
--backend sgl-jax \
|
||||
--dataset-name random \
|
||||
--num-prompts=100 \
|
||||
--random-input=512 \
|
||||
--random-output=128 \
|
||||
--max-concurrency=8 \
|
||||
--random-range-ratio=1 \
|
||||
--warmup-requests=0
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Latency Testing">
|
||||
Measure single-batch latency:
|
||||
|
||||
```bash
|
||||
python3 -m sgl_jax.bench_one_batch_server \
|
||||
--base-url http://127.0.0.1:30000 \
|
||||
--model-path Qwen/Qwen-7B-Chat \
|
||||
--batch-size=32 \
|
||||
--input-len=256 \
|
||||
--output-len=32
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Comprehensive Benchmark Script">
|
||||
For systematic performance evaluation across different configurations:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
backend=${1:-sgl-jax}
|
||||
num_prompts_per_concurrency=3
|
||||
input_seq_lens=(1024 4096 8192)
|
||||
output_seq_lens=(1 1024)
|
||||
max_concurrencies=(8 16 32 64 128 256)
|
||||
|
||||
for input_seq_len in "${input_seq_lens[@]}"; do
|
||||
for output_seq_len in "${output_seq_lens[@]}"; do
|
||||
echo "======================================="
|
||||
echo "Testing ISL/OSL: $input_seq_len/$output_seq_len"
|
||||
echo "======================================="
|
||||
for max_concurrency in "${max_concurrencies[@]}"; do
|
||||
num_prompts=$((num_prompts_per_concurrency * max_concurrency))
|
||||
python3 -m sgl_jax.bench_serving \
|
||||
--backend ${backend} \
|
||||
--dataset-name random \
|
||||
--num-prompts ${num_prompts} \
|
||||
--random-input ${input_seq_len} \
|
||||
--random-output ${output_seq_len} \
|
||||
--max-concurrency ${max_concurrency} \
|
||||
--random-range-ratio 1 \
|
||||
--disable-ignore-eos \
|
||||
--warmup-requests 0
|
||||
done
|
||||
done
|
||||
done
|
||||
```
|
||||
|
||||
For detailed help on all benchmark parameters:
|
||||
|
||||
```bash
|
||||
python3 -m sgl_jax.bench_serving --help
|
||||
```
|
||||
|
||||
See the [Benchmark and Profiling Guide](../developer_guide/benchmark_and_profiling) for advanced benchmarking techniques and profiling with JAX Profiler.
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
---
|
||||
|
||||
# Performance Optimization
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Memory Optimization" icon="memory">
|
||||
**Reduce memory usage:**
|
||||
- Lower `--mem-fraction-static` (from `0.8` → `0.5` → `0.3`)
|
||||
- Decrease `--max-prefill-tokens` (from `16384` → `8192` → `4096`)
|
||||
- Reduce `--max-running-requests`
|
||||
|
||||
**Handle OOM errors:**
|
||||
- Start with conservative memory settings (`--mem-fraction-static=0.5`)
|
||||
- Gradually increase until you find the optimal balance
|
||||
- Increase `--page-size` for better memory locality (`1` → `16` → `64` → `128`)
|
||||
</Accordion>
|
||||
<Accordion title="Throughput Optimization" icon="chart-line">
|
||||
To maximize tokens per second:
|
||||
- Use FlashAttention backend: `--attention-backend=fa`
|
||||
- Enable speculative decoding (EAGLE3) for Qwen3 models (20-40% improvement)
|
||||
- Increase `--max-running-requests` to `256+`
|
||||
- Set `--mem-fraction-static` to `0.8+` (if memory allows)
|
||||
- Use larger page sizes (`64-128`)
|
||||
- Enable chunked prefill: `--chunked-prefill-size=2048`
|
||||
</Accordion>
|
||||
<Accordion title="Latency Optimization" icon="gauge-high">
|
||||
To minimize time-to-first-token (TTFT) and inter-token latency:
|
||||
- Reduce `--page-size` to `1-4`
|
||||
- Lower `--max-running-requests` (`16-32`) for smaller batches
|
||||
- Reduce `--chunked-prefill-size`
|
||||
- Use conservative memory settings to avoid GC pauses
|
||||
</Accordion>
|
||||
<Accordion title="TPU-Specific Optimizations" icon="microchip">
|
||||
**JIT Compilation Cache:**
|
||||
|
||||
```bash
|
||||
export JAX_COMPILATION_CACHE_DIR=/tmp/jit_cache
|
||||
```
|
||||
|
||||
Always set this environment variable to cache compiled kernels and accelerate server startup.
|
||||
|
||||
**Data Type Optimization:** Use `--dtype=bfloat16` for TPU native optimization. TPUs are specifically designed for bfloat16 computations.
|
||||
|
||||
**Tensor Parallelism:** Match `--tp-size` to your TPU core configuration (`1`, `4`, or `8`) for optimal model distribution.
|
||||
|
||||
**Attention Backend:** Always use `--attention-backend=fa` (FlashAttention) for production workloads.
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
---
|
||||
|
||||
# Troubleshooting
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="OOM (Out of Memory) Errors" icon="triangle-exclamation">
|
||||
If you encounter out-of-memory errors:
|
||||
|
||||
1. **Reduce mem-fraction-static**
|
||||
|
||||
Lower `--mem-fraction-static` from `0.8` to `0.5` or lower.
|
||||
|
||||
2. **Decrease max-prefill-tokens**
|
||||
|
||||
Decrease `--max-prefill-tokens` from `8192` to `4096` or `2048`.
|
||||
|
||||
3. **Lower max-running-requests**
|
||||
|
||||
Lower `--max-running-requests` to reduce concurrent batch size.
|
||||
|
||||
4. **Increase page-size**
|
||||
|
||||
Increase `--page-size` for better memory layout efficiency.
|
||||
</Accordion>
|
||||
<Accordion title="Slow Compilation / Long Startup" icon="clock">
|
||||
If the server takes too long to start:
|
||||
|
||||
<Check>Ensure `JAX_COMPILATION_CACHE_DIR` is properly set</Check>
|
||||
<Check>Understand that the first run requires JIT compilation — this is normal</Check>
|
||||
<Check>Subsequent runs will be significantly faster with cached compilations</Check>
|
||||
<Check>Consider using `--skip-server-warmup` to defer compilation until first request</Check>
|
||||
</Accordion>
|
||||
<Accordion title="Low Throughput" icon="gauge">
|
||||
If you're not achieving expected throughput:
|
||||
|
||||
<Check>Verify `--tp-size` matches your TPU core configuration</Check>
|
||||
<Check>Check that `--attention-backend=fa` is enabled</Check>
|
||||
<Check>Increase `--max-running-requests` to enable larger batch formation</Check>
|
||||
<Check>Consider enabling speculative decoding for compatible models</Check>
|
||||
<Check>Ensure memory settings allow for sufficient batch sizes</Check>
|
||||
</Accordion>
|
||||
<Accordion title="Connection Issues" icon="network-wired">
|
||||
If clients cannot connect to the server:
|
||||
|
||||
<Check>Ensure `--host=0.0.0.0` for external access (not just `127.0.0.1`)</Check>
|
||||
<Check>Verify firewall rules allow traffic on the specified port (default: `30000`)</Check>
|
||||
<Check>Check that the server process is running: `curl http://localhost:30000/health`</Check>
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
---
|
||||
|
||||
# Advanced Features
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Speculative Decoding" icon="bolt">
|
||||
SGLang-JAX supports EAGLE and EAGLE3 speculative decoding algorithms for Qwen3 and LLaMA model families. Speculative decoding can improve throughput by 20-40% without affecting output quality.
|
||||
|
||||
See the [Speculative Decoding documentation](../advanced_features/speculative_decoding) for detailed configuration and supported model combinations.
|
||||
</Accordion>
|
||||
<Accordion title="Chunked Prefill" icon="layer-group">
|
||||
Enable mixed prefill-decode batching for better TPU utilization:
|
||||
|
||||
```bash
|
||||
--chunked-prefill-size=2048 --enable-mixed-chunk
|
||||
```
|
||||
|
||||
This allows the scheduler to mix prefill operations with decode operations in the same batch, improving overall throughput.
|
||||
</Accordion>
|
||||
<Accordion title="Custom Attention Backends" icon="puzzle-piece">
|
||||
SGLang-JAX supports a plugin-based attention backend system. You can implement custom attention kernels optimized for specific use cases.
|
||||
|
||||
See the [Attention Backend documentation](https://github.com/sgl-project/sglang-jax/tree/main/docs) for implementation details.
|
||||
</Accordion>
|
||||
<Accordion title="Environment Verification" icon="circle-check">
|
||||
Verify your TPU setup before deploying:
|
||||
|
||||
```bash
|
||||
python -c "from sgl_jax import check_env; check_env.check_env()"
|
||||
```
|
||||
|
||||
This command checks:
|
||||
- Installed package versions
|
||||
- TPU device availability and specifications
|
||||
- System resources and configuration
|
||||
- Compatibility of settings
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
---
|
||||
|
||||
# Contributing
|
||||
|
||||
We welcome contributions to improve TPU support in SGLang-JAX!
|
||||
|
||||
<Note>
|
||||
Check the [Development Roadmap](https://github.com/sgl-project/sglang-jax) to see planned features and find opportunities to contribute new functionality.
|
||||
</Note>
|
||||
|
||||
Current contribution areas include:
|
||||
|
||||
- Performance optimizations for specific TPU generations
|
||||
- Support for additional model architectures
|
||||
- Documentation improvements and examples
|
||||
- Bug reports and fixes
|
||||
- Benchmark results and performance analysis
|
||||
|
||||
<CardGroup cols={3}>
|
||||
<Card title="Repository" icon="github" href="https://github.com/sgl-project/sglang-jax">
|
||||
Visit the sglang-jax repository
|
||||
</Card>
|
||||
<Card title="Contribution Guide" icon="book" href="https://github.com/sgl-project/sglang-jax/blob/main/CONTRIBUTING.md">
|
||||
Read the Contribution Guide
|
||||
</Card>
|
||||
<Card title="Slack Community" icon="slack" href="https://slack.sglang.io/">
|
||||
Join the SGL-JAX Slack community for discussions
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
---
|
||||
|
||||
## Testing on TPU
|
||||
|
||||
For contributors who need TPU access for testing:
|
||||
|
||||
- Refer to the [TPU Resources Guide](https://cloud.google.com/tpu/docs/managing-tpus-tpu-vm) for information on accessing TPU hardware
|
||||
- Use SkyPilot with spot instances for cost-effective testing
|
||||
- Follow the [Benchmark and Profiling Guide](../developer_guide/benchmark_and_profiling) for performance validation
|
||||
|
||||
---
|
||||
|
||||
# References
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="SGLang-JAX Repository" icon="github" href="https://github.com/sgl-project/sglang-jax">
|
||||
Source code and issue tracker for the JAX TPU backend.
|
||||
</Card>
|
||||
<Card title="SGLang-JAX Installation Guide" icon="book-open" href="https://github.com/sgl-project/sglang-jax?tab=readme-ov-file#installation">
|
||||
Step-by-step installation instructions.
|
||||
</Card>
|
||||
<Card title="Qwen Models Quick Start" icon="rocket" href="https://github.com/sgl-project/sglang-jax/tree/main/docs">
|
||||
Get up and running quickly with the Qwen model family.
|
||||
</Card>
|
||||
<Card title="Benchmark and Profiling Guide" icon="chart-bar" href="../developer_guide/benchmark_and_profiling">
|
||||
Advanced benchmarking techniques and JAX Profiler usage.
|
||||
</Card>
|
||||
<Card title="Speculative Decoding" icon="forward" href="../advanced_features/speculative_decoding">
|
||||
EAGLE and EAGLE3 speculative decoding configuration.
|
||||
</Card>
|
||||
<Card title="JAX Documentation" icon="code" href="https://jax.readthedocs.io">
|
||||
Official JAX documentation and API reference.
|
||||
</Card>
|
||||
<Card title="Google Cloud TPU Docs" icon="cloud" href="https://cloud.google.com/tpu/docs">
|
||||
Google Cloud TPU product documentation.
|
||||
</Card>
|
||||
<Card title="SkyPilot Documentation" icon="paper-plane" href="https://docs.skypilot.co/en/latest/">
|
||||
Simplified cloud deployment with SkyPilot.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,146 @@
|
||||
---
|
||||
title: XPU
|
||||
sidebarTitle: Intel GPUs (XPU)
|
||||
---
|
||||
|
||||
The document addresses how to set up the [SGLang](https://github.com/sgl-project/sglang) environment and run LLM inference on Intel GPU, [see more context about Intel GPU support within PyTorch ecosystem](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html).
|
||||
|
||||
Specifically, SGLang is optimized for:
|
||||
- [Intel® Arc™ Pro B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/242616/intel-arc-pro-b-series-graphics.html)
|
||||
- [Intel® Arc™ B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/240391/intel-arc-b-series-graphics.html).
|
||||
|
||||
## Optimized Model List
|
||||
|
||||
A list of LLMs have been optimized on Intel GPU, and more are on the way:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "50%"}} />
|
||||
<col style={{width: "50%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model Name</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-3.2-3B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-3.1-8B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen2.5-1.5B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[Qwen/Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<Note>The model identifiers listed in the table above have been verified on [Intel® Arc™ B580 Graphics](https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.html).</Note>
|
||||
|
||||
## Installation
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Source">
|
||||
|
||||
Currently SGLang XPU only supports installation from source. Please refer to [“Getting Started on Intel GPU”](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html) to install XPU dependency.
|
||||
|
||||
1. **Creation & Activation**
|
||||
|
||||
Create and activate a conda environment.
|
||||
|
||||
```bash
|
||||
conda create -n sgl-xpu python=3.12 -y
|
||||
conda activate sgl-xpu
|
||||
```
|
||||
|
||||
2. **Install PyTorch and Dependencies**
|
||||
|
||||
Set PyTorch XPU as primary pip install channel to avoid installing the larger CUDA-enabled version and prevent potential runtime issues.
|
||||
|
||||
```bash
|
||||
pip3 install torch==2.9.0+xpu torchao torchvision torchaudio pytorch-triton-xpu==3.5.0 --index-url https://download.pytorch.org/whl/xpu
|
||||
pip3 install xgrammar --no-deps # xgrammar will introduce CUDA-enabled triton which might conflict with XPU
|
||||
```
|
||||
|
||||
3. **Cloning**
|
||||
|
||||
Clone the SGLang code
|
||||
|
||||
```bash
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
git checkout <YOUR-DESIRED-VERSION>
|
||||
```
|
||||
|
||||
4. **Configure Build File**
|
||||
|
||||
Use dedicated toml file
|
||||
|
||||
```bash
|
||||
cd python
|
||||
cp pyproject_xpu.toml pyproject.toml
|
||||
```
|
||||
|
||||
5. **Build and Install**
|
||||
|
||||
Install SGLang dependent libs, and build SGLang main package
|
||||
|
||||
```bash
|
||||
pip install --upgrade pip setuptools
|
||||
pip install -v .
|
||||
```
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
<Info>The docker for XPU is under active development. Please stay tuned.</Info>
|
||||
</Tab>
|
||||
</Tabs>
|
||||
## Launch of the Serving Engine
|
||||
|
||||
Example command to launch SGLang serving:
|
||||
|
||||
<CodeGroup>
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model <MODEL_ID_OR_PATH> \
|
||||
--trust-remote-code \
|
||||
--disable-overlap-schedule \
|
||||
--device xpu \
|
||||
--host 0.0.0.0 \
|
||||
--tp 2 \ # using multi GPUs
|
||||
--attention-backend intel_xpu \ # using intel optimized XPU attention backend
|
||||
--page-size \ # intel_xpu attention backend supports [32, 64, 128]
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
## Benchmarking with Requests
|
||||
|
||||
You can benchmark the performance via the `bench_serving` script. Run the command in another terminal.
|
||||
|
||||
<CodeGroup>
|
||||
```bash
|
||||
python -m sglang.bench_serving \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1 \
|
||||
--request-rate inf \
|
||||
--random-range-ratio 1.0
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
The detail explanations of the parameters can be looked up by the command:
|
||||
|
||||
<CodeGroup>
|
||||
```bash
|
||||
python -m sglang.bench_serving -h
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Additionally, the requests can be formed with [OpenAI Completions API](../basic_usage/openai_api_completions) and sent via the command line (e.g. using `curl`) or via your own script.
|
||||
@@ -0,0 +1,54 @@
|
||||
---
|
||||
title: "Custom Chat Template"
|
||||
metatags:
|
||||
description: "SGLang custom chat templates: JSON and Jinja formats for OpenAI-compatible API server. Override tokenizer defaults."
|
||||
---
|
||||
**NOTE**: There are two chat template systems in SGLang project. This document is about setting a custom chat template for the OpenAI-compatible API server (defined at [conversation.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/conversation.py)). It is NOT related to the chat template used in the SGLang language frontend (defined at [chat_template.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/lang/chat_template.py)).
|
||||
|
||||
By default, the server uses the chat template specified in the model tokenizer from Hugging Face.
|
||||
It should just work for most official models such as Llama-2/Llama-3.
|
||||
|
||||
If needed, you can also override the chat template when launching the server:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-2-7b-chat-hf \
|
||||
--port 30000 \
|
||||
--chat-template llama-2
|
||||
```
|
||||
|
||||
If the chat template you are looking for is missing, you are welcome to contribute it or load it from a file.
|
||||
|
||||
## JSON Format
|
||||
|
||||
You can load the JSON format, which is defined by `conversation.py`.
|
||||
|
||||
```json Config
|
||||
{
|
||||
"name": "my_model",
|
||||
"system": "<|im_start|>system",
|
||||
"user": "<|im_start|>user",
|
||||
"assistant": "<|im_start|>assistant",
|
||||
"sep_style": "CHATML",
|
||||
"sep": "<|im_end|>",
|
||||
"stop_str": ["<|im_end|>", "<|im_start|>"]
|
||||
}
|
||||
```
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-2-7b-chat-hf \
|
||||
--port 30000 \
|
||||
--chat-template ./my_model_template.json
|
||||
```
|
||||
|
||||
## Jinja Format
|
||||
|
||||
You can also use the [Jinja template format](https://huggingface.co/docs/transformers/main/en/chat_templating) as defined by Hugging Face Transformers.
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-2-7b-chat-hf \
|
||||
--port 30000 \
|
||||
--chat-template ./my_model_template.jinja
|
||||
```
|
||||
@@ -0,0 +1,652 @@
|
||||
---
|
||||
title: "Environment Variables"
|
||||
metatags:
|
||||
description: "SGLang environment variables: SGLANG_* and SGL_* configs for performance, memory, DeepGEMM, DeepEP, profiling."
|
||||
---
|
||||
SGLang supports various environment variables that can be used to configure its runtime behavior. This document provides a comprehensive list and aims to stay updated over time.
|
||||
|
||||
*Note: SGLang uses two prefixes for environment variables: `SGL_` and `SGLANG_`. This is likely due to historical reasons. While both are currently supported for different settings, future versions might consolidate them.*
|
||||
|
||||
## General Configuration
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Environment Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_USE_MODELSCOPE`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable using models from ModelScope</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_HOST_IP`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Host IP address for the server</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`0.0.0.0`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_PORT`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Port for the server</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>auto-detected</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_LOGGING_CONFIG_PATH`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Custom logging configuration path</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Not set</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_DISABLE_REQUEST_LOGGING`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Disable request logging</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_HEALTH_CHECK_TIMEOUT`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Timeout for health check in seconds</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`20`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The interval of passes to collect the metric of selected count of physical experts on each layer and GPU rank. 0 means disabled.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`0`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_FORWARD_UNKNOWN_TOOLS`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Forward unknown tool calls to clients instead of dropping them</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false` (drop unknown tools)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_QUEUED_TIMEOUT_MS`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Timeout (in ms) for requests in the waiting queue</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`-1`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Environment Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_ENABLE_TORCH_INFERENCE_MODE`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Control whether to use torch.inference_mode</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_ENABLE_TORCH_COMPILE`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable torch.compile</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`true`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_SET_CPU_AFFINITY`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable CPU affinity setting (often set to `1` in Docker builds)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`0`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Allows the scheduler to overwrite longer context length requests (often set to `1` in Docker builds)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`0`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_IS_FLASHINFER_AVAILABLE`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Control FlashInfer availability check</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`true`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_SKIP_P2P_CHECK`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Skip P2P (peer-to-peer) access check</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_CHUNKED_PREFIX_CACHE_THRESHOLD`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Sets the threshold for enabling chunked prefix caching</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`8192`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_FUSED_MLA_ENABLE_ROPE_FUSION`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable RoPE fusion in Fused Multi-Layer Attention</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`1`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_DISABLE_CONSECUTIVE_PREFILL_OVERLAP`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Disable overlap schedule for consecutive prefill batches</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_SCHEDULER_MAX_RECV_PER_POLL`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Set the maximum number of requests per poll, with a negative value indicating no limit</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`-1`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_DISABLE_FA4_WARMUP`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Disable Flash Attention 4 warmup passes (set to `1`, `true`, `yes`, or `on` to disable)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_DATA_PARALLEL_BUDGET_INTERVAL`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Interval for DPBudget updates</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`1`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_SCHEDULER_RECV_SKIPPER_WEIGHT_DEFAULT`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Default weight value for scheduler recv skipper counter (used when forward mode doesn't match specific modes). Only active when `--scheduler-recv-interval > 1`. The counter accumulates weights and triggers request polling when reaching the interval threshold.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`1000`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_SCHEDULER_RECV_SKIPPER_WEIGHT_DECODE`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Weight increment for decode forward mode in scheduler recv skipper. Works with `--scheduler-recv-interval` to control polling frequency during decode phase.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`1`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_SCHEDULER_RECV_SKIPPER_WEIGHT_VERIFY`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Weight increment for target verify forward mode in scheduler recv skipper. Works with `--scheduler-recv-interval` to control polling frequency during verification phase.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`1`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_SCHEDULER_RECV_SKIPPER_WEIGHT_NONE`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Weight increment when forward mode is None in scheduler recv skipper. Works with `--scheduler-recv-interval` to control polling frequency when no specific forward mode is active.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`1`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_MM_BUFFER_SIZE_MB`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Size of preallocated GPU buffer (in MB) for multi-modal feature hashing optimization. When set to a positive value, temporarily moves features to GPU for faster hash computation, then moves them back to CPU to save GPU memory. Larger features benefit more from GPU hashing. Set to `0` to disable.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`0`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_MM_PRECOMPUTE_HASH`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable precomputing of hash values for MultimodalDataItem</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable NCCL for gathering when preparing mlp sync batch under overlap scheduler (without this flag gloo is used for gathering)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_SYMM_MEM_PREALLOC_GB_SIZE`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Size of preallocated GPU buffer (in GB) for NCCL symmetric memory pool to limit memory fragmentation. Only have an effect when server arg `--enable-symm-mem` is set.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`4`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
## DeepGEMM Configuration (Advanced Optimization)
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Environment Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_ENABLE_JIT_DEEPGEMM`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable Just-In-Time compilation of DeepGEMM kernels (enabled by default on NVIDIA Hopper (SM90) and Blackwell (SM100) GPUs when the DeepGEMM package is installed; set to `"0"` to disable)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`"true"`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_JIT_DEEPGEMM_PRECOMPILE`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable precompilation of DeepGEMM kernels</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`"true"`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_JIT_DEEPGEMM_COMPILE_WORKERS`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of workers for parallel DeepGEMM kernel compilation</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`4`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_IN_DEEPGEMM_PRECOMPILE_STAGE`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Indicator flag used during the DeepGEMM precompile script</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`"false"`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_DG_CACHE_DIR`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Directory for caching compiled DeepGEMM kernels</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`~/.cache/deep_gemm`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGL_DG_USE_NVRTC`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Use NVRTC (instead of Triton) for JIT compilation (Experimental)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`"0"`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGL_USE_DEEPGEMM_BMM`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Use DeepGEMM for Batched Matrix Multiplication (BMM) operations</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`"false"`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## DeepEP Configuration
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Environment Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_DEEPEP_BF16_DISPATCH`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Bfloat16 for dispatch</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`"false"`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The maximum number of dispatched tokens on each GPU</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`"128"`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_FLASHINFER_NUM_MAX_DISPATCH_TOKENS_PER_RANK`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The maximum number of dispatched tokens on each GPU for --moe-a2a-backend=flashinfer</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`"1024"`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_DEEPEP_LL_COMBINE_SEND_NUM_SMS`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of SMs used for DeepEP combine when single batch overlap is enabled</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`"32"`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_BLACKWELL_OVERLAP_SHARED_EXPERTS_OUTSIDE_SBO`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Run shared experts on an alternate stream when single batch overlap is enabled on GB200. When not setting this flag, shared experts and down gemm will be overlapped with DeepEP combine together.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`"false"`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## NSA Backend Configuration (For DeepSeek V3.2)
|
||||
|
||||
{/* # Environment variable to control mtp precomputing of metadata for multi-step speculative decoding */}
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Environment Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_NSA_FUSE_TOPK`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Fuse the operation of picking topk logits and picking topk indices from page table</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`true`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_NSA_ENABLE_MTP_PRECOMPUTE_METADATA`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Precompute metadata that can be shared among different draft steps when MTP is enabled</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`true`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
## Memory Management
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Environment Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_DEBUG_MEMORY_POOL`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable memory pool debugging</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_CLIP_MAX_NEW_TOKENS_ESTIMATION`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Clip max new tokens estimation for memory planning</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`4096`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_DETOKENIZER_MAX_STATES`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Maximum states for detokenizer</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Default value based on system</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable checks for memory imbalance across Tensor Parallel ranks</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`true`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Model-Specific Options
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Environment Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_USE_AITER`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Use AITER optimize implementation</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_MOE_PADDING`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable MoE padding (sets padding size to 128 if value is `1`, often set to `1` in Docker builds)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`0`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_CUTLASS_MOE` (deprecated)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Cutlass FP8 MoE kernel on Blackwell GPUs (deprecated, use --moe-runner-backend=cutlass)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Quantization
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Environment Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_INT4_WEIGHT`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable INT4 weight quantization</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_PER_TOKEN_GROUP_QUANT_8BIT_V2`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Apply per token group quantization kernel with fused silu and mul and masked m</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_FORCE_FP8_MARLIN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Force using FP8 MARLIN kernels even if other FP8 kernels are available</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_FLASHINFER_FP4_GEMM_BACKEND` (deprecated)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Select backend for `mm_fp4` on Blackwell GPUs. **DEPRECATED**: Please use `--fp4-gemm-backend` instead.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>``</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_NVFP4_CKPT_FP8_GEMM_IN_ATTN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Quantize q_b_proj from BF16 to FP8 when launching DeepSeek NVFP4 checkpoint</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_MOE_NVFP4_DISPATCH`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Use nvfp4 for moe dispatch (on flashinfer_cutlass or flashinfer_cutedsl moe runner backend)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`"false"`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Quantize moe of nextn layer from BF16 to FP8 when launching DeepSeek NVFP4 checkpoint</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_ENABLE_FLASHINFER_FP8_GEMM` (deprecated)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Use flashinfer kernels when running blockwise fp8 GEMM on Blackwell GPUs. **DEPRECATED**: Please use `--fp8-gemm-backend=flashinfer_trtllm` instead.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_SUPPORT_CUTLASS_BLOCK_FP8` (deprecated)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Cutlass kernels when running blockwise fp8 GEMM on Hopper or Blackwell GPUs. **DEPRECATED**: Please use `--fp8-gemm-backend=cutlass` instead.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
## Distributed Computing
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Environment Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_BLOCK_NONZERO_RANK_CHILDREN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Control blocking of non-zero rank children processes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`1`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_IS_FIRST_RANK_ON_NODE`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Indicates if the current process is the first rank on its node</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`"true"`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_PP_LAYER_PARTITION`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Pipeline parallel layer partition specification</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Not set</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_ONE_VISIBLE_DEVICE_PER_PROCESS`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Set one visible device per process for distributed computing</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Testing & Debugging (Internal/CI)
|
||||
|
||||
*These variables are primarily used for internal testing, continuous integration, or debugging.*
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Environment Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_IS_IN_CI`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Indicates if running in CI environment</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_IS_IN_CI_AMD`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Indicates running in AMD CI environment</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`0`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_TEST_RETRACT`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable retract decode testing</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_TEST_RETRACT_NO_PREFILL_BS`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>When SGLANG_TEST_RETRACT is enabled, no prefill is performed if the batch size exceeds SGLANG_TEST_RETRACT_NO_PREFILL_BS.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`2 ** 31`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_RECORD_STEP_TIME`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Record step time for profiling</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_TEST_REQUEST_TIME_STATS`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Test request time statistics</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Profiling & Benchmarking
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Environment Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_TORCH_PROFILER_DIR`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Directory for PyTorch profiler output</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`/tmp`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_PROFILE_WITH_STACK`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Set `with_stack` option (bool) for PyTorch profiler (capture stack trace)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`true`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_PROFILE_RECORD_SHAPES`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Set `record_shapes` option (bool) for PyTorch profiler (record shapes)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`true`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_OTLP_EXPORTER_SCHEDULE_DELAY_MILLIS`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Config BatchSpanProcessor.schedule_delay_millis if tracing is enabled</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`500`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_OTLP_EXPORTER_MAX_EXPORT_BATCH_SIZE`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Config BatchSpanProcessor.max_export_batch_size if tracing is enabled</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`64`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Storage & Caching
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Environment Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_WAIT_WEIGHTS_READY_TIMEOUT`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Timeout period for waiting on weights</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`120`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_DISABLE_OUTLINES_DISK_CACHE`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Disable Outlines disk cache</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`true`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_USE_CUSTOM_TRITON_KERNEL_CACHE`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Use SGLang's custom Triton kernel cache implementation for lower overheads (automatically enabled on CUDA)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`false`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Function Calling / Tool Use
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
<col style={{width: "33.3%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Environment Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>` SGLANG_TOOL_STRICT_LEVEL`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Controls strictness for tool-call parsing and validation: **Level 0** off (no strict validation); **Level 1** function strict (enables structural tag constraints for all tools, even if none have `strict=True`); **Level 2** parameter strict (enforces strict parameter validation for all tools as if all had `strict=True`).</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>` 0`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user