minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

Add new Mintlify documentation site (docs_new/) (#23001)

Browse Source 
Co-authored-by: AdityaVKochar <adityavardhankochar@gmail.com>
Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com>
Co-authored-by: adhyan-jain <adhyanjain2006@gmail.com>
Co-authored-by: Adhyan Jain <71976554+adhyan-jain@users.noreply.github.com>
Co-authored-by: Maitri-shah29 <maitrirajivshah@gmail.com>
Co-authored-by: Adarsh Shirawalmath <114558126+adarshxs@users.noreply.github.com>
Co-authored-by: Maitri Shah <shah29maitri@gmail.com>
Co-authored-by: Aditya Vardhan Kochar <80113212+AdityaVKochar@users.noreply.github.com>
Co-authored-by: Rishit Shivam <164783543+pokymono@users.noreply.github.com>
Co-authored-by: Rishitshivam <164783543+Rishitshivam@users.noreply.github.com>
Co-authored-by: IshhanKheria <ishhankheria06@gmail.com>
Co-authored-by: Ishita Joshi <ishitata.joshi@gmail.com>
Co-authored-by: Richard Chen <104477092+Richardczl98@users.noreply.github.com>
Co-authored-by: longGGGGGG <553746008@qq.com>
Co-authored-by: Richard <richardchen@radixark.ai>
Co-authored-by: Nakul Sinha <nakul.new4socials@gmail.com>
Co-authored-by: Divyam Agrawal <ludicrouslytrue@gmail.com>
Co-authored-by: Richardczl98 <Zhenlinc@stanford.edu>
Co-authored-by: Krishang Zinzuwadia <krishangzinzuwadia@gmail.com>
Co-authored-by: nimeshas <nimesha.s106@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jignas Paturu <86356085+JignasP@users.noreply.github.com>
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
This commit is contained in:
Mingyi
2026-04-20 15:10:22 -07:00
committed by GitHub
co-authored by AdityaVKochar mintlify[bot] adhyan-jain Adhyan Jain Maitri-shah29 Adarsh Shirawalmath Maitri Shah Aditya Vardhan Kochar Rishit Shivam Rishitshivam IshhanKheria Ishita Joshi Richard Chen longGGGGGG Richard Nakul Sinha Divyam Agrawal Richardczl98 Krishang Zinzuwadia nimeshas Claude Opus 4.6 github-actions[bot] Jignas Paturu zijiexia
parent 575fdc2c4c
commit a3291b5654
330 changed files with 100371 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/cookbook/diffusion/Qwen-Image/Qwen-Image-Edit.mdx
+280
View File
@@ -0,0 +1,280 @@
---
title: Qwen-Image-Edit-2511
metatags:
description: "Deploy Qwen-Image-Edit-2511 with SGLang - 20B image editing model with text rendering, character consistency, and geometric reasoning."
---
import { QwenImageEditDeployment } from '/src/snippets/diffusion/qwen-image-edit-deployment.jsx';
## 1. Model Introduction
[Qwen-Image-Edit-2511](https://huggingface.co/Qwen/Qwen-Image-Edit-2511) is an enhanced version over Qwen-Image-Edit-2509, featuring multiple improvements—including notably better consistency. Built upon the 20B Qwen-Image model, Qwen-Image-Edit-2511 successfully extends Qwen-Image's unique text rendering capabilities to image editing tasks, enabling precise text editing.
Key Enhancements in Qwen-Image-Edit-2511:
- **Mitigate Image Drift**: Reduces unwanted changes in non-edited regions of the image.
- **Improved Character Consistency**: The model can perform imaginative edits based on an input portrait while preserving the identity and visual characteristics of the subject.
- **Multi-Person Consistency**: Enhanced consistency in multi-person group photos, enabling high-fidelity fusion of two separate person images into a coherent group shot.
- **Integrated LoRA Capabilities**: Selected popular community-created LoRAs are integrated directly into the base model, unlocking their effects without extra tuning (e.g., lighting enhancement, viewpoint generation).
- **Enhanced Industrial Design Generation**: Special attention to practical engineering scenarios, including batch industrial product design and material replacement for industrial components.
- **Strengthened Geometric Reasoning**: Stronger geometric reasoning capability for generating auxiliary construction lines for design or annotation purposes.
For more details, please refer to the [official Qwen-Image-Edit-2511 HuggingFace page](https://huggingface.co/Qwen/Qwen-Image-Edit-2511), the [Blog](https://qwenlm.github.io/blog/qwen-image-edit-2511/), and the [Tech Report](https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-Image/Qwen_Image.pdf).
## 2. SGLang-diffusion Installation
SGLang-diffusion offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang-diffusion installation guide](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/install.md) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
Qwen-Image-Edit-2511 is a 20B parameter model optimized for image editing tasks. The recommended launch configurations vary by hardware.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform.
<QwenImageEditDeployment />
### 3.2 Configuration Tips
Current supported optimization all listed [here](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/support_matrix.md).
- `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path.
- `--num-gpus`: Number of GPUs to use
- `--tp-size`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster)
- `--sp-degree`: Sequence parallelism size (typically should match the number of GPUs)
- `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP
- `--ring-degree`: The degree of ring attention-style SP in USP
## 4. API Usage
For complete API documentation, please refer to the [official API usage guide](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/openai_api.md).
### 4.1 Edit an Image
```python Example
import base64
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:3000/v1")
response = client.images.edit(
model="Qwen/Qwen-Image-Edit-2511",
image=open("input.png", "rb"),
prompt="Change the color of the taxi to black.",
n=1,
response_format="b64_json",
)
# Save the edited image
image_bytes = base64.b64decode(response.data[0].b64_json)
with open("output.png", "wb") as f:
f.write(image_bytes)
```
### 4.2 Advanced Usage
#### 4.2.1 Cache-DiT Acceleration
SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/cache_dit.md).
**Basic Usage**
```bash Command
SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path Qwen/Qwen-Image-Edit-2511
```
**Advanced Usage**
- DBCache Parameters: DBCache controls block-level caching behavior:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "25.0%"}} />
<col style={{width: "25.0%"}} />
<col style={{width: "25.0%"}} />
<col style={{width: "25.0%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Env Variable</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Fn</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_FN`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of first blocks to always compute</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Bn</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_BN`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of last blocks to always compute</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>W</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_WARMUP`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>4</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Warmup steps before caching starts</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>R</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_RDT`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.24</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Residual difference threshold</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MC</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_MC`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>3</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Maximum continuous cached steps</td>
</tr>
</tbody>
</table>
- TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "25.0%"}} />
<col style={{width: "25.0%"}} />
<col style={{width: "25.0%"}} />
<col style={{width: "25.0%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Env Variable</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Enable</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_TAYLORSEER`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>false</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable TaylorSeer calibrator</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Order</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_TS_ORDER`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Taylor expansion order (1 or 2)</td>
</tr>
</tbody>
</table>
Combined Configuration Example:
```bash Command
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_FN=2 \
SGLANG_CACHE_DIT_BN=1 \
SGLANG_CACHE_DIT_WARMUP=4 \
SGLANG_CACHE_DIT_RDT=0.4 \
SGLANG_CACHE_DIT_MC=4 \
SGLANG_CACHE_DIT_TAYLORSEER=true \
SGLANG_CACHE_DIT_TS_ORDER=2 \
sglang serve --model-path Qwen/Qwen-Image-Edit-2511
```
#### 4.2.2 CPU Offload
- `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory.
- `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference.
- `--image-encoder-cpu-offload`: Use CPU offload for image encoder inference.
- `--vae-cpu-offload`: Use CPU offload for VAE.
- `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument".
## 5. Benchmark
Test Environment:
- Hardware: NVIDIA B200 GPU (1x)
- Model: Qwen/Qwen-Image-Edit-2511
- sglang diffusion version: 0.5.6.post2
### 5.1 Speedup Benchmark
#### 5.1.1 Edit a image
**Server Command**:
```shell Command
sglang serve --model-path Qwen/Qwen-Image-Edit-2511 --port 30000
```
**Benchmark Command**:
```shell Command
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
--backend sglang-image --dataset vbench --task ti2i --num-prompts 1 --max-concurrency 1
```
**Result**:
```text Output
================= Serving Benchmark Result =================
Backend: sglang-image
Model: Qwen/Qwen-Image-Edit-2511
Dataset: vbench
Task: ti2i
--------------------------------------------------
Benchmark duration (s): 35.31
Request rate: inf
Max request concurrency: 1
Successful requests: 1/1
--------------------------------------------------
Request throughput (req/s): 0.03
Latency Mean (s): 35.3053
Latency Median (s): 35.3053
Latency P99 (s): 35.3053
--------------------------------------------------
Peak Memory Max (MB): 47959.35
Peak Memory Mean (MB): 47959.35
Peak Memory Median (MB): 47959.35
============================================================
```
#### 5.1.2 Edit a image with high concurrency
**Benchmark Command**:
```shell Command
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
--backend sglang-image --dataset vbench --task ti2i --num-prompts 20 --max-concurrency 20
```
**Result**:
```text Output
================= Serving Benchmark Result =================
Backend: sglang-image
Model: Qwen/Qwen-Image-Edit-2511
Dataset: vbench
Task: ti2i
--------------------------------------------------
Benchmark duration (s): 286.11
Request rate: inf
Max request concurrency: 20
Successful requests: 20/20
--------------------------------------------------
Request throughput (req/s): 0.07
Latency Mean (s): 150.0428
Latency Median (s): 150.0600
Latency P99 (s): 283.3843
--------------------------------------------------
Peak Memory Max (MB): 47971.82
Peak Memory Mean (MB): 47971.49
Peak Memory Median (MB): 47971.29
============================================================
```
docs_new/cookbook/diffusion/Qwen-Image/Qwen-Image.mdx
+271
View File
@@ -0,0 +1,271 @@
---
title: Qwen-Image
metatags:
description: "Deploy Qwen-Image with SGLang - community contribution guide for Qwen's image generation model."
---
import { QwenImageDeployment } from '/src/snippets/diffusion/qwen-image-deployment.jsx';
## 1. Model Introduction
[Qwen-Image](https://huggingface.co/Qwen/Qwen-Image) is a text-to-image diffusion model developed by the Qwen team.
For more details, please refer to the [official Qwen-Image HuggingFace page](https://huggingface.co/Qwen/Qwen-Image), the [Blog](https://qwenlm.github.io/blog/qwen-image/), and the [Tech Report](https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-Image/Qwen_Image.pdf).
## 2. SGLang-diffusion Installation
SGLang-diffusion offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang-diffusion installation guide](../../../docs/sglang-diffusion/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
Qwen-Image is a text-to-image model. The recommended launch configurations vary by hardware.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform.
<QwenImageDeployment />
### 3.2 Configuration Tips
Current supported optimization all listed [here](../../../docs/sglang-diffusion/attention-backends#platform-support-matrix).
- `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path.
- `--num-gpus`: Number of GPUs to use
- `--tp-size`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster)
- `--sp-degree`: Sequence parallelism size (typically should match the number of GPUs)
- `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP
- `--ring-degree`: The degree of ring attention-style SP in USP
**AMD ROCm Notes**: Requires SGLang >= v0.5.8.
## 4. API Usage
For complete API documentation, please refer to the [official API usage guide](../../../docs/sglang-diffusion/api/openai-api).
### 4.1 Generate an Image
```python Example
import base64
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:30000/v1")
response = client.images.generate(
model="Qwen/Qwen-Image",
prompt="A logo With Bold Large text: SGL Diffusion",
n=1,
response_format="b64_json",
)
# Save the generated image
image_bytes = base64.b64decode(response.data[0].b64_json)
with open("output.png", "wb") as f:
f.write(image_bytes)
```
### 4.2 Advanced Usage
#### 4.2.1 Cache-DiT Acceleration
SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](../../../docs/sglang-diffusion/cache-dit).
**Basic Usage**
```bash Command
SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path Qwen/Qwen-Image
```
**Advanced Usage**
- DBCache Parameters: DBCache controls block-level caching behavior:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "25.0%"}} />
<col style={{width: "25.0%"}} />
<col style={{width: "25.0%"}} />
<col style={{width: "25.0%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Env Variable</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Fn</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_FN`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of first blocks to always compute</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Bn</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_BN`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of last blocks to always compute</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>W</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_WARMUP`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>4</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Warmup steps before caching starts</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>R</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_RDT`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.24</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Residual difference threshold</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MC</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_MC`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>3</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Maximum continuous cached steps</td>
</tr>
</tbody>
</table>
- TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "25.0%"}} />
<col style={{width: "25.0%"}} />
<col style={{width: "25.0%"}} />
<col style={{width: "25.0%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Env Variable</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Enable</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_TAYLORSEER`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>false</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable TaylorSeer calibrator</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Order</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_TS_ORDER`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Taylor expansion order (1 or 2)</td>
</tr>
</tbody>
</table>
Combined Configuration Example:
```bash Command
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_FN=2 \
SGLANG_CACHE_DIT_BN=1 \
SGLANG_CACHE_DIT_WARMUP=4 \
SGLANG_CACHE_DIT_RDT=0.4 \
SGLANG_CACHE_DIT_MC=4 \
SGLANG_CACHE_DIT_TAYLORSEER=true \
SGLANG_CACHE_DIT_TS_ORDER=2 \
sglang serve --model-path Qwen/Qwen-Image
```
#### 4.2.2 CPU Offload
- `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory.
- `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference.
- `--vae-cpu-offload`: Use CPU offload for VAE.
- `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument".
## 5. Benchmark
Test Environment:
- Hardware: AMD Instinct MI300X GPU (1x)
- Model: Qwen/Qwen-Image
- Docker Image: lmsysorg/sglang:v0.5.8-rocm700-mi30x
- sglang diffusion version: 0.5.8
### 5.1 Speedup Benchmark
#### 5.1.1 Generate an image
**Server Command**:
```shell Command
sglang serve --model-path Qwen/Qwen-Image \
--ulysses-degree=1 --ring-degree=1 --port 30000
```
**Benchmark Command**:
```shell Command
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
--backend sglang-image --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1
```
**Result**:
```text Output
================= Serving Benchmark Result =================
Task: text-to-image
Model: Qwen/Qwen-Image
Dataset: vbench
--------------------------------------------------
Benchmark duration (s): 29.04
Request rate: inf
Max request concurrency: 1
Successful requests: 1/1
--------------------------------------------------
Request throughput (req/s): 0.03
Latency Mean (s): 29.0378
Latency Median (s): 29.0378
Latency P99 (s): 29.0378
--------------------------------------------------
Peak Memory Max (MB): 48018.83
Peak Memory Mean (MB): 48018.83
Peak Memory Median (MB): 48018.83
============================================================
```
#### 5.1.2 Generate images with high concurrency
**Benchmark Command**:
```shell Command
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
--backend sglang-image --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20
```
**Result**:
```text Output
================= Serving Benchmark Result =================
Task: text-to-image
Model: Qwen/Qwen-Image
Dataset: vbench
--------------------------------------------------
Benchmark duration (s): 300.79
Request rate: inf
Max request concurrency: 20
Successful requests: 14/20
--------------------------------------------------
Request throughput (req/s): 0.05
Latency Mean (s): 154.5368
Latency Median (s): 154.8363
Latency P99 (s): 285.4603
--------------------------------------------------
Peak Memory Max (MB): 48030.31
Peak Memory Mean (MB): 48030.30
Peak Memory Median (MB): 48030.29
============================================================
```