minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

[Docs] Rename docs_new/ to docs/ (#32123)

Browse Source 
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
zijiexia
2026-08-03 16:51:00 -07:00
committed by GitHub
co-authored by Claude Opus 4.8
parent c949e91f18
commit b819d2fb5b
491 changed files with 122 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/diffusion/Qwen-Image/Qwen-Image.mdx
+383
View File
@@ -0,0 +1,383 @@
---
title: Qwen-Image
metatags:
description: "Deploy Qwen-Image with SGLang - community contribution guide for Qwen's image generation model."
---
import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx';
import { QwenImageDeployment } from '/src/snippets/diffusion/qwen-image-deployment.jsx';
<DiffusionModelTags tags={["image", "text-to-image", "text rendering", "NVFP4"]} />
## 1. Model Introduction
[Qwen-Image](https://huggingface.co/Qwen/Qwen-Image) is a text-to-image diffusion model developed by the Qwen team.
For more details, please refer to the [official Qwen-Image HuggingFace page](https://huggingface.co/Qwen/Qwen-Image), the [Blog](https://qwenlm.github.io/blog/qwen-image/), and the [Tech Report](https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-Image/Qwen_Image.pdf).
## 2. SGLang-diffusion Installation
SGLang-diffusion offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
Please refer to the [official SGLang-diffusion installation guide](../../../docs/sglang-diffusion/installation) for installation instructions.
## 3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms and use cases.
### 3.1 Basic Configuration
Qwen-Image is a text-to-image model. The recommended launch configurations vary by hardware. SGLang supports serving Qwen-Image on NVIDIA B200, B300, H200, H100, AMD MI300X, MI325X, MI355X GPUs and Ascend A2, A3 NPUs.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform.
<QwenImageDeployment />
For the validated ModelOpt NVFP4 checkpoint on Blackwell, load the published
Qwen-Image-2512 NVFP4 repo directly:
```bash Command
sglang serve \
--model-path lmsys/qwen-image-2512-modelopt-nvfp4-sglang \
--ulysses-degree=1 \
--ring-degree=1
```
For high-resolution B200 generations, the FlashInfer CUTLASS FP4 GEMM backend
can be faster than the default TensorRT-LLM FP4 GEMM backend:
```bash Command
SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND=cutlass \
sglang generate \
--model-path lmsys/qwen-image-2512-modelopt-nvfp4-sglang \
--width 2048 --height 2048 \
--prompt "A tiny astronaut reading a book under a glass greenhouse" \
--save-output
```
### 3.2 Configuration Tips
Currently supported optimizations are listed [here](/docs/sglang-diffusion/compatibility_matrix).
- `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path.
- `--num-gpus`: Number of GPUs to use
- `--tp-size`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster)
- `--sp-degree`: Sequence parallelism size (typically should match the number of GPUs)
- `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP
- `--ring-degree`: The degree of ring attention-style SP in USP
**AMD ROCm Notes**: Requires SGLang >= v0.5.8.
## 4. API Usage
For complete API documentation, please refer to the [official API usage guide](../../../docs/sglang-diffusion/api/openai_api).
### 4.1 Generate an Image
```python Example
import base64
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:30000/v1")
response = client.images.generate(
model="Qwen/Qwen-Image",
prompt="A logo With Bold Large text: SGL Diffusion",
n=1,
response_format="b64_json",
)
# Save the generated image
image_bytes = base64.b64decode(response.data[0].b64_json)
with open("output.png", "wb") as f:
f.write(image_bytes)
```
### 4.2 Advanced Usage
#### 4.2.1 Cache-DiT Acceleration
SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](../../../docs/sglang-diffusion/cache_dit).
**Basic Usage**
```bash Command
SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path Qwen/Qwen-Image
```
**Advanced Usage**
- DBCache Parameters: DBCache controls block-level caching behavior:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "25.0%"}} />
<col style={{width: "25.0%"}} />
<col style={{width: "25.0%"}} />
<col style={{width: "25.0%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Env Variable</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Fn</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_FN`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of first blocks to always compute</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Bn</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_BN`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of last blocks to always compute</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>W</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_WARMUP`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>4</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Warmup steps before caching starts</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>R</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_RDT`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.24</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Residual difference threshold</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MC</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_MC`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>3</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Maximum continuous cached steps</td>
</tr>
</tbody>
</table>
- TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "25.0%"}} />
<col style={{width: "25.0%"}} />
<col style={{width: "25.0%"}} />
<col style={{width: "25.0%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Env Variable</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Enable</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_TAYLORSEER`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>false</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable TaylorSeer calibrator</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Order</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_TS_ORDER`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Taylor expansion order (1 or 2)</td>
</tr>
</tbody>
</table>
Combined Configuration Example:
```bash Command
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_FN=2 \
SGLANG_CACHE_DIT_BN=1 \
SGLANG_CACHE_DIT_WARMUP=4 \
SGLANG_CACHE_DIT_RDT=0.4 \
SGLANG_CACHE_DIT_MC=4 \
SGLANG_CACHE_DIT_TAYLORSEER=true \
SGLANG_CACHE_DIT_TS_ORDER=2 \
sglang serve --model-path Qwen/Qwen-Image
```
#### 4.2.2 CPU Offload
- `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory.
- `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference.
- `--vae-cpu-offload`: Use CPU offload for VAE.
- `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument".
## 5. Benchmark
Test Environment:
- Hardware: AMD Instinct MI300X GPU (1x)
- Model: Qwen/Qwen-Image
- Docker Image: lmsysorg/sglang:v0.5.8-rocm700-mi30x
- sglang diffusion version: 0.5.8
### 5.1 Speedup Benchmark
#### 5.1.1 Generate an image
<Tabs>
<Tab title="AMD MI300X">
**Server Command**:
```shell Command
sglang serve --model-path Qwen/Qwen-Image \
--ulysses-degree=1 --ring-degree=1 --port 30000
```
**Benchmark Command**:
```shell Command
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
--backend sglang-image --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1
```
**Result**:
```text Output
================= Serving Benchmark Result =================
Task: text-to-image
Model: Qwen/Qwen-Image
Dataset: vbench
--------------------------------------------------
Benchmark duration (s): 29.04
Request rate: inf
Max request concurrency: 1
Successful requests: 1/1
--------------------------------------------------
Request throughput (req/s): 0.03
Latency Mean (s): 29.0378
Latency Median (s): 29.0378
Latency P99 (s): 29.0378
--------------------------------------------------
Peak Memory Max (MB): 48018.83
Peak Memory Mean (MB): 48018.83
Peak Memory Median (MB): 48018.83
============================================================
```
</Tab>
<Tab title="Ascend A3">
**Server Command**:
```shell Command
#One A3 card has 2 npu chips
sglang serve --tp-size 2 --sp-degree 1 --model-path Qwen/Qwen-Image --num-gpus 2
```
**Benchmark Command**:
```shell Command
python -m sglang.multimodal_gen.benchmarks.bench_serving --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1
```
**Result**:
```text Output
================= Serving Benchmark Result =================
Task: text-to-image
Model: Qwen/Qwen-Image
Dataset: vbench
--------------------------------------------------
Benchmark duration (s): 36.26
Request rate: inf
Max request concurrency: 1
Successful requests: 1/1
Completed outputs: 1
Outputs per prompt: 1
--------------------------------------------------
Request throughput (req/s): 0.03
Output throughput (outputs/s): 0.03
Latency Mean (s): 36.26
Latency Median (s): 36.26
Latency P90 (s): 36.26
Latency P95 (s): 36.26
Latency P99 (s): 36.26
--------------------------------------------------
Peak Memory Max (MB): 36984.00
Peak Memory Mean (MB): 36984.00
Peak Memory Median (MB): 36984.00
------------------------------------------------------------
```
</Tab>
</Tabs>
#### 5.1.2 Generate images with high concurrency
<Tabs>
<Tab title="AMD MI300X">
**Benchmark Command**:
```shell Command
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
--backend sglang-image --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20 --port 30000
```
**Result**:
```text Output
================= Serving Benchmark Result =================
Task: text-to-image
Model: Qwen/Qwen-Image
Dataset: vbench
--------------------------------------------------
Benchmark duration (s): 300.79
Request rate: inf
Max request concurrency: 20
Successful requests: 14/20
--------------------------------------------------
Request throughput (req/s): 0.05
Latency Mean (s): 154.5368
Latency Median (s): 154.8363
Latency P99 (s): 285.4603
--------------------------------------------------
Peak Memory Max (MB): 48030.31
Peak Memory Mean (MB): 48030.30
Peak Memory Median (MB): 48030.29
============================================================
```
</Tab>
<Tab title="Ascend A3">
**Benchmark Command**:
```shell Command
python -m sglang.multimodal_gen.benchmarks.bench_serving --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20
```
**Result**:
```text Output
================= Serving Benchmark Result =================
Task: text-to-image
Model: Qwen/Qwen-Image
Dataset: vbench
--------------------------------------------------
Benchmark duration (s): 300.81
Request rate: inf
Max request concurrency: 20
Successful requests: 8/20
Completed outputs: 8
Outputs per prompt: 1
--------------------------------------------------
Request throughput (req/s): 0.03
Output throughput (outputs/s): 0.03
Latency Mean (s): 166.61
Latency Median (s): 167.02
Latency P90 (s): 270.80
Latency P95 (s): 283.48
Latency P99 (s): 293.64
--------------------------------------------------
Peak Memory Max (MB): 36984.00
Peak Memory Mean (MB): 36984.00
Peak Memory Median (MB): 36984.00
------------------------------------------------------------
```
</Tab>
</Tabs>