Add new Mintlify documentation site (docs_new/) (#23001)
Co-authored-by: AdityaVKochar <adityavardhankochar@gmail.com> Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com> Co-authored-by: adhyan-jain <adhyanjain2006@gmail.com> Co-authored-by: Adhyan Jain <71976554+adhyan-jain@users.noreply.github.com> Co-authored-by: Maitri-shah29 <maitrirajivshah@gmail.com> Co-authored-by: Adarsh Shirawalmath <114558126+adarshxs@users.noreply.github.com> Co-authored-by: Maitri Shah <shah29maitri@gmail.com> Co-authored-by: Aditya Vardhan Kochar <80113212+AdityaVKochar@users.noreply.github.com> Co-authored-by: Rishit Shivam <164783543+pokymono@users.noreply.github.com> Co-authored-by: Rishitshivam <164783543+Rishitshivam@users.noreply.github.com> Co-authored-by: IshhanKheria <ishhankheria06@gmail.com> Co-authored-by: Ishita Joshi <ishitata.joshi@gmail.com> Co-authored-by: Richard Chen <104477092+Richardczl98@users.noreply.github.com> Co-authored-by: longGGGGGG <553746008@qq.com> Co-authored-by: Richard <richardchen@radixark.ai> Co-authored-by: Nakul Sinha <nakul.new4socials@gmail.com> Co-authored-by: Divyam Agrawal <ludicrouslytrue@gmail.com> Co-authored-by: Richardczl98 <Zhenlinc@stanford.edu> Co-authored-by: Krishang Zinzuwadia <krishangzinzuwadia@gmail.com> Co-authored-by: nimeshas <nimesha.s106@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jignas Paturu <86356085+JignasP@users.noreply.github.com> Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
This commit is contained in:
Mingyi
co-authored by
AdityaVKochar
mintlify[bot]
adhyan-jain
Adhyan Jain
Maitri-shah29
Adarsh Shirawalmath
Maitri Shah
Aditya Vardhan Kochar
Rishit Shivam
Rishitshivam
IshhanKheria
Ishita Joshi
Richard Chen
longGGGGGG
Richard
Nakul Sinha
Divyam Agrawal
Richardczl98
Krishang Zinzuwadia
nimeshas
Claude Opus 4.6
github-actions[bot]
Jignas Paturu
zijiexia
AdityaVKochar
mintlify[bot]
adhyan-jain
Adhyan Jain
Maitri-shah29
Adarsh Shirawalmath
Maitri Shah
Aditya Vardhan Kochar
Rishit Shivam
Rishitshivam
IshhanKheria
Ishita Joshi
Richard Chen
longGGGGGG
Richard
Nakul Sinha
Divyam Agrawal
Richardczl98
Krishang Zinzuwadia
nimeshas
Claude Opus 4.6
github-actions[bot]
Jignas Paturu
zijiexia
parent
575fdc2c4c
commit
a3291b5654
@@ -0,0 +1,294 @@
|
||||
---
|
||||
title: FLUX
|
||||
metatags:
|
||||
description: "Deploy FLUX diffusion models with SGLang - 12B/32B rectified flow transformers for high-quality text-to-image generation."
|
||||
---
|
||||
|
||||
import { FluxDeployment } from '/src/snippets/diffusion/flux-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[FLUX](https://blackforestlabs.ai/) is a family of rectified flow transformer models developed by Black Forest Labs for high-quality image generation from text descriptions.
|
||||
|
||||
[FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) is a 12 billion parameter rectified flow transformer capable of generating images from text descriptions.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Cutting-edge Output Quality**: Second only to the state-of-the-art FLUX.1 [pro] model
|
||||
- **Competitive Prompt Following**: Matches the performance of closed-source alternatives
|
||||
- **Guidance Distillation**: Trained using guidance distillation for improved efficiency
|
||||
- **Open Weights**: Available for personal, scientific, and commercial purposes under the FLUX [dev] Non-Commercial License
|
||||
|
||||
[FLUX.2-dev](https://huggingface.co/black-forest-labs/FLUX.2-dev) is a 32 billion parameter rectified flow transformer capable of generating, editing, and combining images based on text instructions.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **State-of-the-art Performance**: Leading open model in text-to-image generation, single-reference editing, and multi-reference editing
|
||||
- **No Finetuning Required**: Character, object, and style reference without additional training in one model
|
||||
- **Guidance Distillation**: Trained using guidance distillation for improved efficiency
|
||||
- **Open Weights**: Available for personal, scientific, and commercial purposes under the FLUX [dev] Non-Commercial License
|
||||
|
||||
For more details, please refer to the [FLUX.1-dev HuggingFace page](https://huggingface.co/black-forest-labs/FLUX.1-dev), [FLUX.2-dev HuggingFace page](https://huggingface.co/black-forest-labs/FLUX.2-dev), and the [official blog post](https://blackforestlabs.ai/announcing-black-forest-labs/).
|
||||
|
||||
## 2. SGLang-diffusion Installation
|
||||
|
||||
SGLang-diffusion offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang-diffusion installation guide](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/install.md) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
FLUX models are optimized for high-quality image generation. The recommended launch configurations vary by hardware and model version.
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model version. SGLang supports serving FLUX on NVIDIA B200, H200, H100, and AMD MI355X, MI325X, MI300X GPUs.
|
||||
|
||||
<FluxDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
Current supported optimization all listed [here](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/support_matrix.md).
|
||||
|
||||
- `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path.
|
||||
- `--num-gpus`: Number of GPUs to use
|
||||
- `--tp-size`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster)
|
||||
- `--sp-degree`: Sequence parallelism size (typically should match the number of GPUs)
|
||||
- `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP
|
||||
- `--ring-degree`: The degree of ring attention-style SP in USP
|
||||
|
||||
## 4. API Usage
|
||||
|
||||
For complete API documentation, please refer to the [official API usage guide](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/openai_api.md).
|
||||
|
||||
### 4.1 Generate an Image
|
||||
|
||||
```python Example
|
||||
import base64
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:3000/v1")
|
||||
|
||||
response = client.images.generate(
|
||||
model="black-forest-labs/FLUX.1-dev",
|
||||
prompt="A cat holding a sign that says hello world",
|
||||
size="1024x1024",
|
||||
n=1,
|
||||
response_format="b64_json",
|
||||
)
|
||||
|
||||
# Save the generated image
|
||||
image_bytes = base64.b64decode(response.data[0].b64_json)
|
||||
with open("output.png", "wb") as f:
|
||||
f.write(image_bytes)
|
||||
```
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Cache-DiT Acceleration
|
||||
|
||||
SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/cache_dit.md).
|
||||
|
||||
**Basic Usage**
|
||||
|
||||
```bash Command
|
||||
SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path black-forest-labs/FLUX.1-dev
|
||||
```
|
||||
|
||||
**Advanced Usage**
|
||||
|
||||
- DBCache Parameters: DBCache controls block-level caching behavior:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Env Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Fn</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_FN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of first blocks to always compute</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Bn</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_BN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of last blocks to always compute</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>W</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_WARMUP`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Warmup steps before caching starts</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>R</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_RDT`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.24</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Residual difference threshold</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MC</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_MC`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>3</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Maximum continuous cached steps</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
- TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Env Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Enable</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_TAYLORSEER`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>false</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable TaylorSeer calibrator</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Order</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_TS_ORDER`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Taylor expansion order (1 or 2)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
Combined Configuration Example:
|
||||
|
||||
```bash Command
|
||||
SGLANG_CACHE_DIT_ENABLED=true \
|
||||
SGLANG_CACHE_DIT_FN=2 \
|
||||
SGLANG_CACHE_DIT_BN=1 \
|
||||
SGLANG_CACHE_DIT_WARMUP=4 \
|
||||
SGLANG_CACHE_DIT_RDT=0.4 \
|
||||
SGLANG_CACHE_DIT_MC=4 \
|
||||
SGLANG_CACHE_DIT_TAYLORSEER=true \
|
||||
SGLANG_CACHE_DIT_TS_ORDER=2 \
|
||||
sglang serve --model-path black-forest-labs/FLUX.1-dev
|
||||
```
|
||||
|
||||
#### 4.2.2 CPU Offload
|
||||
|
||||
- `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory.
|
||||
- `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference.
|
||||
- `--vae-cpu-offload`: Use CPU offload for VAE.
|
||||
- `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument".
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speedup Benchmark
|
||||
|
||||
#### 5.1.1 Generate a image
|
||||
|
||||
Test Environment:
|
||||
|
||||
- Hardware: NVIDIA B200 GPU (1x)
|
||||
- Model: black-forest-labs/FLUX.1-dev
|
||||
- sglang diffusion version: 0.5.6.post2
|
||||
|
||||
**Server Command**:
|
||||
|
||||
```shell Command
|
||||
sglang serve --model-path black-forest-labs/FLUX.1-dev --port 30000
|
||||
```
|
||||
|
||||
**Benchmark Command**:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
|
||||
--backend sglang-video --dataset vbench --task t2v --num-prompts 1 --max-concurrency 1
|
||||
```
|
||||
|
||||
**Result**:
|
||||
|
||||
```text Output
|
||||
================= Serving Benchmark Result =================
|
||||
Backend: sglang-image
|
||||
Model: black-forest-labs/FLUX.1-dev
|
||||
Dataset: vbench
|
||||
Task: t2v
|
||||
--------------------------------------------------
|
||||
Benchmark duration (s): 50.97
|
||||
Request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 1/1
|
||||
--------------------------------------------------
|
||||
Request throughput (req/s): 0.02
|
||||
Latency Mean (s): 50.9681
|
||||
Latency Median (s): 50.9681
|
||||
Latency P99 (s): 50.9681
|
||||
--------------------------------------------------
|
||||
Peak Memory Max (MB): 27905.19
|
||||
Peak Memory Mean (MB): 27905.19
|
||||
Peak Memory Median (MB): 27905.19
|
||||
============================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Generate images with high concurrency
|
||||
|
||||
**Server Command** :
|
||||
|
||||
```shell Command
|
||||
sglang serve --model-path black-forest-labs/FLUX.1-dev --port 30000
|
||||
```
|
||||
|
||||
**Benchmark Command** :
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
|
||||
--backend sglang-image --dataset vbench --task t2v --num-prompts 20 --max-concurrency 20
|
||||
```
|
||||
|
||||
**Result** :
|
||||
|
||||
```text Output
|
||||
================= Serving Benchmark Result =================
|
||||
Backend: sglang-image
|
||||
Model: black-forest-labs/FLUX.1-dev
|
||||
Dataset: vbench
|
||||
Task: t2v
|
||||
--------------------------------------------------
|
||||
Benchmark duration (s): 111.79
|
||||
Request rate: inf
|
||||
Max request concurrency: 20
|
||||
Successful requests: 20/20
|
||||
--------------------------------------------------
|
||||
Request throughput (req/s): 0.18
|
||||
Latency Mean (s): 67.0646
|
||||
Latency Median (s): 66.9691
|
||||
Latency P99 (s): 110.8949
|
||||
--------------------------------------------------
|
||||
Peak Memory Max (MB): 27917.19
|
||||
Peak Memory Mean (MB): 27916.59
|
||||
Peak Memory Median (MB): 27917.19
|
||||
============================================================
|
||||
```
|
||||
@@ -0,0 +1,268 @@
|
||||
---
|
||||
title: MOVA
|
||||
metatags:
|
||||
description: "Deploy MOVA with SGLang - simultaneous video and audio generation with asymmetric dual-tower architecture, precise lip-sync, and environment-aware sound effects."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[MOVA](https://github.com/OpenMOSS/MOVA) (MOSS Video and Audio) is a foundation model developed by the SII-OpenMOSS Team, designed to break the "silent era" of open-source video generation. Unlike cascaded pipelines that generate sound as an afterthought, MOVA synthesizes video and audio simultaneously in a single inference pass for perfect alignment. It adopts an Asymmetric Dual-Tower Architecture, fusing pre-trained video and audio towers through a bidirectional cross-attention mechanism to maintain tight synchronization between video and audio during generation.
|
||||
|
||||
[MOVA-360p](https://huggingface.co/OpenMOSS-Team/MOVA-360p) is suitable for fast inference and resource-constrained environments. [MOVA-720p](https://huggingface.co/OpenMOSS-Team/MOVA-720p) provides higher resolution video generation. Both versions support generating up to 8 seconds of video-audio content.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Native Bimodal Generation**: Generates high-fidelity video and synchronized audio in a single inference pass, eliminating error accumulation from cascaded pipelines
|
||||
- **Precise Lip-Sync**: Achieves state-of-the-art performance in multilingual lip-synchronization (LSE-D: 7.094, LSE-C: 7.452 with Dual CFG on Verse-Bench Set3)
|
||||
- **Environment-Aware Sound Effects**: Generates corresponding environmental sound effects including physical interaction sounds, ambient sounds, and spatial/textural sound feedback
|
||||
- **Fully Open-Source**: Model weights, inference code, training pipelines, and LoRA fine-tuning scripts are all open-sourced
|
||||
|
||||
For more details, please refer to the [MOVA-360p HuggingFace page](https://huggingface.co/OpenMOSS-Team/MOVA-360p), the [MOVA-720p HuggingFace page](https://huggingface.co/OpenMOSS-Team/MOVA-720p), the [GitHub repository](https://github.com/OpenMOSS/MOVA), and the [technical report (arXiv)](https://arxiv.org/abs/2602.08794).
|
||||
|
||||
## 2. SGLang-diffusion Installation
|
||||
|
||||
SGLang-diffusion offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang-diffusion installation guide](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/install.md) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
MOVA supports both online serving and CLI generation modes. The recommended launch configurations vary by hardware and resolution.
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform.
|
||||
|
||||
import { MOVADeployment } from '/src/snippets/diffusion/mova-deployment.jsx'
|
||||
|
||||
<MOVADeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
Current supported optimization all listed [here](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/support_matrix.md).
|
||||
|
||||
- `--num-gpus`: Number of GPUs to use
|
||||
- `--tp`: Tensor parallelism size (should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster)
|
||||
- `--ring-degree`: The degree of ring attention-style SP in USP
|
||||
- `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP
|
||||
- `--adjust-frames`: Whether to adjust frames automatically (set to `false` for MOVA)
|
||||
- `--enable-torch-compile`: Enable torch.compile for faster inference
|
||||
|
||||
## 4. API Usage
|
||||
|
||||
For complete API documentation, please refer to the [official API usage guide](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/openai_api.md).
|
||||
|
||||
### 4.1 CLI Generation (sglang generate)
|
||||
|
||||
```bash Command
|
||||
sglang generate \
|
||||
--model-path OpenMOSS-Team/MOVA-720p \
|
||||
--prompt "A man in a blue blazer and glasses speaks in a formal indoor setting, \
|
||||
framed by wooden furniture and a filled bookshelf. \
|
||||
Quiet room acoustics underscore his measured tone as he delivers his remarks. \
|
||||
At one point, he says, \"I would also believe that this advance in AI recently wasn't unexpected.\"" \
|
||||
--image-path "<YOUR-IMAGE-PATH>" \
|
||||
--adjust-frames false \
|
||||
--num-gpus 8 \
|
||||
--ring-degree 2 \
|
||||
--ulysses-degree 4 \
|
||||
--num-frames 193 \
|
||||
--fps 24 \
|
||||
--seed 67 \
|
||||
--num-inference-steps 25 \
|
||||
--enable-torch-compile \
|
||||
--save-output
|
||||
```
|
||||
|
||||
### 4.2 Generate a Video
|
||||
|
||||
```bash Command
|
||||
curl -X POST "http://0.0.0.0:30002/v1/videos" \
|
||||
-F "prompt=A man in a blue blazer and glasses speaks in a formal indoor setting, framed by wooden furniture and a filled bookshelf. Quiet room acoustics underscore his measured tone as he delivers his remarks. At one point, he says, \"I would also believe that this advance in AI recently wasn't unexpected.\"" \
|
||||
-F "input_reference=@<YOUR-IMAGE-PATH>" \
|
||||
-F "size=640x352" \
|
||||
-F "num_frames=193" \
|
||||
-F "fps=24" \
|
||||
-F "seed=67" \
|
||||
-F "guidance_scale=5.0" \
|
||||
-F "num_inference_steps=25" \
|
||||
-o create_video.json
|
||||
```
|
||||
|
||||
### 4.3 Advanced Usage
|
||||
|
||||
#### 4.3.1 Cache-DiT Acceleration
|
||||
|
||||
SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/cache_dit.md).
|
||||
|
||||
**Basic Usage**
|
||||
|
||||
```bash Command
|
||||
SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path OpenMOSS-Team/MOVA-720p
|
||||
```
|
||||
|
||||
**Advanced Usage**
|
||||
|
||||
- DBCache Parameters: DBCache controls block-level caching behavior:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Env Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Fn</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_FN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of first blocks to always compute</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Bn</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_BN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of last blocks to always compute</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>W</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_WARMUP`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Warmup steps before caching starts</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>R</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_RDT`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.24</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Residual difference threshold</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MC</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_MC`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>3</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Maximum continuous cached steps</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
- TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Env Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Enable</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_TAYLORSEER`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>false</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable TaylorSeer calibrator</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Order</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_TS_ORDER`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Taylor expansion order (1 or 2)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
Combined Configuration Example:
|
||||
|
||||
```bash Command
|
||||
SGLANG_CACHE_DIT_ENABLED=true \
|
||||
SGLANG_CACHE_DIT_FN=2 \
|
||||
SGLANG_CACHE_DIT_BN=1 \
|
||||
SGLANG_CACHE_DIT_WARMUP=4 \
|
||||
SGLANG_CACHE_DIT_RDT=0.4 \
|
||||
SGLANG_CACHE_DIT_MC=4 \
|
||||
SGLANG_CACHE_DIT_TAYLORSEER=true \
|
||||
SGLANG_CACHE_DIT_TS_ORDER=2 \
|
||||
sglang serve --model-path OpenMOSS-Team/MOVA-720p
|
||||
```
|
||||
|
||||
#### 4.3.2 CPU Offload
|
||||
|
||||
- `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory.
|
||||
- `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference.
|
||||
- `--vae-cpu-offload`: Use CPU offload for VAE.
|
||||
- `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument".
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speedup Benchmark
|
||||
|
||||
#### 5.1.1 Generate a video
|
||||
|
||||
Test Environment:
|
||||
|
||||
- Hardware: NVIDIA H200 x 8
|
||||
- git revision: 443b1a8
|
||||
- Model: OpenMOSS-Team/MOVA-720p
|
||||
|
||||
**Server Command**:
|
||||
|
||||
```bash Command
|
||||
sglang serve --model-path OpenMOSS-Team/MOVA-720p --port 30002 \
|
||||
--adjust-frames false --num-gpus 8 --ring-degree 2 --ulysses-degree 4 \
|
||||
--tp 1 --enable-torch-compile
|
||||
```
|
||||
|
||||
**Benchmark Command**:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
|
||||
--task image-to-video --dataset vbench --num-prompts 1 --max-concurrency 1 \
|
||||
--port 30002
|
||||
```
|
||||
|
||||
**Result**:
|
||||
```text Output
|
||||
================= Serving Benchmark Result =================
|
||||
Task: image-to-video
|
||||
Model: OpenMOSS-Team/MOVA-720p
|
||||
Dataset: vbench
|
||||
--------------------------------------------------
|
||||
Benchmark duration (s): 590.76
|
||||
Request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 1/1
|
||||
--------------------------------------------------
|
||||
Request throughput (req/s): 0.00
|
||||
Latency Mean (s): 590.7549
|
||||
Latency Median (s): 590.7549
|
||||
Latency P99 (s): 590.7549
|
||||
--------------------------------------------------
|
||||
Peak Memory Max (MB): 74996.00
|
||||
Peak Memory Mean (MB): 74996.00
|
||||
Peak Memory Median (MB): 74996.00
|
||||
============================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Generate videos with high concurrency
|
||||
|
||||
**Server Command**:
|
||||
|
||||
```bash Command
|
||||
sglang serve --model-path OpenMOSS-Team/MOVA-720p --port 30002 \
|
||||
--adjust-frames false --num-gpus 8 --ring-degree 2 --ulysses-degree 4 \
|
||||
--tp 1 --enable-torch-compile
|
||||
```
|
||||
|
||||
**Benchmark Command**:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
|
||||
--task image-to-video --dataset vbench --num-prompts 20 --max-concurrency 20 \
|
||||
--port 30002
|
||||
```
|
||||
@@ -0,0 +1,280 @@
|
||||
---
|
||||
title: Qwen-Image-Edit-2511
|
||||
metatags:
|
||||
description: "Deploy Qwen-Image-Edit-2511 with SGLang - 20B image editing model with text rendering, character consistency, and geometric reasoning."
|
||||
---
|
||||
|
||||
import { QwenImageEditDeployment } from '/src/snippets/diffusion/qwen-image-edit-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Qwen-Image-Edit-2511](https://huggingface.co/Qwen/Qwen-Image-Edit-2511) is an enhanced version over Qwen-Image-Edit-2509, featuring multiple improvements—including notably better consistency. Built upon the 20B Qwen-Image model, Qwen-Image-Edit-2511 successfully extends Qwen-Image's unique text rendering capabilities to image editing tasks, enabling precise text editing.
|
||||
|
||||
Key Enhancements in Qwen-Image-Edit-2511:
|
||||
|
||||
- **Mitigate Image Drift**: Reduces unwanted changes in non-edited regions of the image.
|
||||
- **Improved Character Consistency**: The model can perform imaginative edits based on an input portrait while preserving the identity and visual characteristics of the subject.
|
||||
- **Multi-Person Consistency**: Enhanced consistency in multi-person group photos, enabling high-fidelity fusion of two separate person images into a coherent group shot.
|
||||
- **Integrated LoRA Capabilities**: Selected popular community-created LoRAs are integrated directly into the base model, unlocking their effects without extra tuning (e.g., lighting enhancement, viewpoint generation).
|
||||
- **Enhanced Industrial Design Generation**: Special attention to practical engineering scenarios, including batch industrial product design and material replacement for industrial components.
|
||||
- **Strengthened Geometric Reasoning**: Stronger geometric reasoning capability for generating auxiliary construction lines for design or annotation purposes.
|
||||
|
||||
For more details, please refer to the [official Qwen-Image-Edit-2511 HuggingFace page](https://huggingface.co/Qwen/Qwen-Image-Edit-2511), the [Blog](https://qwenlm.github.io/blog/qwen-image-edit-2511/), and the [Tech Report](https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-Image/Qwen_Image.pdf).
|
||||
|
||||
## 2. SGLang-diffusion Installation
|
||||
|
||||
SGLang-diffusion offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang-diffusion installation guide](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/install.md) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
Qwen-Image-Edit-2511 is a 20B parameter model optimized for image editing tasks. The recommended launch configurations vary by hardware.
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform.
|
||||
|
||||
<QwenImageEditDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
Current supported optimization all listed [here](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/support_matrix.md).
|
||||
|
||||
- `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path.
|
||||
- `--num-gpus`: Number of GPUs to use
|
||||
- `--tp-size`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster)
|
||||
- `--sp-degree`: Sequence parallelism size (typically should match the number of GPUs)
|
||||
- `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP
|
||||
- `--ring-degree`: The degree of ring attention-style SP in USP
|
||||
|
||||
## 4. API Usage
|
||||
|
||||
For complete API documentation, please refer to the [official API usage guide](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/openai_api.md).
|
||||
|
||||
### 4.1 Edit an Image
|
||||
|
||||
```python Example
|
||||
import base64
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:3000/v1")
|
||||
|
||||
response = client.images.edit(
|
||||
model="Qwen/Qwen-Image-Edit-2511",
|
||||
image=open("input.png", "rb"),
|
||||
prompt="Change the color of the taxi to black.",
|
||||
n=1,
|
||||
response_format="b64_json",
|
||||
)
|
||||
|
||||
# Save the edited image
|
||||
image_bytes = base64.b64decode(response.data[0].b64_json)
|
||||
with open("output.png", "wb") as f:
|
||||
f.write(image_bytes)
|
||||
```
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Cache-DiT Acceleration
|
||||
|
||||
SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/cache_dit.md).
|
||||
|
||||
**Basic Usage**
|
||||
|
||||
```bash Command
|
||||
SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path Qwen/Qwen-Image-Edit-2511
|
||||
```
|
||||
|
||||
**Advanced Usage**
|
||||
|
||||
- DBCache Parameters: DBCache controls block-level caching behavior:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Env Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Fn</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_FN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of first blocks to always compute</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Bn</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_BN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of last blocks to always compute</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>W</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_WARMUP`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Warmup steps before caching starts</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>R</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_RDT`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.24</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Residual difference threshold</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MC</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_MC`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>3</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Maximum continuous cached steps</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
- TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Env Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Enable</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_TAYLORSEER`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>false</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable TaylorSeer calibrator</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Order</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_TS_ORDER`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Taylor expansion order (1 or 2)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
Combined Configuration Example:
|
||||
|
||||
```bash Command
|
||||
SGLANG_CACHE_DIT_ENABLED=true \
|
||||
SGLANG_CACHE_DIT_FN=2 \
|
||||
SGLANG_CACHE_DIT_BN=1 \
|
||||
SGLANG_CACHE_DIT_WARMUP=4 \
|
||||
SGLANG_CACHE_DIT_RDT=0.4 \
|
||||
SGLANG_CACHE_DIT_MC=4 \
|
||||
SGLANG_CACHE_DIT_TAYLORSEER=true \
|
||||
SGLANG_CACHE_DIT_TS_ORDER=2 \
|
||||
sglang serve --model-path Qwen/Qwen-Image-Edit-2511
|
||||
```
|
||||
|
||||
#### 4.2.2 CPU Offload
|
||||
|
||||
- `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory.
|
||||
- `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference.
|
||||
- `--image-encoder-cpu-offload`: Use CPU offload for image encoder inference.
|
||||
- `--vae-cpu-offload`: Use CPU offload for VAE.
|
||||
- `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument".
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
Test Environment:
|
||||
|
||||
- Hardware: NVIDIA B200 GPU (1x)
|
||||
- Model: Qwen/Qwen-Image-Edit-2511
|
||||
- sglang diffusion version: 0.5.6.post2
|
||||
|
||||
### 5.1 Speedup Benchmark
|
||||
|
||||
#### 5.1.1 Edit a image
|
||||
|
||||
**Server Command**:
|
||||
|
||||
```shell Command
|
||||
sglang serve --model-path Qwen/Qwen-Image-Edit-2511 --port 30000
|
||||
```
|
||||
|
||||
**Benchmark Command**:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
|
||||
--backend sglang-image --dataset vbench --task ti2i --num-prompts 1 --max-concurrency 1
|
||||
```
|
||||
|
||||
**Result**:
|
||||
|
||||
```text Output
|
||||
================= Serving Benchmark Result =================
|
||||
Backend: sglang-image
|
||||
Model: Qwen/Qwen-Image-Edit-2511
|
||||
Dataset: vbench
|
||||
Task: ti2i
|
||||
--------------------------------------------------
|
||||
Benchmark duration (s): 35.31
|
||||
Request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 1/1
|
||||
--------------------------------------------------
|
||||
Request throughput (req/s): 0.03
|
||||
Latency Mean (s): 35.3053
|
||||
Latency Median (s): 35.3053
|
||||
Latency P99 (s): 35.3053
|
||||
--------------------------------------------------
|
||||
Peak Memory Max (MB): 47959.35
|
||||
Peak Memory Mean (MB): 47959.35
|
||||
Peak Memory Median (MB): 47959.35
|
||||
============================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Edit a image with high concurrency
|
||||
|
||||
**Benchmark Command**:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
|
||||
--backend sglang-image --dataset vbench --task ti2i --num-prompts 20 --max-concurrency 20
|
||||
```
|
||||
|
||||
**Result**:
|
||||
|
||||
```text Output
|
||||
================= Serving Benchmark Result =================
|
||||
Backend: sglang-image
|
||||
Model: Qwen/Qwen-Image-Edit-2511
|
||||
Dataset: vbench
|
||||
Task: ti2i
|
||||
--------------------------------------------------
|
||||
Benchmark duration (s): 286.11
|
||||
Request rate: inf
|
||||
Max request concurrency: 20
|
||||
Successful requests: 20/20
|
||||
--------------------------------------------------
|
||||
Request throughput (req/s): 0.07
|
||||
Latency Mean (s): 150.0428
|
||||
Latency Median (s): 150.0600
|
||||
Latency P99 (s): 283.3843
|
||||
--------------------------------------------------
|
||||
Peak Memory Max (MB): 47971.82
|
||||
Peak Memory Mean (MB): 47971.49
|
||||
Peak Memory Median (MB): 47971.29
|
||||
============================================================
|
||||
```
|
||||
@@ -0,0 +1,271 @@
|
||||
---
|
||||
title: Qwen-Image
|
||||
metatags:
|
||||
description: "Deploy Qwen-Image with SGLang - community contribution guide for Qwen's image generation model."
|
||||
---
|
||||
|
||||
import { QwenImageDeployment } from '/src/snippets/diffusion/qwen-image-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Qwen-Image](https://huggingface.co/Qwen/Qwen-Image) is a text-to-image diffusion model developed by the Qwen team.
|
||||
|
||||
For more details, please refer to the [official Qwen-Image HuggingFace page](https://huggingface.co/Qwen/Qwen-Image), the [Blog](https://qwenlm.github.io/blog/qwen-image/), and the [Tech Report](https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-Image/Qwen_Image.pdf).
|
||||
|
||||
## 2. SGLang-diffusion Installation
|
||||
|
||||
SGLang-diffusion offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang-diffusion installation guide](../../../docs/sglang-diffusion/installation) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
Qwen-Image is a text-to-image model. The recommended launch configurations vary by hardware.
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform.
|
||||
|
||||
<QwenImageDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
Current supported optimization all listed [here](../../../docs/sglang-diffusion/attention-backends#platform-support-matrix).
|
||||
|
||||
- `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path.
|
||||
- `--num-gpus`: Number of GPUs to use
|
||||
- `--tp-size`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster)
|
||||
- `--sp-degree`: Sequence parallelism size (typically should match the number of GPUs)
|
||||
- `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP
|
||||
- `--ring-degree`: The degree of ring attention-style SP in USP
|
||||
|
||||
**AMD ROCm Notes**: Requires SGLang >= v0.5.8.
|
||||
|
||||
## 4. API Usage
|
||||
|
||||
For complete API documentation, please refer to the [official API usage guide](../../../docs/sglang-diffusion/api/openai-api).
|
||||
|
||||
### 4.1 Generate an Image
|
||||
|
||||
```python Example
|
||||
import base64
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:30000/v1")
|
||||
|
||||
response = client.images.generate(
|
||||
model="Qwen/Qwen-Image",
|
||||
prompt="A logo With Bold Large text: SGL Diffusion",
|
||||
n=1,
|
||||
response_format="b64_json",
|
||||
)
|
||||
|
||||
# Save the generated image
|
||||
image_bytes = base64.b64decode(response.data[0].b64_json)
|
||||
with open("output.png", "wb") as f:
|
||||
f.write(image_bytes)
|
||||
```
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Cache-DiT Acceleration
|
||||
|
||||
SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](../../../docs/sglang-diffusion/cache-dit).
|
||||
|
||||
**Basic Usage**
|
||||
|
||||
```bash Command
|
||||
SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path Qwen/Qwen-Image
|
||||
```
|
||||
|
||||
**Advanced Usage**
|
||||
|
||||
- DBCache Parameters: DBCache controls block-level caching behavior:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Env Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Fn</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_FN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of first blocks to always compute</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Bn</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_BN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of last blocks to always compute</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>W</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_WARMUP`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Warmup steps before caching starts</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>R</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_RDT`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.24</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Residual difference threshold</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MC</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_MC`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>3</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Maximum continuous cached steps</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
- TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Env Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Enable</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_TAYLORSEER`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>false</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable TaylorSeer calibrator</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Order</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_TS_ORDER`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Taylor expansion order (1 or 2)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
Combined Configuration Example:
|
||||
|
||||
```bash Command
|
||||
SGLANG_CACHE_DIT_ENABLED=true \
|
||||
SGLANG_CACHE_DIT_FN=2 \
|
||||
SGLANG_CACHE_DIT_BN=1 \
|
||||
SGLANG_CACHE_DIT_WARMUP=4 \
|
||||
SGLANG_CACHE_DIT_RDT=0.4 \
|
||||
SGLANG_CACHE_DIT_MC=4 \
|
||||
SGLANG_CACHE_DIT_TAYLORSEER=true \
|
||||
SGLANG_CACHE_DIT_TS_ORDER=2 \
|
||||
sglang serve --model-path Qwen/Qwen-Image
|
||||
```
|
||||
|
||||
#### 4.2.2 CPU Offload
|
||||
|
||||
- `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory.
|
||||
- `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference.
|
||||
- `--vae-cpu-offload`: Use CPU offload for VAE.
|
||||
- `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument".
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
Test Environment:
|
||||
|
||||
- Hardware: AMD Instinct MI300X GPU (1x)
|
||||
- Model: Qwen/Qwen-Image
|
||||
- Docker Image: lmsysorg/sglang:v0.5.8-rocm700-mi30x
|
||||
- sglang diffusion version: 0.5.8
|
||||
|
||||
### 5.1 Speedup Benchmark
|
||||
|
||||
#### 5.1.1 Generate an image
|
||||
|
||||
**Server Command**:
|
||||
|
||||
```shell Command
|
||||
sglang serve --model-path Qwen/Qwen-Image \
|
||||
--ulysses-degree=1 --ring-degree=1 --port 30000
|
||||
```
|
||||
|
||||
**Benchmark Command**:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
|
||||
--backend sglang-image --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1
|
||||
```
|
||||
|
||||
**Result**:
|
||||
|
||||
```text Output
|
||||
================= Serving Benchmark Result =================
|
||||
Task: text-to-image
|
||||
Model: Qwen/Qwen-Image
|
||||
Dataset: vbench
|
||||
--------------------------------------------------
|
||||
Benchmark duration (s): 29.04
|
||||
Request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 1/1
|
||||
--------------------------------------------------
|
||||
Request throughput (req/s): 0.03
|
||||
Latency Mean (s): 29.0378
|
||||
Latency Median (s): 29.0378
|
||||
Latency P99 (s): 29.0378
|
||||
--------------------------------------------------
|
||||
Peak Memory Max (MB): 48018.83
|
||||
Peak Memory Mean (MB): 48018.83
|
||||
Peak Memory Median (MB): 48018.83
|
||||
============================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Generate images with high concurrency
|
||||
|
||||
**Benchmark Command**:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
|
||||
--backend sglang-image --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20
|
||||
```
|
||||
|
||||
**Result**:
|
||||
|
||||
```text Output
|
||||
================= Serving Benchmark Result =================
|
||||
Task: text-to-image
|
||||
Model: Qwen/Qwen-Image
|
||||
Dataset: vbench
|
||||
--------------------------------------------------
|
||||
Benchmark duration (s): 300.79
|
||||
Request rate: inf
|
||||
Max request concurrency: 20
|
||||
Successful requests: 14/20
|
||||
--------------------------------------------------
|
||||
Request throughput (req/s): 0.05
|
||||
Latency Mean (s): 154.5368
|
||||
Latency Median (s): 154.8363
|
||||
Latency P99 (s): 285.4603
|
||||
--------------------------------------------------
|
||||
Peak Memory Max (MB): 48030.31
|
||||
Peak Memory Mean (MB): 48030.30
|
||||
Peak Memory Median (MB): 48030.29
|
||||
============================================================
|
||||
```
|
||||
@@ -0,0 +1,91 @@
|
||||
---
|
||||
title: "Diffusion Cookbook"
|
||||
description: "Cookbook recipes for running diffusion models with SGLang"
|
||||
metatags:
|
||||
description: "Explore SGLang diffusion cookbook structure, categories, and contribution guidance for image and video generation recipes."
|
||||
---
|
||||
|
||||
# SGLang Diffusion Cookbook
|
||||
|
||||
<div style={{display: 'flex', gap: '8px'}}>
|
||||
<a href="https://opensource.org/licenses/Apache-2.0"><img src="https://img.shields.io/badge/License-Apache_2.0-blue.svg" alt="License" /></a>
|
||||
<a href="https://github.com/sgl-project/sgl-cookbook/pulls"><img src="https://img.shields.io/badge/PRs-welcome-brightgreen.svg" alt="PRs Welcome" /></a>
|
||||
</div>
|
||||
|
||||
Create a comprehensive cookbook for diffusion models in SGLang, demonstrating SGLang's performance advantages for image and video generation workloads.
|
||||
|
||||
## 🎯 What You'll Find Here
|
||||
|
||||
This cookbook aggregates battle-tested SGLang recipes covering:
|
||||
|
||||
- **Models**: Mainstream Image and Video generation Models
|
||||
- **Use Cases**: Inference serving, deployment strategies
|
||||
- **Hardware**: GPU and CPU configurations, optimization for different accelerators
|
||||
- **Best Practices**: Configuration templates, performance tuning, troubleshooting guides
|
||||
|
||||
Each recipe provides step-by-step instructions to help you quickly implement SGLang solutions for your specific requirements.
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
1. Browse the recipe index above to find your model
|
||||
2. Follow the step-by-step instructions in each guide
|
||||
3. Adapt configurations to your specific hardware and requirements
|
||||
4. Join our community to share feedback and improvements
|
||||
|
||||
The sglang diffusion cookbook directory structure are shown below:
|
||||
|
||||
```text Example
|
||||
sgl-cookbook/docs/diffusion/
|
||||
├── README.md # Main cookbook (this file)
|
||||
├── Qwen-Image/ # Qwen-Image series models docs
|
||||
│ ├── Qwen-Image.md
|
||||
│ └── Qwen-Image-Edit.md
|
||||
├── Wan/ # Wan series models docs
|
||||
│ ├── Wan2.1.md
|
||||
│ └── Wan2.2.md
|
||||
├── Z-Image/ # Z-Image series models docs
|
||||
│ └── Z-Image-Turbo.md
|
||||
└── ...
|
||||
```
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
We believe the best documentation comes from practitioners. Whether you've optimized SGLang for a specific model, solved a tricky deployment challenge, or discovered performance improvements, we encourage you to contribute your recipes!
|
||||
|
||||
**💪How to Contribute**
|
||||
|
||||
- Comment below if interested (mention which role)
|
||||
- Join discussion on implementation details
|
||||
- Fork repo and work on assigned section
|
||||
- Submit PR following SGLang cookbook standards
|
||||
- Iterate based on review feedback
|
||||
|
||||
**To contribute:**
|
||||
|
||||
```shell Command
|
||||
# Fork the repo and clone locally
|
||||
git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git
|
||||
cd sglang-cookbook
|
||||
|
||||
# Create a new branch
|
||||
git checkout -b add-my-recipe
|
||||
|
||||
# Add your recipe following the template in DeepSeek-V3.2
|
||||
# Submit a PR!
|
||||
```
|
||||
|
||||
## 📖 Resources
|
||||
|
||||
- [SGLang GitHub](https://github.com/sgl-project/sglang)
|
||||
- [SGLang Documentation](https://sgl-project.github.io)
|
||||
- [SGLANG Diffusion Documentation](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/README.md)
|
||||
- [SLACK Channel](https://sgl-fru7574.slack.com/archives/C07GLLLESNR)
|
||||
- [Community Slack/Discord](https://discord.gg/MpEEuAeb)
|
||||
|
||||
## 📄 License
|
||||
|
||||
This project is licensed under the Apache License 2.0 - see the [LICENSE](https://github.com/sgl-project/sgl-cookbook/blob/main/LICENSE) file for details.
|
||||
|
||||
---
|
||||
|
||||
**Let's build this resource together!** 🚀 Star the repo and contribute your recipes to help the SGLang community grow.
|
||||
@@ -0,0 +1,268 @@
|
||||
---
|
||||
title: Wan2.1
|
||||
metatags:
|
||||
description: "Deploy Wan2.1 video generation models with SGLang - community contribution guide for Wan Video's diffusion models."
|
||||
---
|
||||
|
||||
import { Wan21Deployment } from '/src/snippets/diffusion/wan21-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Wan2.1 series](https://github.com/Wan-Video/Wan2.1) is an open and advanced suite of large-scale video generative models from Wan-AI.
|
||||
|
||||
Key characteristics:
|
||||
|
||||
- **State-of-the-art video quality**: Consistently outperforms many open-source and commercial video models on internal and public benchmarks, especially for motion richness and temporal consistency.
|
||||
- **Consumer GPU friendly**: The T2V-1.3B variant can generate 5-second 480P videos on consumer GPUs with modest VRAM requirements.
|
||||
- **Multi-capability suite**: Supports Text-to-Video (T2V), Image-to-Video (I2V), video editing, text-to-image, and video-to-audio generation.
|
||||
- **Robust text rendering**: First-generation Wan model capable of generating both Chinese and English text in videos with strong readability.
|
||||
- **Powerful Wan-VAE**: A 3D causal VAE that encodes/decodes long 1080P videos while preserving temporal information, enabling efficient high-resolution video generation.
|
||||
|
||||
For more details, refer to the official Wan2.1 resources:
|
||||
|
||||
- **GitHub**: [Wan-Video/Wan2.1](https://github.com/Wan-Video/Wan2.1)
|
||||
- **Hugging Face collection**: [Wan-AI Wan2.1](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B)
|
||||
|
||||
## 2. SGLang-diffusion Installation
|
||||
|
||||
SGLang-diffusion offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang-diffusion installation guide](../../../docs/sglang-diffusion/installation) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
The Wan2.1 series offers models in multiple sizes and resolutions, optimized for different hardware platforms. The recommended launch configurations vary by hardware and model size.
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate an appropriate deployment command for your model variant and options.
|
||||
|
||||
<Wan21Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
Current supported optimization options are listed in the [SGLang diffusion support matrix](../../../docs/sglang-diffusion/attention-backends#platform-support-matrix).
|
||||
|
||||
- `--vae-path`: Path to a custom VAE model or HuggingFace model ID. If not specified, the VAE will be loaded from the main model path.
|
||||
- `--num-gpus {NUM_GPUS}`: Number of GPUs to use.
|
||||
- `--tp-size {TP_SIZE}`: Tensor parallelism size (for the encoder/DiT; keep \(\leq 1\) if relying heavily on CPU offload).
|
||||
- `--sp-degree {SP_SIZE}`: Sequence parallelism degree.
|
||||
- `--ulysses-degree {ULYSSES_DEGREE}`: Degree of DeepSpeed-Ulysses-style SP in USP.
|
||||
- `--ring-degree {RING_DEGREE}`: Degree of ring attention-style SP in USP.
|
||||
- `--text-encoder-cpu-offload`, `--dit-cpu-offload`, `--vae-cpu-offload`: Use CPU offload to reduce peak GPU memory when needed.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For more API usage and request examples, please refer to:
|
||||
[SGLang Diffusion OpenAI API](../../../docs/sglang-diffusion/api/openai-api)
|
||||
|
||||
#### 4.1.1 Launch a server and then send requests
|
||||
|
||||
```bash Command
|
||||
sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers --port 30000
|
||||
|
||||
curl http://127.0.0.1:30000/v1/images/generations \
|
||||
-o >(jq -r '.data[0].b64_json' | base64 --decode > example.png) \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer $OPENAI_API_KEY" \
|
||||
-d '{
|
||||
"model": "Wan-AI/Wan2.1-T2V-14B-Diffusers",
|
||||
"prompt": "A cute baby sea otter",
|
||||
"n": 1,
|
||||
"size": "1024x1024",
|
||||
"response_format": "b64_json"
|
||||
}'
|
||||
```
|
||||
|
||||
#### 4.1.2 Generate a video without launching a server
|
||||
|
||||
```bash Command
|
||||
SERVER_ARGS=(
|
||||
--model-path Wan-AI/Wan2.1-T2V-14B-Diffusers
|
||||
--text-encoder-cpu-offload
|
||||
--pin-cpu-memory
|
||||
--num-gpus 4
|
||||
--ulysses-degree=2
|
||||
--enable-cfg-parallel
|
||||
)
|
||||
|
||||
SAMPLING_ARGS=(
|
||||
--prompt "A curious raccoon"
|
||||
--save-output
|
||||
--output-path outputs
|
||||
--output-file-name "A curious raccoon.mp4"
|
||||
)
|
||||
|
||||
sglang generate "${SERVER_ARGS[@]}" "${SAMPLING_ARGS[@]}"
|
||||
```
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Cache-DiT Acceleration
|
||||
|
||||
SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve significant inference speedups with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](../../../docs/sglang-diffusion/cache-dit).
|
||||
|
||||
**Basic Usage**
|
||||
|
||||
```bash Command
|
||||
SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers
|
||||
```
|
||||
|
||||
**Advanced Usage**
|
||||
|
||||
Combined Configuration Example:
|
||||
```bash Command
|
||||
SGLANG_CACHE_DIT_ENABLED=true \
|
||||
SGLANG_CACHE_DIT_FN=2 \
|
||||
SGLANG_CACHE_DIT_BN=1 \
|
||||
SGLANG_CACHE_DIT_WARMUP=4 \
|
||||
SGLANG_CACHE_DIT_RDT=0.4 \
|
||||
SGLANG_CACHE_DIT_MC=4 \
|
||||
SGLANG_CACHE_DIT_TAYLORSEER=true \
|
||||
SGLANG_CACHE_DIT_TS_ORDER=2 \
|
||||
sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers
|
||||
```
|
||||
|
||||
#### 4.2.2 GPU Optimization
|
||||
|
||||
- `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if you run out of memory with FSDP.
|
||||
- `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference.
|
||||
- `--image-encoder-cpu-offload`: Use CPU offload for image encoder inference.
|
||||
- `--vae-cpu-offload`: Use CPU offload for VAE.
|
||||
- `--pin-cpu-memory`: Pin memory for CPU offload. Use as a workaround if you see "CUDA error: invalid argument".
|
||||
|
||||
#### 4.2.3 Supported LoRA Registry
|
||||
|
||||
SGLang supports applying Wan2.1 LoRA adapters on top of base models:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "50%"}} />
|
||||
<col style={{width: "50%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>origin model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>supported LoRA</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[Wan-AI/Wan2.1-T2V-14B](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[NIVEDAN/wan2.1-lora](https://huggingface.co/NIVEDAN/wan2.1-lora)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[Wan-AI/Wan2.1-I2V-14B-720P](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-720P)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[valiantcat/Wan2.1-Fight-LoRA](https://huggingface.co/valiantcat/Wan2.1-Fight-LoRA)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Example**:
|
||||
|
||||
```bash Command
|
||||
sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers --port 30000 \
|
||||
--lora-path NIVEDAN/wan2.1-lora
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
Test Environment:
|
||||
|
||||
- Hardware: AMD MI300X GPU (1x)
|
||||
- Model: Wan-AI/Wan2.1-T2V-14B-Diffusers
|
||||
- SGLang Docker Image Version: 0.5.9
|
||||
|
||||
### 5.1 How to Run Benchmarks with SGLang
|
||||
|
||||
You can use the built-in SGLang diffusion benchmark script to evaluate Wan2.1 performance on your hardware.
|
||||
|
||||
#### 5.1.1 Generate a single video
|
||||
|
||||
**Server Command**:
|
||||
|
||||
```bash Command
|
||||
sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers
|
||||
```
|
||||
|
||||
**Benchmark Command**:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
|
||||
--backend sglang-video --dataset vbench --task text-to-video --num-prompts 1 --max-concurrency 1
|
||||
```
|
||||
|
||||
**Result**:
|
||||
|
||||
```text Output
|
||||
================= Serving Benchmark Result =================
|
||||
Task: text-to-video
|
||||
Model: Wan-AI/Wan2.1-T2V-14B-Diffusers
|
||||
Dataset: vbench
|
||||
--------------------------------------------------
|
||||
Benchmark duration (s): 1958.41
|
||||
Request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 1/1
|
||||
--------------------------------------------------
|
||||
Request throughput (req/s): 0.00
|
||||
Latency Mean (s): 1958.4059
|
||||
Latency Median (s): 1958.4059
|
||||
Latency P99 (s): 1958.4059
|
||||
--------------------------------------------------
|
||||
Peak Memory Max (MB): 59662.00
|
||||
Peak Memory Mean (MB): 59662.00
|
||||
Peak Memory Median (MB): 59662.00
|
||||
============================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Generate videos with Cache-DiT acceleration
|
||||
|
||||
**Server Command**:
|
||||
|
||||
```bash Command
|
||||
SGLANG_CACHE_DIT_ENABLED=true \
|
||||
SGLANG_CACHE_DIT_FN=2 \
|
||||
SGLANG_CACHE_DIT_BN=1 \
|
||||
SGLANG_CACHE_DIT_WARMUP=4 \
|
||||
SGLANG_CACHE_DIT_RDT=0.4 \
|
||||
SGLANG_CACHE_DIT_MC=4 \
|
||||
SGLANG_CACHE_DIT_TAYLORSEER=true \
|
||||
SGLANG_CACHE_DIT_TS_ORDER=2 \
|
||||
sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers
|
||||
```
|
||||
|
||||
**Benchmark Command**:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
|
||||
--backend sglang-video --dataset vbench --task text-to-video --num-prompts 1 --max-concurrency 1
|
||||
```
|
||||
|
||||
**Result**:
|
||||
|
||||
```text Output
|
||||
================= Serving Benchmark Result =================
|
||||
Task: text-to-video
|
||||
Model: Wan-AI/Wan2.1-T2V-14B-Diffusers
|
||||
Dataset: vbench
|
||||
--------------------------------------------------
|
||||
Benchmark duration (s): 556.99
|
||||
Request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 1/1
|
||||
--------------------------------------------------
|
||||
Request throughput (req/s): 0.00
|
||||
Latency Mean (s): 556.9885
|
||||
Latency Median (s): 556.9885
|
||||
Latency P99 (s): 556.9885
|
||||
--------------------------------------------------
|
||||
Peak Memory Max (MB): 69306.00
|
||||
Peak Memory Mean (MB): 69306.00
|
||||
Peak Memory Median (MB): 69306.00
|
||||
============================================================
|
||||
```
|
||||
@@ -0,0 +1,346 @@
|
||||
---
|
||||
title: Wan2.2
|
||||
metatags:
|
||||
description: "Deploy Wan2.2 video generation models with SGLang - MoE architecture, cinematic aesthetics, and efficient 720P@24fps generation."
|
||||
---
|
||||
|
||||
import { Wan22Deployment } from '/src/snippets/diffusion/wan22-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Wan2.2 series](https://github.com/Wan-Video/Wan2.2) are the most popular and open and advanced large-scale video generative models.
|
||||
|
||||
This generation delivers comprehensive upgrades across the board:
|
||||
|
||||
- **Effective MoE Architecture**: Introduces a Mixture-of-Experts (MoE) architecture into video diffusion models. By separating the denoising process cross timesteps with specialized powerful expert models, this enlarges the overall model capacity while maintaining the same computational cost.
|
||||
- **Cinematic-level Aesthetics**: Incorporates meticulously curated aesthetic data, complete with detailed labels for lighting, composition, contrast, color tone, and more. This allows for more precise and controllable cinematic style generation, facilitating the creation of videos with customizable aesthetic preferences.
|
||||
- **Complex Motion Generation**: Trained on a significantly larger data, with +65.6% more images and +83.2% more videos. This expansion notably enhances the model's generalization across multiple dimensions such as motions, semantics, and aesthetics, achieving TOP performance among all open-sourced and closed-sourced models.
|
||||
- **Efficient High-Definition Hybrid TI2V**: Open-sources a 5B model built with our advanced Wan2.2-VAE that achieves a compression ratio of 16×16×4. This model supports both text-to-video and image-to-video generation at 720P resolution with 24fps and can also run on consumer-grade graphics cards like 4090. It is one of the fastest 720P@24fps models currently available, capable of serving both the industrial and academic sectors simultaneously.
|
||||
|
||||
For more details, please refer to the [official Wan2.2 GitHub Repository](https://github.com/Wan-Video/Wan2.2).
|
||||
|
||||
## 2. SGLang-diffusion Installation
|
||||
|
||||
SGLang-diffusion offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang-diffusion installation guide](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/install.md) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
The Wan2.2 series offers models in various sizes, architectures and input types, optimized for different hardware platforms. The recommended launch configurations vary by hardware and model size.
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size. SGLang supports serving Wan2.2 on NVIDIA B200, H200 and AMD MI300X, MI325X and MI355X GPUs.
|
||||
|
||||
<Wan22Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
Current supported optimization all listed [here](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/support_matrix.md).
|
||||
|
||||
- `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path.
|
||||
- `--num-gpus {NUM_GPUS}`: Number of GPUs to use
|
||||
- `--tp-size {TP_SIZE}`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster)
|
||||
- `--sp-degree {SP_SIZE}`: Sequence parallelism size (typically should match the number of GPUs)
|
||||
- `--ulysses-degree {ULYSSES_DEGREE}`: The degree of DeepSpeed-Ulysses-style SP in USP
|
||||
- `--ring-degree {RING_DEGREE}`: The degree of ring attention-style SP in USP
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For more API usage and request examples, please refer to:
|
||||
[SGLang Diffusion OpenAI API](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/openai_api.md)
|
||||
|
||||
#### 4.1.1 Launch a server and then send requests
|
||||
|
||||
```shell Command
|
||||
sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers --port 3000
|
||||
|
||||
curl http://127.0.0.1:3000/v1/images/generations \
|
||||
-o >(jq -r '.data[0].b64_json' | base64 --decode > example.png) \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer $OPENAI_API_KEY" \
|
||||
-d '{
|
||||
"model": "black-forest-labs/FLUX.1-dev",
|
||||
"prompt": "A cute baby sea otter",
|
||||
"n": 1,
|
||||
"size": "1024x1024",
|
||||
"response_format": "b64_json"
|
||||
}'
|
||||
```
|
||||
|
||||
#### 4.1.2 Generate a video without launching a server
|
||||
|
||||
```shell Command
|
||||
SERVER_ARGS=(
|
||||
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers
|
||||
--text-encoder-cpu-offload
|
||||
--pin-cpu-memory
|
||||
--num-gpus 4
|
||||
--ulysses-degree=2
|
||||
--enable-cfg-parallel
|
||||
)
|
||||
|
||||
SAMPLING_ARGS=(
|
||||
--prompt "A curious raccoon"
|
||||
--save-output
|
||||
--output-path outputs
|
||||
--output-file-name "A curious raccoon.mp4"
|
||||
)
|
||||
|
||||
sglang generate "${SERVER_ARGS[@]}" "${SAMPLING_ARGS[@]}"
|
||||
|
||||
```
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Cache-DiT Acceleration
|
||||
|
||||
SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/cache/cache_dit.md).
|
||||
|
||||
**Basic Usage**
|
||||
|
||||
```shell Command
|
||||
SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers
|
||||
```
|
||||
|
||||
**Advanced Usage**
|
||||
|
||||
- DBCache Parameters: DBCache controls block-level caching behavior:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Env Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Fn</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_FN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of first blocks to always compute</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Bn</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_BN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of last blocks to always compute</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>W</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_WARMUP`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Warmup steps before caching starts</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>R</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_RDT`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.24</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Residual difference threshold</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MC</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_MC`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>3</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Maximum continuous cached steps</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
- TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Env Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Enable</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_TAYLORSEER`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>false</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable TaylorSeer calibrator</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Order</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_TS_ORDER`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Taylor expansion order (1 or 2)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
Combined Configuration Example:
|
||||
|
||||
```shell Command
|
||||
SGLANG_CACHE_DIT_ENABLED=true \
|
||||
SGLANG_CACHE_DIT_FN=2 \
|
||||
SGLANG_CACHE_DIT_BN=1 \
|
||||
SGLANG_CACHE_DIT_WARMUP=4 \
|
||||
SGLANG_CACHE_DIT_RDT=0.4 \
|
||||
SGLANG_CACHE_DIT_MC=4 \
|
||||
SGLANG_CACHE_DIT_TAYLORSEER=true \
|
||||
SGLANG_CACHE_DIT_TS_ORDER=2 \
|
||||
sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers
|
||||
```
|
||||
|
||||
#### 4.2.2 GPU Optimization
|
||||
|
||||
- `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory with FSDP.
|
||||
- `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference. Enable if run out of memory with FSDP.
|
||||
- `--image-encoder-cpu-offload`: Use CPU offload for image encoder inference. Enable if run out of memory with FSDP.
|
||||
- `--vae-cpu-offload`: Use CPU offload for VAE. Enable if run out of memory.
|
||||
- `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument".
|
||||
|
||||
#### 4.2.3 Supported LoRA Registry
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "50%"}} />
|
||||
<col style={{width: "50%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>origin model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>supported LoRA</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[Wan-AI/Wan2.2-I2V-A14B-Diffusers](https://huggingface.co/Wan-AI/Wan2.2-I2V-A14B-Diffusers)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[lightx2v/Wan2.2-Distill-Loras](https://huggingface.co/lightx2v/Wan2.2-Distill-Loras)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[Wan-AI/Wan2.2-T2V-A14B-Diffusers](https://huggingface.co/Wan-AI/Wan2.2-T2V-A14B-Diffusers)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[Cseti/wan2.2-14B-Arcane_Jinx-lora-v1](https://huggingface.co/Cseti/wan2.2-14B-Arcane_Jinx-lora-v1)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
**Example**:
|
||||
```shell Command
|
||||
sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers --port 3000 \
|
||||
--lora-path Cseti/wan2.2-14B-Arcane_Jinx-lora-v1
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
Test Environment:
|
||||
|
||||
- Hardware: NVIDIA B200 GPU (1x)
|
||||
- Model: Wan-AI/Wan2.2-T2V-A14B-Diffusers
|
||||
- sglang diffusion version: 0.5.6.post2
|
||||
|
||||
### 5.1 Speedup Benchmark
|
||||
|
||||
#### 5.1.1 Generate a video
|
||||
|
||||
**Server Command**:
|
||||
|
||||
```shell Command
|
||||
sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers
|
||||
```
|
||||
|
||||
**Benchmark Command**:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
|
||||
--backend sglang-video --dataset vbench --task t2v --num-prompts 1 --max-concurrency 1
|
||||
```
|
||||
|
||||
**Result**:
|
||||
|
||||
```text Output
|
||||
================= Serving Benchmark Result =================
|
||||
Backend: sglang-video
|
||||
Model: Wan-AI/Wan2.2-T2V-A14B-Diffusers
|
||||
Dataset: vbench
|
||||
Task: t2v
|
||||
--------------------------------------------------
|
||||
Benchmark duration (s): 630.43
|
||||
Request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 1/1
|
||||
--------------------------------------------------
|
||||
Request throughput (req/s): 0.00
|
||||
Latency Mean (s): 630.4277
|
||||
Latency Median (s): 630.4277
|
||||
Latency P99 (s): 630.4277
|
||||
--------------------------------------------------
|
||||
Peak Memory Max (MB): 62627.41
|
||||
Peak Memory Mean (MB): 62627.41
|
||||
Peak Memory Median (MB): 62627.41
|
||||
|
||||
============================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Generate videos with high concurrency
|
||||
|
||||
**Server Command**:
|
||||
|
||||
```shell Command
|
||||
SGLANG_CACHE_DIT_ENABLED=true \
|
||||
SGLANG_CACHE_DIT_FN=2 \
|
||||
SGLANG_CACHE_DIT_BN=1 \
|
||||
SGLANG_CACHE_DIT_WARMUP=4 \
|
||||
SGLANG_CACHE_DIT_RDT=0.4 \
|
||||
SGLANG_CACHE_DIT_MC=4 \
|
||||
SGLANG_CACHE_DIT_TAYLORSEER=true \
|
||||
SGLANG_CACHE_DIT_TS_ORDER=2 \
|
||||
sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers
|
||||
```
|
||||
|
||||
**Benchmark Command**:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
|
||||
--backend sglang-video --dataset vbench --task t2v --num-prompts 20 --max-concurrency 20
|
||||
```
|
||||
|
||||
**Result**:
|
||||
|
||||
```text Output
|
||||
================= Serving Benchmark Result =================
|
||||
Backend: sglang-video
|
||||
Model: Wan-AI/Wan2.2-T2V-A14B-Diffusers
|
||||
Dataset: vbench
|
||||
Task: t2v
|
||||
--------------------------------------------------
|
||||
Benchmark duration (s): 5163.21
|
||||
Request rate: inf
|
||||
Max request concurrency: 20
|
||||
Successful requests: 20/20
|
||||
--------------------------------------------------
|
||||
Request throughput (req/s): 0.00
|
||||
Latency Mean (s): 2739.7695
|
||||
Latency Median (s): 2742.0673
|
||||
Latency P99 (s): 5121.6331
|
||||
--------------------------------------------------
|
||||
Peak Memory Max (MB): 72523.56
|
||||
Peak Memory Mean (MB): 70253.34
|
||||
Peak Memory Median (MB): 70824.46
|
||||
|
||||
============================================================
|
||||
```
|
||||
@@ -0,0 +1,281 @@
|
||||
---
|
||||
title: Z-Image-Turbo
|
||||
metatags:
|
||||
description: "Deploy Z-Image-Turbo with SGLang - community contribution guide for Z-Image's fast image generation model."
|
||||
---
|
||||
|
||||
import { ZImageTurboDeployment } from '/src/snippets/diffusion/zimage-turbo-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Z-Image](https://github.com/Tongyi-MAI/Z-Image) is a powerful and highly efficient image generation model family with 6B parameters, developed by Tongyi-MAI. It adopts a Scalable Single-Stream DiT (S3-DiT) architecture, where text, visual semantic tokens, and image VAE tokens are concatenated at the sequence level to serve as a unified input stream, maximizing parameter efficiency compared to dual-stream approaches.
|
||||
|
||||
[Z-Image-Turbo](https://huggingface.co/Tongyi-MAI/Z-Image-Turbo) is a distilled version of Z-Image that matches or exceeds leading competitors with only 8 NFEs (Number of Function Evaluations). It is powered by two core techniques: **Decoupled-DMD** (few-step distillation) and **DMDR** (fusing DMD with Reinforcement Learning).
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Sub-second Inference Latency**: Achieves sub-second inference on enterprise-grade H800 GPUs and fits comfortably within 16GB VRAM consumer devices
|
||||
- **Photorealistic Image Generation**: Excels in high-quality photorealistic image generation with rich aesthetics
|
||||
- **Bilingual Text Rendering**: Supports accurate bilingual text rendering in both English and Chinese
|
||||
- **Robust Instruction Adherence**: Strong prompt following and instruction adherence capabilities
|
||||
- **#1 Open-Source Model**: Ranked 8th overall and #1 among open-source models on the [Artificial Analysis Text-to-Image Leaderboard](https://artificialanalysis.ai/image/leaderboard/text-to-image)
|
||||
|
||||
For more details, please refer to the [Z-Image-Turbo HuggingFace page](https://huggingface.co/Tongyi-MAI/Z-Image-Turbo), the [GitHub repository](https://github.com/Tongyi-MAI/Z-Image), and the [technical report (arXiv)](https://arxiv.org/abs/2511.22699).
|
||||
|
||||
## 2. SGLang-diffusion Installation
|
||||
|
||||
SGLang-diffusion offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang-diffusion installation guide](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/install.md) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
Z-Image-Turbo is optimized for high-quality image generation with only 8 inference steps. The recommended launch configurations vary by hardware.
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform.
|
||||
|
||||
<ZImageTurboDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
Current supported optimization all listed [here](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/support_matrix.md).
|
||||
|
||||
- `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path.
|
||||
- `--num-gpus`: Number of GPUs to use
|
||||
- `--tp-size`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster)
|
||||
- `--sp-degree`: Sequence parallelism size (typically should match the number of GPUs)
|
||||
- `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP
|
||||
- `--ring-degree`: The degree of ring attention-style SP in USP
|
||||
|
||||
**AMD ROCm Notes**: Requires SGLang >= v0.5.8.
|
||||
|
||||
## 4. API Usage
|
||||
|
||||
For complete API documentation, please refer to the [official API usage guide](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/openai_api.md).
|
||||
|
||||
### 4.1 Generate an Image
|
||||
|
||||
```python Example
|
||||
import base64
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:30000/v1")
|
||||
|
||||
response = client.images.generate(
|
||||
model="Tongyi-MAI/Z-Image-Turbo",
|
||||
prompt="A logo With Bold Large text: SGL Diffusion",
|
||||
n=1,
|
||||
response_format="b64_json",
|
||||
)
|
||||
|
||||
# Save the generated image
|
||||
image_bytes = base64.b64decode(response.data[0].b64_json)
|
||||
with open("output.png", "wb") as f:
|
||||
f.write(image_bytes)
|
||||
```
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Cache-DiT Acceleration
|
||||
|
||||
SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/docs/cache_dit.md).
|
||||
|
||||
**Basic Usage**
|
||||
|
||||
```bash Command
|
||||
SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path Tongyi-MAI/Z-Image-Turbo
|
||||
```
|
||||
|
||||
**Advanced Usage**
|
||||
|
||||
- DBCache Parameters: DBCache controls block-level caching behavior:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Env Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Fn</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_FN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of first blocks to always compute</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Bn</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_BN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of last blocks to always compute</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>W</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_WARMUP`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Warmup steps before caching starts</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>R</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_RDT`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.24</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Residual difference threshold</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MC</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_MC`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>3</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Maximum continuous cached steps</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
- TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Env Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Enable</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_TAYLORSEER`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>false</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable TaylorSeer calibrator</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Order</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_TS_ORDER`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Taylor expansion order (1 or 2)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
Combined Configuration Example:
|
||||
|
||||
```bash Command
|
||||
SGLANG_CACHE_DIT_ENABLED=true \
|
||||
SGLANG_CACHE_DIT_FN=2 \
|
||||
SGLANG_CACHE_DIT_BN=1 \
|
||||
SGLANG_CACHE_DIT_WARMUP=4 \
|
||||
SGLANG_CACHE_DIT_RDT=0.4 \
|
||||
SGLANG_CACHE_DIT_MC=4 \
|
||||
SGLANG_CACHE_DIT_TAYLORSEER=true \
|
||||
SGLANG_CACHE_DIT_TS_ORDER=2 \
|
||||
sglang serve --model-path Tongyi-MAI/Z-Image-Turbo
|
||||
```
|
||||
|
||||
#### 4.2.2 CPU Offload
|
||||
|
||||
- `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory.
|
||||
- `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference.
|
||||
- `--vae-cpu-offload`: Use CPU offload for VAE.
|
||||
- `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument".
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
Test Environment:
|
||||
|
||||
- Hardware: AMD Instinct MI300X GPU (1x)
|
||||
- Model: Tongyi-MAI/Z-Image-Turbo
|
||||
- Docker Image: lmsysorg/sglang:v0.5.8-rocm700-mi30x
|
||||
- sglang diffusion version: 0.5.8
|
||||
|
||||
### 5.1 Speedup Benchmark
|
||||
|
||||
#### 5.1.1 Generate an image
|
||||
|
||||
**Server Command**:
|
||||
|
||||
```shell Command
|
||||
sglang serve --model-path Tongyi-MAI/Z-Image-Turbo \
|
||||
--ulysses-degree=1 --ring-degree=1 --port 30000
|
||||
```
|
||||
|
||||
**Benchmark Command**:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
|
||||
--backend sglang-image --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1
|
||||
```
|
||||
|
||||
**Result**:
|
||||
|
||||
```text Output
|
||||
================= Serving Benchmark Result =================
|
||||
Task: text-to-image
|
||||
Model: Tongyi-MAI/Z-Image-Turbo
|
||||
Dataset: vbench
|
||||
--------------------------------------------------
|
||||
Benchmark duration (s): 1.84
|
||||
Request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 1/1
|
||||
--------------------------------------------------
|
||||
Request throughput (req/s): 0.54
|
||||
Latency Mean (s): 1.8435
|
||||
Latency Median (s): 1.8435
|
||||
Latency P99 (s): 1.8435
|
||||
--------------------------------------------------
|
||||
Peak Memory Max (MB): 30689.20
|
||||
Peak Memory Mean (MB): 30689.20
|
||||
Peak Memory Median (MB): 30689.20
|
||||
============================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Generate images with high concurrency
|
||||
|
||||
**Benchmark Command**:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
|
||||
--backend sglang-image --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20
|
||||
```
|
||||
|
||||
**Result**:
|
||||
|
||||
```text Output
|
||||
================= Serving Benchmark Result =================
|
||||
Task: text-to-image
|
||||
Model: Tongyi-MAI/Z-Image-Turbo
|
||||
Dataset: vbench
|
||||
--------------------------------------------------
|
||||
Benchmark duration (s): 35.32
|
||||
Request rate: inf
|
||||
Max request concurrency: 20
|
||||
Successful requests: 20/20
|
||||
--------------------------------------------------
|
||||
Request throughput (req/s): 0.57
|
||||
Latency Mean (s): 18.5672
|
||||
Latency Median (s): 18.5573
|
||||
Latency P99 (s): 34.9880
|
||||
--------------------------------------------------
|
||||
Peak Memory Max (MB): 30689.26
|
||||
Peak Memory Mean (MB): 30689.21
|
||||
Peak Memory Median (MB): 30689.21
|
||||
============================================================
|
||||
```
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
title: Overview
|
||||
mode: wide
|
||||
description: Practical guides for deploying and using diffusion models with SGLang.
|
||||
metatags:
|
||||
description: "Explore SGLang diffusion model cookbooks for image and video generation deployment, invocation, optimization, and benchmarking examples."
|
||||
---
|
||||
|
||||
<CardGroup cols={3}>
|
||||
<Card
|
||||
title="FLUX"
|
||||
mode="card"
|
||||
href="/cookbook/diffusion/FLUX/FLUX"
|
||||
img="/cards/logos/flux.png"
|
||||
/>
|
||||
<Card
|
||||
title="Wan"
|
||||
mode="card"
|
||||
href="/cookbook/diffusion/Wan/Wan2.2"
|
||||
img="/cards/logos/wan.png"
|
||||
/>
|
||||
<Card
|
||||
title="Qwen-Image"
|
||||
mode="card"
|
||||
href="/cookbook/diffusion/Qwen-Image/Qwen-Image"
|
||||
img="/cards/logos/qwen.png"
|
||||
/>
|
||||
<Card
|
||||
title="Z-Image"
|
||||
mode="card"
|
||||
href="/cookbook/diffusion/Z-Image/Z-Image-Turbo"
|
||||
img="/cards/logos/zimage.png"
|
||||
/>
|
||||
<Card
|
||||
title="MOVA"
|
||||
mode="card"
|
||||
href="/cookbook/diffusion/MOVA/MOVA"
|
||||
img="/cards/logos/mova.png"
|
||||
/>
|
||||
</CardGroup>
|
||||
Reference in New Issue
Block a user