[diffusion] doc: consolidate documentation (#21373)
This commit is contained in:
Mick
+103
-232
@@ -1,121 +1,81 @@
|
||||
# SGLang diffusion CLI Inference
|
||||
# SGLang Diffusion CLI
|
||||
|
||||
The SGLang-diffusion CLI provides a quick way to access the inference pipeline for image and video generation.
|
||||
Use the CLI for one-off generation with `sglang generate` or to start a persistent HTTP server with `sglang serve`.
|
||||
|
||||
## Prerequisites
|
||||
## Quick Start
|
||||
|
||||
- A working SGLang diffusion installation and the `sglang` CLI available in `$PATH`.
|
||||
### Generate
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path Qwen/Qwen-Image \
|
||||
--prompt "A beautiful sunset over the mountains" \
|
||||
--save-output
|
||||
```
|
||||
|
||||
## Supported Arguments
|
||||
### Serve
|
||||
|
||||
```bash
|
||||
sglang serve \
|
||||
--model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
|
||||
--num-gpus 4 \
|
||||
--ulysses-degree 2 \
|
||||
--ring-degree 2 \
|
||||
--port 30010
|
||||
```
|
||||
|
||||
### Server Arguments
|
||||
For request and response examples, see [OpenAI-Compatible API](openai_api.md).
|
||||
|
||||
- `--model-path {MODEL_PATH}`: Path to the model or model ID
|
||||
- `--lora-path {LORA_PATH}`: Path to a LoRA adapter (local path or HuggingFace model ID). If not specified, LoRA will not be applied.
|
||||
- `--lora-nickname {NAME}`: Nickname for the LoRA adapter. (default: `default`).
|
||||
- `--num-gpus {NUM_GPUS}`: Number of GPUs to use
|
||||
- `--tp-size {TP_SIZE}`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster)
|
||||
- `--sp-degree {SP_SIZE}`: Sequence parallelism size (typically should match the number of GPUs)
|
||||
- `--ulysses-degree {ULYSSES_DEGREE}`: The degree of DeepSpeed-Ulysses-style SP in USP
|
||||
- `--ring-degree {RING_DEGREE}`: The degree of ring attention-style SP in USP
|
||||
- `--attention-backend {BACKEND}`: Attention backend to use. For SGLang-native pipelines use `fa`, `torch_sdpa`, `sage_attn`, etc. For diffusers pipelines use diffusers backend names like `flash`, `_flash_3_hub`, `sage`, `xformers`.
|
||||
- `--attention-backend-config {CONFIG}`: Configuration for the attention backend. Can be a JSON string (e.g., '{"k": "v"}'), a path to a JSON/YAML file, or key=value pairs (e.g., "k=v,k2=v2").
|
||||
- `--cache-dit-config {PATH}`: Path to a Cache-DiT YAML/JSON config (diffusers backend only)
|
||||
- `--dit-precision {DTYPE}`: Precision for the DiT model (currently supports fp32, fp16, and bf16).
|
||||
```{tip}
|
||||
Use `sglang generate --help` and `sglang serve --help` for the full argument list. The CLI help output is the source of truth for exhaustive flags.
|
||||
```
|
||||
|
||||
### Quantized Transformers
|
||||
## Common Options
|
||||
|
||||
### Model and runtime
|
||||
|
||||
- `--model-path {MODEL}`: model path or Hugging Face model ID
|
||||
- `--lora-path {PATH}` and `--lora-nickname {NAME}`: load a LoRA adapter
|
||||
- `--num-gpus {N}`: number of GPUs to use
|
||||
- `--tp-size {N}`: tensor parallelism size, mainly for encoders
|
||||
- `--sp-degree {N}`: sequence parallelism size
|
||||
- `--ulysses-degree {N}` and `--ring-degree {N}`: USP parallelism controls
|
||||
- `--attention-backend {BACKEND}`: attention backend for native SGLang pipelines
|
||||
- `--attention-backend-config {CONFIG}`: attention backend configuration
|
||||
|
||||
### Sampling and output
|
||||
|
||||
- `--prompt {PROMPT}` and `--negative-prompt {PROMPT}`
|
||||
- `--num-inference-steps {STEPS}` and `--seed {SEED}`
|
||||
- `--height {HEIGHT}`, `--width {WIDTH}`, `--num-frames {N}`, `--fps {FPS}`
|
||||
- `--output-path {PATH}`, `--output-file-name {NAME}`, `--save-output`, `--return-frames`
|
||||
|
||||
For frame interpolation and upscaling, see [Post-Processing](post_processing.md).
|
||||
|
||||
### Quantized transformers
|
||||
|
||||
For quantized transformer checkpoints, prefer:
|
||||
|
||||
- `--model-path` for the base model (the pipeline)
|
||||
- `--model-path` for the base pipeline
|
||||
- `--transformer-path` for a quantized `transformers` transformer component folder
|
||||
- `--transformer-weights-path` for a quantized safetensors file, directory, or repo
|
||||
|
||||
See [Quantization](../quantization.md) for the supported quantization families and examples.
|
||||
See [Quantization](../quantization.md) for supported quantization families and examples.
|
||||
|
||||
## Configuration Files
|
||||
|
||||
### Sampling Parameters
|
||||
|
||||
- `--prompt {PROMPT}`: Text description for the video you want to generate
|
||||
- `--num-inference-steps {STEPS}`: Number of denoising steps
|
||||
- `--negative-prompt {PROMPT}`: Negative prompt to guide generation away from certain concepts
|
||||
- `--seed {SEED}`: Random seed for reproducible generation
|
||||
|
||||
|
||||
**Image/Video Configuration**
|
||||
|
||||
- `--height {HEIGHT}`: Height of the generated output
|
||||
- `--width {WIDTH}`: Width of the generated output
|
||||
- `--num-frames {NUM_FRAMES}`: Number of frames to generate
|
||||
- `--fps {FPS}`: Frames per second for the saved output, if this is a video-generation task
|
||||
|
||||
|
||||
**Post-Processing** (frame interpolation & upscaling)
|
||||
|
||||
SGLang diffusion supports optional post-processing steps — frame interpolation
|
||||
(RIFE) for smoother video and upscaling (Real-ESRGAN) for higher resolution.
|
||||
See the dedicated **[Post-Processing](post_processing.md)** page for full
|
||||
details, supported models, and examples.
|
||||
|
||||
**Output Options**
|
||||
|
||||
- `--output-path {PATH}`: Directory to save the generated video
|
||||
- `--save-output`: Whether to save the image/video to disk
|
||||
- `--return-frames`: Whether to return the raw frames
|
||||
|
||||
### Using Configuration Files
|
||||
|
||||
Instead of specifying all parameters on the command line, you can use a configuration file:
|
||||
Use `--config` to load JSON or YAML configuration. Command-line flags override values from the config file.
|
||||
|
||||
```bash
|
||||
sglang generate --config {CONFIG_FILE_PATH}
|
||||
sglang generate --config config.yaml
|
||||
```
|
||||
|
||||
The configuration file should be in JSON or YAML format with the same parameter names as the CLI options. Command-line arguments take precedence over settings in the configuration file, allowing you to override specific values while keeping the rest from the configuration file.
|
||||
|
||||
Example configuration file (config.json):
|
||||
|
||||
```json
|
||||
{
|
||||
"model_path": "FastVideo/FastHunyuan-diffusers",
|
||||
"prompt": "A beautiful woman in a red dress walking down a street",
|
||||
"output_path": "outputs/",
|
||||
"num_gpus": 2,
|
||||
"sp_size": 2,
|
||||
"tp_size": 1,
|
||||
"num_frames": 45,
|
||||
"height": 720,
|
||||
"width": 1280,
|
||||
"num_inference_steps": 6,
|
||||
"seed": 1024,
|
||||
"fps": 24,
|
||||
"precision": "bf16",
|
||||
"vae_precision": "fp16",
|
||||
"vae_tiling": true,
|
||||
"vae_sp": true,
|
||||
"vae_config": {
|
||||
"load_encoder": false,
|
||||
"load_decoder": true,
|
||||
"tile_sample_min_height": 256,
|
||||
"tile_sample_min_width": 256
|
||||
},
|
||||
"text_encoder_precisions": [
|
||||
"fp16",
|
||||
"fp16"
|
||||
],
|
||||
"mask_strategy_file_path": null,
|
||||
"enable_torch_compile": false
|
||||
}
|
||||
```
|
||||
|
||||
Or using YAML format (config.yaml):
|
||||
Example:
|
||||
|
||||
```yaml
|
||||
model_path: "FastVideo/FastHunyuan-diffusers"
|
||||
prompt: "A beautiful woman in a red dress walking down a street"
|
||||
output_path: "outputs/"
|
||||
model_path: FastVideo/FastHunyuan-diffusers
|
||||
prompt: A beautiful woman in a red dress walking down a street
|
||||
output_path: outputs/
|
||||
num_gpus: 2
|
||||
sp_size: 2
|
||||
tp_size: 1
|
||||
@@ -125,163 +85,95 @@ width: 1280
|
||||
num_inference_steps: 6
|
||||
seed: 1024
|
||||
fps: 24
|
||||
precision: "bf16"
|
||||
vae_precision: "fp16"
|
||||
precision: bf16
|
||||
vae_precision: fp16
|
||||
vae_tiling: true
|
||||
vae_sp: true
|
||||
vae_config:
|
||||
load_encoder: false
|
||||
load_decoder: true
|
||||
tile_sample_min_height: 256
|
||||
tile_sample_min_width: 256
|
||||
text_encoder_precisions:
|
||||
- "fp16"
|
||||
- "fp16"
|
||||
mask_strategy_file_path: null
|
||||
enable_torch_compile: false
|
||||
```
|
||||
|
||||
## Generate
|
||||
|
||||
To see all the options, you can use the `--help` flag:
|
||||
`sglang generate` runs a single generation job and exits when the job finishes.
|
||||
|
||||
```bash
|
||||
sglang generate --help
|
||||
sglang generate \
|
||||
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
|
||||
--text-encoder-cpu-offload \
|
||||
--pin-cpu-memory \
|
||||
--num-gpus 4 \
|
||||
--ulysses-degree 2 \
|
||||
--ring-degree 2 \
|
||||
--prompt "A curious raccoon" \
|
||||
--save-output \
|
||||
--output-path outputs \
|
||||
--output-file-name "a-curious-raccoon.mp4"
|
||||
```
|
||||
|
||||
```{note}
|
||||
HTTP server-only arguments are ignored by `sglang generate`.
|
||||
```
|
||||
|
||||
For diffusers pipelines, Cache-DiT can be enabled with `SGLANG_CACHE_DIT_ENABLED=true` or `--cache-dit-config`. See [Cache-DiT](../performance/cache/cache_dit.md).
|
||||
|
||||
## Serve
|
||||
|
||||
Launch the SGLang diffusion HTTP server and interact with it using the OpenAI SDK and curl.
|
||||
|
||||
### Start the server
|
||||
|
||||
Use the following command to launch the server:
|
||||
`sglang serve` starts the HTTP server and keeps the model loaded for repeated requests.
|
||||
|
||||
```bash
|
||||
SERVER_ARGS=(
|
||||
--model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers
|
||||
--text-encoder-cpu-offload
|
||||
--pin-cpu-memory
|
||||
--num-gpus 4
|
||||
--ulysses-degree=2
|
||||
--ring-degree=2
|
||||
)
|
||||
|
||||
sglang serve "${SERVER_ARGS[@]}"
|
||||
sglang serve \
|
||||
--model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
|
||||
--text-encoder-cpu-offload \
|
||||
--pin-cpu-memory \
|
||||
--num-gpus 4 \
|
||||
--ulysses-degree 2 \
|
||||
--ring-degree 2 \
|
||||
--port 30010
|
||||
```
|
||||
|
||||
- **--model-path**: Which model to load. The example uses `Wan-AI/Wan2.1-T2V-1.3B-Diffusers`.
|
||||
- **--port**: HTTP port to listen on (the default here is `30010`).
|
||||
### Cloud Storage
|
||||
|
||||
For detailed API usage, including Image, Video Generation and LoRA management, please refer to the [OpenAI API Documentation](openai_api.md).
|
||||
|
||||
### Cloud Storage Support
|
||||
|
||||
SGLang diffusion supports automatically uploading generated images and videos to S3-compatible cloud storage (e.g., AWS S3, MinIO, Alibaba Cloud OSS, Tencent Cloud COS).
|
||||
|
||||
When enabled, the server follows a **Generate -> Upload -> Delete** workflow:
|
||||
1. The artifact is generated to a temporary local file.
|
||||
2. The file is immediately uploaded to the configured S3 bucket in a background thread.
|
||||
3. Upon successful upload, the local file is deleted.
|
||||
4. The API response returns the public URL of the uploaded object.
|
||||
|
||||
**Configuration**
|
||||
|
||||
Cloud storage is enabled via environment variables. Note that `boto3` must be installed separately (`pip install boto3`) to use this feature.
|
||||
SGLang Diffusion can upload generated images and videos to S3-compatible object storage after generation.
|
||||
|
||||
```bash
|
||||
# Enable S3 storage
|
||||
export SGLANG_CLOUD_STORAGE_TYPE=s3
|
||||
export SGLANG_S3_BUCKET_NAME=my-bucket
|
||||
export SGLANG_S3_ACCESS_KEY_ID=your-access-key
|
||||
export SGLANG_S3_SECRET_ACCESS_KEY=your-secret-key
|
||||
|
||||
# Optional: Custom endpoint for MinIO/OSS/COS
|
||||
export SGLANG_S3_ENDPOINT_URL=https://minio.example.com
|
||||
```
|
||||
|
||||
See [Environment Variables Documentation](../environment_variables.md) for more details.
|
||||
|
||||
## Generate
|
||||
|
||||
Run a one-off generation task without launching a persistent server.
|
||||
|
||||
To use it, pass both server arguments and sampling parameters in one command, after the `generate` subcommand, for example:
|
||||
|
||||
```bash
|
||||
SERVER_ARGS=(
|
||||
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers
|
||||
--text-encoder-cpu-offload
|
||||
--pin-cpu-memory
|
||||
--num-gpus 4
|
||||
--ulysses-degree=2
|
||||
--ring-degree=2
|
||||
)
|
||||
|
||||
SAMPLING_ARGS=(
|
||||
--prompt "A curious raccoon"
|
||||
--save-output
|
||||
--output-path outputs
|
||||
--output-file-name "A curious raccoon.mp4"
|
||||
)
|
||||
|
||||
sglang generate "${SERVER_ARGS[@]}" "${SAMPLING_ARGS[@]}"
|
||||
|
||||
# Or, users can set `SGLANG_CACHE_DIT_ENABLED` env as `true` to enable cache acceleration
|
||||
SGLANG_CACHE_DIT_ENABLED=true sglang generate "${SERVER_ARGS[@]}" "${SAMPLING_ARGS[@]}"
|
||||
```
|
||||
|
||||
Once the generation task has finished, the server will shut down automatically.
|
||||
|
||||
> [!NOTE]
|
||||
> The HTTP server-related arguments are ignored in this subcommand.
|
||||
See [Environment Variables](../environment_variables.md) for the full set of storage options.
|
||||
|
||||
## Component Path Overrides
|
||||
|
||||
SGLang diffusion allows you to override any pipeline component (e.g., `vae`, `transformer`, `text_encoder`) by specifying a custom checkpoint path. This is useful for:
|
||||
|
||||
### Example: FLUX.2-dev with Tiny AutoEncoder
|
||||
|
||||
You can override **any** component by using `--<component>-path`, where `<component>` matches the key in the model's `model_index.json`:
|
||||
|
||||
For example, replace the default VAE with a distilled tiny autoencoder for ~3x faster decoding:
|
||||
Override individual pipeline components such as `vae`, `transformer`, or `text_encoder` with `--<component>-path`.
|
||||
|
||||
```bash
|
||||
sglang serve \
|
||||
--model-path=black-forest-labs/FLUX.2-dev \
|
||||
# with a Huggingface Repo ID
|
||||
--vae-path=fal/FLUX.2-Tiny-AutoEncoder
|
||||
# or use a local path
|
||||
--vae-path=~/.cache/huggingface/hub/models--fal--FLUX.2-Tiny-AutoEncoder/snapshots/.../vae
|
||||
--model-path black-forest-labs/FLUX.2-dev \
|
||||
--vae-path fal/FLUX.2-Tiny-AutoEncoder
|
||||
```
|
||||
|
||||
**Important:**
|
||||
- The component key must match the one in your model's `model_index.json` (e.g., `vae`).
|
||||
- The path must:
|
||||
- either be a Huggingface Repo ID (e.g., fal/FLUX.2-Tiny-AutoEncoder)
|
||||
- or point to a **complete component folder**, containing `config.json` and safetensors files
|
||||
|
||||
The component key must match the key in the model's `model_index.json`, and the path must be either a Hugging Face repo ID or a complete component directory.
|
||||
|
||||
## Diffusers Backend
|
||||
|
||||
SGLang diffusion supports a **diffusers backend** that allows you to run any diffusers-compatible model through SGLang's infrastructure using vanilla diffusers pipelines. This is useful for running models without native SGLang implementations or models with custom pipeline classes.
|
||||
Use `--backend diffusers` to force vanilla diffusers pipelines when no native SGLang implementation exists or when a model requires a custom pipeline class.
|
||||
|
||||
### Arguments
|
||||
### Key Options
|
||||
|
||||
| Argument | Values | Description |
|
||||
|----------|--------|-------------|
|
||||
| `--backend` | `auto` (default), `sglang`, `diffusers` | `auto`: prefer native SGLang, fallback to diffusers. `sglang`: force native (fails if unavailable). `diffusers`: force vanilla diffusers pipeline. |
|
||||
| `--diffusers-attention-backend` | `flash`, `_flash_3_hub`, `sage`, `xformers`, `native` | Attention backend for diffusers pipelines. See [diffusers attention backends](https://huggingface.co/docs/diffusers/main/en/optimization/attention_backends). |
|
||||
| `--trust-remote-code` | flag | Required for models with custom pipeline classes (e.g., Ovis). |
|
||||
| `--vae-tiling` | flag | Enable VAE tiling for large image support (decodes tile-by-tile). |
|
||||
| `--vae-slicing` | flag | Enable VAE slicing for lower memory usage (decodes slice-by-slice). |
|
||||
| `--dit-precision` | `fp16`, `bf16`, `fp32` | Precision for the diffusion transformer. |
|
||||
| `--vae-precision` | `fp16`, `bf16`, `fp32` | Precision for the VAE. |
|
||||
| `--enable-torch-compile` | flag | Enable `torch.compile` for diffusers pipelines. |
|
||||
| `--cache-dit-config` | `{PATH}` | Path to a Cache-DiT YAML/JSON config file for accelerating diffusers pipelines with Cache-DiT. |
|
||||
| `--backend` | `auto`, `sglang`, `diffusers` | Choose native SGLang, force native, or force diffusers |
|
||||
| `--diffusers-attention-backend` | `flash`, `_flash_3_hub`, `sage`, `xformers`, `native` | Attention backend for diffusers pipelines |
|
||||
| `--trust-remote-code` | flag | Required for models with custom pipeline classes |
|
||||
| `--vae-tiling` and `--vae-slicing` | flag | Lower memory usage for VAE decode |
|
||||
| `--dit-precision` and `--vae-precision` | `fp16`, `bf16`, `fp32` | Precision controls |
|
||||
| `--enable-torch-compile` | flag | Enable `torch.compile` |
|
||||
| `--cache-dit-config` | `{PATH}` | Cache-DiT config for diffusers pipelines |
|
||||
|
||||
### Example: Running Ovis-Image-7B
|
||||
|
||||
[Ovis-Image-7B](https://huggingface.co/AIDC-AI/Ovis-Image-7B) is a 7B text-to-image model optimized for high-quality text rendering.
|
||||
### Example
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
@@ -298,25 +190,4 @@ sglang generate \
|
||||
--output-file-name ovis_garden.png
|
||||
```
|
||||
|
||||
### Extra Diffusers Arguments
|
||||
|
||||
For pipeline-specific parameters not exposed via CLI, use `diffusers_kwargs` in a config file:
|
||||
|
||||
```json
|
||||
{
|
||||
"model_path": "AIDC-AI/Ovis-Image-7B",
|
||||
"backend": "diffusers",
|
||||
"prompt": "A beautiful landscape",
|
||||
"diffusers_kwargs": {
|
||||
"cross_attention_kwargs": {"scale": 0.5}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```bash
|
||||
sglang generate --config config.json
|
||||
```
|
||||
|
||||
### Cache-DiT Acceleration
|
||||
|
||||
Users who use the diffusers backend can also leverage Cache-DiT acceleration and load custom cache configs from a YAML file to boost performance of diffusers pipelines. See the [Cache-DiT Acceleration](https://docs.sglang.io/diffusion/performance/cache/cache_dit.html) documentation for details.
|
||||
For pipeline-specific arguments not exposed in the CLI, pass `diffusers_kwargs` in a config file.
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
# CI Performance
|
||||
|
||||
## Perf Baseline Generation Script
|
||||
|
||||
`python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py` starts a local diffusion server, issues requests for selected test cases, aggregates stage/denoise-step/E2E timings from the perf log, and writes the results back to the `scenarios` section of `perf_baselines.json`.
|
||||
|
||||
@@ -2,6 +2,18 @@
|
||||
|
||||
This guide outlines the requirements for contributing to the SGLang Diffusion module (`sglang.multimodal_gen`).
|
||||
|
||||
## Contributor Guides
|
||||
|
||||
- [Support New Models](support_new_models.md): implementation guide for adding new diffusion pipelines
|
||||
- [CI Performance](ci_perf.md): update and regenerate perf baselines
|
||||
|
||||
```{toctree}
|
||||
:maxdepth: 1
|
||||
|
||||
support_new_models
|
||||
ci_perf
|
||||
```
|
||||
|
||||
## On AI-Assisted ("Vibe Coding") PRs
|
||||
|
||||
Vibe-coded PRs are welcome — we judge code quality, not how it was produced. The bar is the same for all PRs:
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
# Development
|
||||
|
||||
This page collects lower-level development material for SGLang Diffusion.
|
||||
|
||||
- [Contributing](contributing.md): contribution workflow, adding new models, and CI perf baselines
|
||||
@@ -1,3 +1,5 @@
|
||||
# Environment Variables
|
||||
|
||||
## Apple MPS
|
||||
|
||||
| Environment Variable | Default | Description |
|
||||
|
||||
+22
-74
@@ -1,100 +1,48 @@
|
||||
# SGLang Diffusion
|
||||
|
||||
SGLang Diffusion is an inference framework for accelerated image and video generation using diffusion models. It provides an end-to-end unified pipeline with optimized kernels and an efficient scheduler loop.
|
||||
SGLang Diffusion is a high-performance inference framework for image and video generation. It provides native SGLang pipelines, diffusers backend support, an OpenAI-compatible server, and an optimized kernel stack built on both precompiled `sgl-kernel` operators and JIT kernels for key inference paths.
|
||||
|
||||
## Key Features
|
||||
|
||||
- **Broad Model Support**: Wan series, FastWan series, Hunyuan, Qwen-Image, Qwen-Image-Edit, Flux, Z-Image, GLM-Image, and more
|
||||
- **Fast Inference**: Optimized kernels, efficient scheduler loop, and Cache-DiT acceleration
|
||||
- **Ease of Use**: OpenAI-compatible API, CLI, and Python SDK
|
||||
- **Multi-Platform**:
|
||||
- NVIDIA GPUs (H100, H200, A100, B200, 4090)
|
||||
- AMD GPUs (MI300X, MI325X)
|
||||
- Ascend NPU (A2, A3)
|
||||
- Apple Silicon (M-series via MPS)
|
||||
- Moore Threads GPUs (MTT S5000)
|
||||
|
||||
---
|
||||
- Broad model support across Wan, Hunyuan, Qwen-Image, FLUX, Z-Image, GLM-Image, and more
|
||||
- Fast inference with `sgl-kernel`, JIT kernels, scheduler improvements, and caching acceleration
|
||||
- Multiple interfaces: `sglang generate`, `sglang serve`, and an OpenAI-compatible API
|
||||
- Multi-platform support for NVIDIA, AMD, Ascend, Apple Silicon, and Moore Threads
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
uv pip install "sglang[diffusion]" --prerelease=allow
|
||||
```
|
||||
|
||||
See [Installation Guide](installation.md) for more installation methods and ROCm-specific instructions.
|
||||
|
||||
### Basic Usage
|
||||
|
||||
Generate an image with the CLI:
|
||||
|
||||
```bash
|
||||
sglang generate --model-path Qwen/Qwen-Image \
|
||||
--prompt "A beautiful sunset over the mountains" \
|
||||
--save-output
|
||||
--prompt "A beautiful sunset over the mountains" \
|
||||
--save-output
|
||||
```
|
||||
|
||||
Or start a server with the OpenAI-compatible API:
|
||||
|
||||
```bash
|
||||
sglang serve --model-path Qwen/Qwen-Image --port 30010
|
||||
```
|
||||
|
||||
---
|
||||
## Start Here
|
||||
|
||||
## Documentation
|
||||
- [Installation](installation.md): install SGLang Diffusion and platform dependencies
|
||||
- [Compatibility Matrix](compatibility_matrix.md): check model and optimization support
|
||||
- [CLI](api/cli.md): run one-off generation jobs or launch a persistent server
|
||||
- [OpenAI-Compatible API](api/openai_api.md): send image and video requests to the HTTP server
|
||||
- [Attention Backends](performance/attention_backends.md): choose the best backend for your model and hardware
|
||||
- [Caching Acceleration](performance/cache/index.md): use Cache-DiT or TeaCache to reduce denoising cost
|
||||
- [Quantization](quantization.md): load quantized transformer checkpoints
|
||||
- [Contributing](contributing.md): contribution workflow, adding new models, and CI perf baselines
|
||||
|
||||
### Getting Started
|
||||
## Additional Documentation
|
||||
|
||||
- **[Installation](installation.md)** - Install SGLang Diffusion via pip, uv, Docker, or from source
|
||||
- **[Compatibility Matrix](compatibility_matrix.md)** - Supported models and optimization compatibility
|
||||
|
||||
### Usage
|
||||
|
||||
- **[CLI Documentation](api/cli.md)** - Command-line interface for `sglang generate` and `sglang serve`
|
||||
- **[Quantization](quantization.md)** - Quantized transformer checkpoint usage and supported quantization families
|
||||
- **[OpenAI API](api/openai_api.md)** - OpenAI-compatible API for image/video generation and LoRA management
|
||||
- **[Post-Processing](api/post_processing.md)** - Frame interpolation (RIFE) and upscaling (Real-ESRGAN)
|
||||
|
||||
### Performance Optimization
|
||||
|
||||
- **[Performance Overview](performance/index.md)** - Overview of all performance optimization strategies
|
||||
- **[Attention Backends](performance/attention_backends.md)** - Available attention backends (FlashAttention, SageAttention, etc.)
|
||||
- **[Caching Strategies](performance/cache/)** - Cache-DiT and TeaCache acceleration
|
||||
- **[Profiling](performance/profiling.md)** - Profiling techniques with PyTorch Profiler and Nsight Systems
|
||||
|
||||
### Reference
|
||||
|
||||
- **[Environment Variables](environment_variables.md)** - Configuration via environment variables
|
||||
- **[Support New Models](support_new_models.md)** - Guide for adding new diffusion models
|
||||
- **[Contributing](contributing.md)** - Contribution guidelines and commit message conventions
|
||||
- **[CI Performance](ci_perf.md)** - Performance baseline generation script
|
||||
|
||||
---
|
||||
|
||||
## CLI Quick Reference
|
||||
|
||||
### Generate (one-off generation)
|
||||
|
||||
```bash
|
||||
sglang generate --model-path <MODEL> --prompt "<PROMPT>" --save-output
|
||||
```
|
||||
|
||||
### Serve (HTTP server)
|
||||
|
||||
```bash
|
||||
sglang serve --model-path <MODEL> --port 30010
|
||||
```
|
||||
|
||||
### Enable Cache-DiT acceleration
|
||||
|
||||
```bash
|
||||
SGLANG_CACHE_DIT_ENABLED=true sglang generate --model-path <MODEL> --prompt "<PROMPT>"
|
||||
```
|
||||
|
||||
---
|
||||
- [Post-Processing](api/post_processing.md): frame interpolation and upscaling
|
||||
- [Performance Overview](performance/index.md): overview of attention, caching, and profiling
|
||||
- [Environment Variables](environment_variables.md): platform, caching, storage, and debugging configuration
|
||||
- [Support New Models](support_new_models.md): implementation guide for new diffusion pipelines
|
||||
- [CI Performance](ci_perf.md): performance baseline generation
|
||||
|
||||
## References
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Install SGLang-Diffusion
|
||||
|
||||
You can install SGLang-Diffusion using one of the methods below.
|
||||
You can install SGLang-Diffusion using one of the methods below. The standard installation already includes SGLang's optimized kernel stack, including both `sgl-kernel` and JIT kernels used by diffusion workloads.
|
||||
|
||||
## Standard Installation (NVIDIA GPUs)
|
||||
|
||||
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
# Cache-DiT Acceleration
|
||||
# Cache-DiT
|
||||
|
||||
SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to **1.69x inference speedup** with minimal quality loss.
|
||||
|
||||
|
||||
+9
-4
@@ -1,6 +1,6 @@
|
||||
# Caching Acceleration for Diffusion Models
|
||||
# Caching Acceleration
|
||||
|
||||
SGLang provides multiple caching acceleration strategies for Diffusion Transformer (DiT) models. These strategies can significantly reduce inference time by skipping redundant computation.
|
||||
SGLang provides two complementary caching strategies for Diffusion Transformer (DiT) models. Both reduce denoising cost by skipping redundant computation, but they operate at different levels.
|
||||
|
||||
## Overview
|
||||
|
||||
@@ -11,8 +11,6 @@ SGLang supports two complementary caching approaches:
|
||||
| **Cache-DiT** | Block-level | Skip individual transformer blocks dynamically | Advanced, higher speedup |
|
||||
| **TeaCache** | Timestep-level | Skip entire denoising steps based on L1 similarity | Simple, built-in |
|
||||
|
||||
|
||||
|
||||
## Cache-DiT
|
||||
|
||||
[Cache-DiT](https://github.com/vipshop/cache-dit) provides block-level caching with
|
||||
@@ -54,6 +52,13 @@ See [teacache.md](teacache.md) for detailed documentation.
|
||||
|
||||
For Flux and Qwen models, TeaCache is automatically disabled when CFG is enabled.
|
||||
|
||||
```{toctree}
|
||||
:maxdepth: 1
|
||||
|
||||
cache_dit
|
||||
teacache
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Cache-DiT Repository](https://github.com/vipshop/cache-dit)
|
||||
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
# TeaCache Acceleration
|
||||
# TeaCache
|
||||
|
||||
> **Note**: This is one of two caching strategies available in SGLang.
|
||||
> For an overview of all caching options, see [caching](../index.md).
|
||||
|
||||
@@ -1,71 +1,35 @@
|
||||
# Performance Optimization
|
||||
# Performance
|
||||
|
||||
SGLang-Diffusion provides multiple performance optimization strategies to accelerate inference. This section covers all available performance tuning options.
|
||||
This section covers the main performance levers for SGLang Diffusion: attention backends, caching acceleration, and profiling.
|
||||
|
||||
## Overview
|
||||
|
||||
| Optimization | Type | Description |
|
||||
|--------------|------|-------------|
|
||||
| **Cache-DiT** | Caching | Block-level caching with DBCache, TaylorSeer, and SCM |
|
||||
| **TeaCache** | Caching | Timestep-level caching using L1 similarity |
|
||||
| **TeaCache** | Caching | Timestep-level caching based on temporal similarity |
|
||||
| **Attention Backends** | Kernel | Optimized attention implementations (FlashAttention, SageAttention, etc.) |
|
||||
| **Profiling** | Diagnostics | PyTorch Profiler and Nsight Systems guidance |
|
||||
|
||||
## Caching Strategies
|
||||
## Start Here
|
||||
|
||||
SGLang supports two complementary caching approaches:
|
||||
- Use [Attention Backends](attention_backends.md) to choose the best backend for your model and hardware.
|
||||
- Use [Caching Acceleration](cache/index.md) to reduce denoising cost with Cache-DiT or TeaCache.
|
||||
- Use [Profiling](profiling.md) when you need to diagnose a bottleneck rather than guess.
|
||||
|
||||
### Cache-DiT
|
||||
## Caching at a Glance
|
||||
|
||||
[Cache-DiT](https://github.com/vipshop/cache-dit) provides block-level caching with advanced strategies. It can achieve up to **1.69x speedup**.
|
||||
- [Cache-DiT](cache/cache_dit.md) is block-level caching for diffusers pipelines and higher speedup-oriented tuning.
|
||||
- [TeaCache](cache/teacache.md) is timestep-level caching built into SGLang model families.
|
||||
|
||||
**Quick Start:**
|
||||
```bash
|
||||
SGLANG_CACHE_DIT_ENABLED=true \
|
||||
sglang generate --model-path Qwen/Qwen-Image \
|
||||
--prompt "A beautiful sunset over the mountains"
|
||||
```{toctree}
|
||||
:maxdepth: 1
|
||||
|
||||
attention_backends
|
||||
cache/index
|
||||
profiling
|
||||
```
|
||||
|
||||
**Key Features:**
|
||||
- **DBCache**: Dynamic block-level caching based on residual differences
|
||||
- **TaylorSeer**: Taylor expansion-based calibration for optimized caching
|
||||
- **SCM**: Step-level computation masking for additional speedup
|
||||
|
||||
See [Cache-DiT Documentation](cache/cache_dit.md) for detailed configuration.
|
||||
|
||||
### TeaCache
|
||||
|
||||
TeaCache (Temporal similarity-based caching) accelerates diffusion inference by detecting when consecutive denoising steps are similar enough to skip computation entirely.
|
||||
|
||||
**Quick Overview:**
|
||||
- Tracks L1 distance between modulated inputs across timesteps
|
||||
- When accumulated distance is below threshold, reuses cached residual
|
||||
- Supports CFG with separate positive/negative caches
|
||||
|
||||
**Supported Models:** Wan (wan2.1, wan2.2), Hunyuan (HunyuanVideo), Z-Image
|
||||
|
||||
See [TeaCache Documentation](cache/teacache.md) for detailed configuration.
|
||||
|
||||
## Attention Backends
|
||||
|
||||
Different attention backends offer varying performance characteristics depending on your hardware and model:
|
||||
|
||||
- **FlashAttention**: Fastest on NVIDIA GPUs with fp16/bf16
|
||||
- **SageAttention**: Alternative optimized implementation
|
||||
- **xformers**: Memory-efficient attention
|
||||
- **SDPA**: PyTorch native scaled dot-product attention
|
||||
|
||||
See [Attention Backends](attention_backends.md) for platform support and configuration options.
|
||||
|
||||
## Profiling
|
||||
|
||||
To diagnose performance bottlenecks, SGLang-Diffusion supports profiling tools:
|
||||
|
||||
- **PyTorch Profiler**: Built-in Python profiling
|
||||
- **Nsight Systems**: GPU kernel-level analysis
|
||||
|
||||
See [Profiling Guide](profiling.md) for detailed instructions.
|
||||
|
||||
## References
|
||||
|
||||
- [Cache-DiT Repository](https://github.com/vipshop/cache-dit)
|
||||
|
||||
@@ -141,8 +141,7 @@ Recommended auto-detected flow:
|
||||
sglang generate \
|
||||
--model-path Qwen/Qwen-Image \
|
||||
--transformer-weights-path /path/to/svdq-int4_r32-qwen-image.safetensors \
|
||||
--prompt "change the raccoon to a cute cat" \
|
||||
--attention-backend torch_sdpa \
|
||||
--prompt "a beautiful sunset" \
|
||||
--save-output
|
||||
```
|
||||
|
||||
@@ -156,7 +155,6 @@ sglang generate \
|
||||
--quantization-precision int4 \
|
||||
--quantization-rank 128 \
|
||||
--prompt "a beautiful sunset" \
|
||||
--attention-backend torch_sdpa \
|
||||
--save-output
|
||||
```
|
||||
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
# Reference
|
||||
|
||||
Reference material for environment-based configuration and runtime behavior.
|
||||
|
||||
- [Environment Variables](environment_variables.md): platform, caching, cloud storage, and debugging variables
|
||||
|
||||
```{toctree}
|
||||
:maxdepth: 1
|
||||
|
||||
environment_variables
|
||||
```
|
||||
@@ -0,0 +1,17 @@
|
||||
# Usage
|
||||
|
||||
Use this section for day-to-day inference workflows with SGLang Diffusion.
|
||||
|
||||
- [CLI](api/cli.md): run one-off jobs with `sglang generate` or start a server with `sglang serve`
|
||||
- [OpenAI-Compatible API](api/openai_api.md): request format, endpoints, and SDK examples
|
||||
- [Post-Processing](api/post_processing.md): frame interpolation and upscaling
|
||||
- [Quantization](quantization.md): quantized transformer checkpoints and supported quantization families
|
||||
|
||||
```{toctree}
|
||||
:maxdepth: 1
|
||||
|
||||
api/cli
|
||||
api/openai_api
|
||||
api/post_processing
|
||||
quantization
|
||||
```
|
||||
+1
-8
@@ -85,18 +85,11 @@ Its core features include:
|
||||
diffusion/installation
|
||||
diffusion/compatibility_matrix
|
||||
diffusion/api/cli
|
||||
diffusion/quantization
|
||||
diffusion/api/openai_api
|
||||
diffusion/performance/index
|
||||
diffusion/performance/attention_backends
|
||||
diffusion/performance/profiling
|
||||
diffusion/performance/cache/index
|
||||
diffusion/performance/cache/cache_dit
|
||||
diffusion/performance/cache/teacache
|
||||
diffusion/support_new_models
|
||||
diffusion/quantization
|
||||
diffusion/contributing
|
||||
diffusion/ci_perf
|
||||
diffusion/environment_variables
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
Reference in New Issue
Block a user