Add new Mintlify documentation site (docs_new/) (#23001)
Co-authored-by: AdityaVKochar <adityavardhankochar@gmail.com> Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com> Co-authored-by: adhyan-jain <adhyanjain2006@gmail.com> Co-authored-by: Adhyan Jain <71976554+adhyan-jain@users.noreply.github.com> Co-authored-by: Maitri-shah29 <maitrirajivshah@gmail.com> Co-authored-by: Adarsh Shirawalmath <114558126+adarshxs@users.noreply.github.com> Co-authored-by: Maitri Shah <shah29maitri@gmail.com> Co-authored-by: Aditya Vardhan Kochar <80113212+AdityaVKochar@users.noreply.github.com> Co-authored-by: Rishit Shivam <164783543+pokymono@users.noreply.github.com> Co-authored-by: Rishitshivam <164783543+Rishitshivam@users.noreply.github.com> Co-authored-by: IshhanKheria <ishhankheria06@gmail.com> Co-authored-by: Ishita Joshi <ishitata.joshi@gmail.com> Co-authored-by: Richard Chen <104477092+Richardczl98@users.noreply.github.com> Co-authored-by: longGGGGGG <553746008@qq.com> Co-authored-by: Richard <richardchen@radixark.ai> Co-authored-by: Nakul Sinha <nakul.new4socials@gmail.com> Co-authored-by: Divyam Agrawal <ludicrouslytrue@gmail.com> Co-authored-by: Richardczl98 <Zhenlinc@stanford.edu> Co-authored-by: Krishang Zinzuwadia <krishangzinzuwadia@gmail.com> Co-authored-by: nimeshas <nimesha.s106@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jignas Paturu <86356085+JignasP@users.noreply.github.com> Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
This commit is contained in:
Mingyi
co-authored by
AdityaVKochar
mintlify[bot]
adhyan-jain
Adhyan Jain
Maitri-shah29
Adarsh Shirawalmath
Maitri Shah
Aditya Vardhan Kochar
Rishit Shivam
Rishitshivam
IshhanKheria
Ishita Joshi
Richard Chen
longGGGGGG
Richard
Nakul Sinha
Divyam Agrawal
Richardczl98
Krishang Zinzuwadia
nimeshas
Claude Opus 4.6
github-actions[bot]
Jignas Paturu
zijiexia
AdityaVKochar
mintlify[bot]
adhyan-jain
Adhyan Jain
Maitri-shah29
Adarsh Shirawalmath
Maitri Shah
Aditya Vardhan Kochar
Rishit Shivam
Rishitshivam
IshhanKheria
Ishita Joshi
Richard Chen
longGGGGGG
Richard
Nakul Sinha
Divyam Agrawal
Richardczl98
Krishang Zinzuwadia
nimeshas
Claude Opus 4.6
github-actions[bot]
Jignas Paturu
zijiexia
parent
575fdc2c4c
commit
a3291b5654
@@ -0,0 +1,366 @@
|
||||
---
|
||||
title: CLI reference
|
||||
sidebarTitle: CLI
|
||||
description: Run one-off generation tasks and launch the HTTP server from the command line.
|
||||
---
|
||||
|
||||
The `sglang` CLI provides two main subcommands for diffusion inference:
|
||||
|
||||
- **`sglang generate`** -- run a one-off generation without a persistent server
|
||||
- **`sglang serve`** -- launch the OpenAI-compatible HTTP server
|
||||
|
||||
## Prerequisites
|
||||
|
||||
A working SGLang Diffusion installation with the `sglang` CLI available in your `$PATH`. See the [installation guide](../installation) for setup instructions.
|
||||
|
||||
## Generate
|
||||
|
||||
Run a one-off generation task without launching a persistent server. Pass both server arguments and sampling parameters after the `generate` subcommand:
|
||||
|
||||
```bash
|
||||
SERVER_ARGS=(
|
||||
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers
|
||||
--text-encoder-cpu-offload
|
||||
--pin-cpu-memory
|
||||
--num-gpus 4
|
||||
--ulysses-degree=2
|
||||
--ring-degree=2
|
||||
)
|
||||
|
||||
SAMPLING_ARGS=(
|
||||
--prompt "A curious raccoon"
|
||||
--save-output
|
||||
--output-path outputs
|
||||
--output-file-name "A curious raccoon.mp4"
|
||||
)
|
||||
|
||||
sglang generate "${SERVER_ARGS[@]}" "${SAMPLING_ARGS[@]}"
|
||||
```
|
||||
|
||||
You can also enable Cache-DiT acceleration via an environment variable:
|
||||
|
||||
```bash
|
||||
SGLANG_CACHE_DIT_ENABLED=true sglang generate "${SERVER_ARGS[@]}" "${SAMPLING_ARGS[@]}"
|
||||
```
|
||||
|
||||
<Note>
|
||||
HTTP server-related arguments are ignored in `generate` mode. The process shuts down automatically once generation completes.
|
||||
</Note>
|
||||
|
||||
## Serve
|
||||
|
||||
Launch the SGLang Diffusion HTTP server and interact through the OpenAI-compatible API.
|
||||
|
||||
```bash
|
||||
SERVER_ARGS=(
|
||||
--model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers
|
||||
--text-encoder-cpu-offload
|
||||
--pin-cpu-memory
|
||||
--num-gpus 4
|
||||
--ulysses-degree=2
|
||||
--ring-degree=2
|
||||
)
|
||||
|
||||
sglang serve "${SERVER_ARGS[@]}"
|
||||
```
|
||||
|
||||
- `--model-path` -- which model to load (e.g. `Wan-AI/Wan2.1-T2V-1.3B-Diffusers`)
|
||||
- `--port` -- HTTP port to listen on (default: `30010`)
|
||||
|
||||
For full API usage including image/video generation and LoRA management, see the [OpenAI API documentation](./openai-api).
|
||||
|
||||
---
|
||||
|
||||
## Supported arguments
|
||||
|
||||
### Server arguments
|
||||
|
||||
<Accordion title="Server arguments reference">
|
||||
|
||||
| Argument | Description |
|
||||
|:--|:--|
|
||||
| `--model-path MODEL_PATH` | Path to the model or HuggingFace model ID |
|
||||
| `--lora-path LORA_PATH` | Path to a LoRA adapter (local or HuggingFace ID). If omitted, LoRA is not applied |
|
||||
| `--lora-nickname NAME` | Nickname for the LoRA adapter (default: `default`) |
|
||||
| `--num-gpus NUM` | Number of GPUs to use |
|
||||
| `--tp-size SIZE` | Tensor parallelism size (encoder only; keep at most 1 when text encoder offload is enabled) |
|
||||
| `--sp-degree SIZE` | Sequence parallelism size (typically should match the number of GPUs) |
|
||||
| `--ulysses-degree SIZE` | DeepSpeed-Ulysses-style SP degree in USP |
|
||||
| `--ring-degree SIZE` | Ring attention-style SP degree in USP |
|
||||
| `--attention-backend BACKEND` | Attention backend. Native pipelines: `fa`, `torch_sdpa`, `sage_attn`, etc. Diffusers pipelines: `flash`, `_flash_3_hub`, `sage`, `xformers` |
|
||||
| `--attention-backend-config CONFIG` | Config for the attention backend. Accepts a JSON string, a JSON/YAML file path, or `key=value` pairs |
|
||||
| `--cache-dit-config PATH` | Path to a Cache-DiT YAML/JSON config (diffusers backend only) |
|
||||
| `--dit-precision DTYPE` | Precision for the DiT model (`fp32`, `fp16`, `bf16`) |
|
||||
| `--text-encoder-cpu-offload` | Offload text encoders to CPU |
|
||||
| `--pin-cpu-memory` | Pin CPU memory for faster transfers |
|
||||
|
||||
</Accordion>
|
||||
|
||||
### Sampling parameters
|
||||
|
||||
<Accordion title="Generation parameters">
|
||||
|
||||
| Argument | Description |
|
||||
|:--|:--|
|
||||
| `--prompt PROMPT` | Text description for the image or video to generate |
|
||||
| `--negative-prompt PROMPT` | Negative prompt to guide generation away from certain concepts |
|
||||
| `--num-inference-steps STEPS` | Number of denoising steps |
|
||||
| `--seed SEED` | Random seed for reproducible generation |
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Image/video configuration">
|
||||
|
||||
| Argument | Description |
|
||||
|:--|:--|
|
||||
| `--height HEIGHT` | Height of the generated output |
|
||||
| `--width WIDTH` | Width of the generated output |
|
||||
| `--num-frames NUM` | Number of frames to generate (video only) |
|
||||
| `--fps FPS` | Frames per second for the saved output (video only) |
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Output options">
|
||||
|
||||
| Argument | Description |
|
||||
|:--|:--|
|
||||
| `--save-output` | Save the image or video to disk |
|
||||
| `--output-path PATH` | Directory to save the generated output |
|
||||
| `--output-file-name NAME` | File name for the saved output |
|
||||
| `--return-frames` | Return the raw frames instead of saving |
|
||||
|
||||
</Accordion>
|
||||
|
||||
### Frame interpolation (video only)
|
||||
|
||||
Frame interpolation is a post-processing step that synthesizes new frames between each pair of consecutive generated frames, producing smoother motion without re-running the diffusion model.
|
||||
|
||||
The `--frame-interpolation-exp` flag controls how many rounds of interpolation to apply: each round inserts one new frame into every gap between adjacent frames, so the output frame count follows the formula:
|
||||
|
||||
$$
|
||||
\text{output frames} = (N - 1) \times 2^{\text{exp}} + 1
|
||||
$$
|
||||
|
||||
For example, 5 original frames with `exp=1` -> 4 gaps x 1 new frame + 5 originals = **9 frames**; with `exp=2` -> **17 frames**.
|
||||
|
||||
| Argument | Description |
|
||||
|:--|:--|
|
||||
| `--enable-frame-interpolation` | Enable frame interpolation. Model weights are downloaded automatically on first use |
|
||||
| `--frame-interpolation-exp EXP` | Interpolation exponent -- `1` = 2x temporal resolution, `2` = 4x, etc. (default: `1`) |
|
||||
| `--frame-interpolation-scale SCALE` | RIFE inference scale; use `0.5` for high-resolution inputs to save memory (default: `1.0`) |
|
||||
| `--frame-interpolation-model-path PATH` | Local directory or HuggingFace repo ID containing RIFE `flownet.pkl` weights (default: `elfgum/RIFE-4.22.lite`, downloaded automatically) |
|
||||
|
||||
**Example** -- generate a 5-frame video and interpolate to 9 frames ($(5 - 1) \times 2^1 + 1 = 9$):
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
|
||||
--prompt "A dog running through a park" \
|
||||
--num-frames 5 \
|
||||
--enable-frame-interpolation \
|
||||
--frame-interpolation-exp 1 \
|
||||
--save-output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration files
|
||||
|
||||
Instead of passing every parameter on the command line, you can use a JSON or YAML config file. Command-line arguments take precedence over config values.
|
||||
|
||||
```bash
|
||||
sglang generate --config config.json
|
||||
```
|
||||
|
||||
<Tabs>
|
||||
<Tab title="JSON">
|
||||
```json config.json
|
||||
{
|
||||
"model_path": "FastVideo/FastHunyuan-diffusers",
|
||||
"prompt": "A beautiful woman in a red dress walking down a street",
|
||||
"output_path": "outputs/",
|
||||
"num_gpus": 2,
|
||||
"sp_size": 2,
|
||||
"tp_size": 1,
|
||||
"num_frames": 45,
|
||||
"height": 720,
|
||||
"width": 1280,
|
||||
"num_inference_steps": 6,
|
||||
"seed": 1024,
|
||||
"fps": 24,
|
||||
"precision": "bf16",
|
||||
"vae_precision": "fp16",
|
||||
"vae_tiling": true,
|
||||
"vae_sp": true,
|
||||
"vae_config": {
|
||||
"load_encoder": false,
|
||||
"load_decoder": true,
|
||||
"tile_sample_min_height": 256,
|
||||
"tile_sample_min_width": 256
|
||||
},
|
||||
"text_encoder_precisions": ["fp16", "fp16"],
|
||||
"mask_strategy_file_path": null,
|
||||
"enable_torch_compile": false
|
||||
}
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="YAML">
|
||||
```yaml config.yaml
|
||||
model_path: "FastVideo/FastHunyuan-diffusers"
|
||||
prompt: "A beautiful woman in a red dress walking down a street"
|
||||
output_path: "outputs/"
|
||||
num_gpus: 2
|
||||
sp_size: 2
|
||||
tp_size: 1
|
||||
num_frames: 45
|
||||
height: 720
|
||||
width: 1280
|
||||
num_inference_steps: 6
|
||||
seed: 1024
|
||||
fps: 24
|
||||
precision: "bf16"
|
||||
vae_precision: "fp16"
|
||||
vae_tiling: true
|
||||
vae_sp: true
|
||||
vae_config:
|
||||
load_encoder: false
|
||||
load_decoder: true
|
||||
tile_sample_min_height: 256
|
||||
tile_sample_min_width: 256
|
||||
text_encoder_precisions:
|
||||
- "fp16"
|
||||
- "fp16"
|
||||
mask_strategy_file_path: null
|
||||
enable_torch_compile: false
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
To see all available options:
|
||||
|
||||
```bash
|
||||
sglang generate --help
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Component path overrides
|
||||
|
||||
You can override any pipeline component (e.g. `vae`, `transformer`, `text_encoder`) by specifying a custom checkpoint path with `--<component>-path`, where `<component>` matches the key in the model's `model_index.json`.
|
||||
|
||||
### Example: FLUX.2-dev with Tiny AutoEncoder
|
||||
|
||||
Replace the default VAE with a distilled tiny autoencoder for ~3x faster decoding:
|
||||
|
||||
```bash
|
||||
sglang serve \
|
||||
--model-path=black-forest-labs/FLUX.2-dev \
|
||||
--vae-path=fal/FLUX.2-Tiny-AutoEncoder
|
||||
```
|
||||
|
||||
You can also use a local path:
|
||||
|
||||
```bash
|
||||
sglang serve \
|
||||
--model-path=black-forest-labs/FLUX.2-dev \
|
||||
--vae-path=~/.cache/huggingface/hub/models--fal--FLUX.2-Tiny-AutoEncoder/snapshots/.../vae
|
||||
```
|
||||
|
||||
<Warning>
|
||||
The component key must match the one in the model's `model_index.json` (e.g. `vae`).
|
||||
The path must be either a HuggingFace repo ID or point to a complete component folder containing `config.json` and safetensors files.
|
||||
</Warning>
|
||||
|
||||
---
|
||||
|
||||
## Diffusers backend
|
||||
|
||||
SGLang Diffusion supports a diffusers backend that runs any diffusers-compatible model through SGLang's infrastructure using vanilla diffusers pipelines. This is useful for models without native SGLang implementations or models with custom pipeline classes.
|
||||
|
||||
### Backend arguments
|
||||
|
||||
| Argument | Values | Description |
|
||||
|:--|:--|:--|
|
||||
| `--backend` | `auto` (default), `sglang`, `diffusers` | `auto`: prefer native SGLang, fallback to diffusers. `sglang`: force native (fails if unavailable). `diffusers`: force vanilla diffusers pipeline |
|
||||
| `--diffusers-attention-backend` | `flash`, `_flash_3_hub`, `sage`, `xformers`, `native` | Attention backend for diffusers pipelines |
|
||||
| `--trust-remote-code` | flag | Required for models with custom pipeline classes |
|
||||
| `--vae-tiling` | flag | Enable VAE tiling for large image support (decodes tile-by-tile) |
|
||||
| `--vae-slicing` | flag | Enable VAE slicing for lower memory usage (decodes slice-by-slice) |
|
||||
| `--dit-precision` | `fp16`, `bf16`, `fp32` | Precision for the diffusion transformer |
|
||||
| `--vae-precision` | `fp16`, `bf16`, `fp32` | Precision for the VAE |
|
||||
|
||||
### Example: running Ovis-Image-7B
|
||||
|
||||
[Ovis-Image-7B](https://huggingface.co/AIDC-AI/Ovis-Image-7B) is a 7B text-to-image model optimized for high-quality text rendering.
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path AIDC-AI/Ovis-Image-7B \
|
||||
--backend diffusers \
|
||||
--trust-remote-code \
|
||||
--diffusers-attention-backend flash \
|
||||
--prompt "A serene Japanese garden with cherry blossoms" \
|
||||
--height 1024 \
|
||||
--width 1024 \
|
||||
--num-inference-steps 30 \
|
||||
--save-output \
|
||||
--output-path outputs \
|
||||
--output-file-name ovis_garden.png
|
||||
```
|
||||
|
||||
### Extra diffusers arguments
|
||||
|
||||
For pipeline-specific parameters not exposed via CLI, use `diffusers_kwargs` in a config file:
|
||||
|
||||
```json config.json
|
||||
{
|
||||
"model_path": "AIDC-AI/Ovis-Image-7B",
|
||||
"backend": "diffusers",
|
||||
"prompt": "A beautiful landscape",
|
||||
"diffusers_kwargs": {
|
||||
"cross_attention_kwargs": {"scale": 0.5}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```bash
|
||||
sglang generate --config config.json
|
||||
```
|
||||
|
||||
### Cache-DiT acceleration
|
||||
|
||||
Users on the diffusers backend can leverage Cache-DiT acceleration by loading custom cache configs from a YAML file. See the [Cache-DiT documentation](../cache-dit) for details.
|
||||
|
||||
---
|
||||
|
||||
## Cloud storage support
|
||||
|
||||
The server supports automatically uploading generated artifacts to S3-compatible cloud storage (AWS S3, MinIO, Alibaba Cloud OSS, Tencent Cloud COS).
|
||||
|
||||
The workflow is: **Generate -> Upload -> Delete local file**. The API response returns the public URL of the uploaded object.
|
||||
|
||||
1. **Install boto3**
|
||||
|
||||
```bash
|
||||
pip install boto3
|
||||
```
|
||||
|
||||
2. **Set environment variables**
|
||||
|
||||
```bash
|
||||
export SGLANG_CLOUD_STORAGE_TYPE=s3
|
||||
export SGLANG_S3_BUCKET_NAME=my-bucket
|
||||
export SGLANG_S3_ACCESS_KEY_ID=your-access-key
|
||||
export SGLANG_S3_SECRET_ACCESS_KEY=your-secret-key
|
||||
|
||||
# Optional: custom endpoint for MinIO/OSS/COS
|
||||
export SGLANG_S3_ENDPOINT_URL=https://minio.example.com
|
||||
```
|
||||
|
||||
3. **Launch the server**
|
||||
|
||||
```bash
|
||||
sglang serve --model-path MODEL_PATH
|
||||
```
|
||||
|
||||
See the [environment variables reference](../environment-variables) for all storage-related variables.
|
||||
@@ -0,0 +1,421 @@
|
||||
---
|
||||
title: OpenAI API
|
||||
sidebarTitle: OpenAI API
|
||||
description: Image and video generation endpoints with LoRA adapter management.
|
||||
---
|
||||
|
||||
The SGLang Diffusion HTTP server implements an OpenAI-compatible API for image and video generation, as well as dynamic LoRA adapter management.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.11+ if you plan to use the OpenAI Python SDK.
|
||||
- A running SGLang Diffusion server (see the [CLI reference](./cli) for launch instructions).
|
||||
|
||||
## Start the server
|
||||
|
||||
```bash
|
||||
SERVER_ARGS=(
|
||||
--model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers
|
||||
--text-encoder-cpu-offload
|
||||
--pin-cpu-memory
|
||||
--num-gpus 4
|
||||
--ulysses-degree=2
|
||||
--ring-degree=2
|
||||
--port 30010
|
||||
)
|
||||
|
||||
sglang serve "${SERVER_ARGS[@]}"
|
||||
```
|
||||
|
||||
- `--model-path` -- path to the model or HuggingFace model ID
|
||||
- `--port` -- HTTP port to listen on (default: `30000`)
|
||||
|
||||
### Get model information
|
||||
|
||||
**Endpoint:** `GET /models`
|
||||
|
||||
Returns model path, task type, pipeline configuration, and precision settings.
|
||||
|
||||
<CodeGroup>
|
||||
```bash curl
|
||||
curl -sS -X GET "http://localhost:30010/models"
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"model_path": "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
|
||||
"task_type": "T2V",
|
||||
"pipeline_name": "wan_pipeline",
|
||||
"pipeline_class": "WanPipeline",
|
||||
"num_gpus": 4,
|
||||
"dit_precision": "bf16",
|
||||
"vae_precision": "fp16"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Image generation
|
||||
|
||||
The server implements an OpenAI-compatible Images API under the `/v1/images` namespace.
|
||||
|
||||
### Create an image
|
||||
|
||||
**Endpoint:** `POST /v1/images/generations`
|
||||
|
||||
<CodeGroup>
|
||||
```python Python
|
||||
import base64
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="sk-proj-1234567890", base_url="http://localhost:30010/v1")
|
||||
|
||||
img = client.images.generate(
|
||||
prompt="A calico cat playing a piano on stage",
|
||||
size="1024x1024",
|
||||
n=1,
|
||||
response_format="b64_json",
|
||||
)
|
||||
|
||||
image_bytes = base64.b64decode(img.data[0].b64_json)
|
||||
with open("output.png", "wb") as f:
|
||||
f.write(image_bytes)
|
||||
```
|
||||
|
||||
```bash curl
|
||||
curl -sS -X POST "http://localhost:30010/v1/images/generations" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-d '{
|
||||
"prompt": "A calico cat playing a piano on stage",
|
||||
"size": "1024x1024",
|
||||
"n": 1,
|
||||
"response_format": "b64_json"
|
||||
}'
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
<Note>
|
||||
If `response_format=url` is used and cloud storage is not configured, the API returns a relative URL like `/v1/images/<IMAGE_ID>/content`.
|
||||
</Note>
|
||||
|
||||
### Edit an image
|
||||
|
||||
**Endpoint:** `POST /v1/images/edits`
|
||||
|
||||
Accepts a multipart form upload with input images and a text prompt. Returns either a base64-encoded image or a URL.
|
||||
|
||||
<Tabs>
|
||||
<Tab title="b64_json response">
|
||||
```bash
|
||||
curl -sS -X POST "http://localhost:30010/v1/images/edits" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-F "image=@local_input_image.png" \
|
||||
-F "url=image_url.jpg" \
|
||||
-F "prompt=A calico cat playing a piano on stage" \
|
||||
-F "size=1024x1024" \
|
||||
-F "response_format=b64_json"
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="URL response">
|
||||
```bash
|
||||
curl -sS -X POST "http://localhost:30010/v1/images/edits" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-F "image=@local_input_image.png" \
|
||||
-F "url=image_url.jpg" \
|
||||
-F "prompt=A calico cat playing a piano on stage" \
|
||||
-F "size=1024x1024" \
|
||||
-F "response_format=url"
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
### Download image content
|
||||
|
||||
When `response_format=url` is used, the API returns a relative URL like `/v1/images/<IMAGE_ID>/content`.
|
||||
|
||||
**Endpoint:** `GET /v1/images/{image_id}/content`
|
||||
|
||||
```bash
|
||||
curl -sS -L "http://localhost:30010/v1/images/<IMAGE_ID>/content" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-o output.png
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Video generation
|
||||
|
||||
The server implements a subset of the OpenAI Videos API under the `/v1/videos` namespace.
|
||||
|
||||
### Create a video
|
||||
|
||||
**Endpoint:** `POST /v1/videos`
|
||||
|
||||
<CodeGroup>
|
||||
```python Python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="sk-proj-1234567890", base_url="http://localhost:30010/v1")
|
||||
|
||||
video = client.videos.create(
|
||||
prompt="A calico cat playing a piano on stage",
|
||||
size="1280x720"
|
||||
)
|
||||
print(f"Video ID: {video.id}, Status: {video.status}")
|
||||
```
|
||||
|
||||
```bash curl
|
||||
curl -sS -X POST "http://localhost:30010/v1/videos" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-d '{
|
||||
"prompt": "A calico cat playing a piano on stage",
|
||||
"size": "1280x720"
|
||||
}'
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### List videos
|
||||
|
||||
**Endpoint:** `GET /v1/videos`
|
||||
|
||||
<CodeGroup>
|
||||
```python Python
|
||||
videos = client.videos.list()
|
||||
for item in videos.data:
|
||||
print(item.id, item.status)
|
||||
```
|
||||
|
||||
```bash curl
|
||||
curl -sS -X GET "http://localhost:30010/v1/videos" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890"
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Download video content
|
||||
|
||||
**Endpoint:** `GET /v1/videos/{video_id}/content`
|
||||
|
||||
<CodeGroup>
|
||||
```python Python
|
||||
import time
|
||||
|
||||
# Poll for completion
|
||||
while True:
|
||||
page = client.videos.list()
|
||||
item = next((v for v in page.data if v.id == video_id), None)
|
||||
if item and item.status == "completed":
|
||||
break
|
||||
time.sleep(5)
|
||||
|
||||
# Download content
|
||||
resp = client.videos.download_content(video_id=video_id)
|
||||
with open("output.mp4", "wb") as f:
|
||||
f.write(resp.read())
|
||||
```
|
||||
|
||||
```bash curl
|
||||
curl -sS -L "http://localhost:30010/v1/videos/<VIDEO_ID>/content" \
|
||||
-H "Authorization: Bearer sk-proj-1234567890" \
|
||||
-o output.mp4
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
---
|
||||
|
||||
## LoRA management
|
||||
|
||||
The server supports dynamic loading, merging, and unmerging of LoRA adapters.
|
||||
|
||||
<Info>
|
||||
- **Mutual exclusion:** Only one LoRA can be merged (active) at a time.
|
||||
- **Switching:** To switch LoRAs, you must first unmerge the current one, then set the new one.
|
||||
- **Caching:** The server caches loaded LoRA weights in memory. Switching back to a previously loaded LoRA (same path) has negligible cost.
|
||||
</Info>
|
||||
|
||||
### Set LoRA adapter
|
||||
|
||||
Loads one or more LoRA adapters and merges their weights into the model. Supports both single LoRA (backward compatible) and multiple LoRA adapters.
|
||||
|
||||
**Endpoint:** `POST /v1/set_lora`
|
||||
|
||||
**Parameters:**
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|:--|:--|:--|
|
||||
| `lora_nickname` | string or list | A unique identifier for the LoRA adapter(s). Required |
|
||||
| `lora_path` | string or list | Path to `.safetensors` file(s) or HuggingFace repo ID(s). Required for first load; optional when re-activating a cached nickname |
|
||||
| `target` | string or list | Which transformer(s) to apply the LoRA to: `"all"` (default), `"transformer"`, `"transformer_2"`, `"critic"` |
|
||||
| `strength` | float or list | LoRA strength for merge (default: `1.0`). Values < 1.0 reduce the effect, > 1.0 amplify it |
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Single LoRA">
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/set_lora \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"lora_nickname": "lora_name",
|
||||
"lora_path": "/path/to/lora.safetensors",
|
||||
"target": "all",
|
||||
"strength": 0.8
|
||||
}'
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Multiple LoRAs">
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/set_lora \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"lora_nickname": ["lora_1", "lora_2"],
|
||||
"lora_path": ["/path/to/lora1.safetensors", "/path/to/lora2.safetensors"],
|
||||
"target": ["transformer", "transformer_2"],
|
||||
"strength": [0.8, 1.0]
|
||||
}'
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Same target">
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/set_lora \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"lora_nickname": ["style_lora", "character_lora"],
|
||||
"lora_path": ["/path/to/style.safetensors", "/path/to/character.safetensors"],
|
||||
"target": "all",
|
||||
"strength": [0.7, 0.9]
|
||||
}'
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
<Note>
|
||||
When using multiple LoRAs:
|
||||
- All list parameters (`lora_nickname`, `lora_path`, `target`, `strength`) must have the same length.
|
||||
- If `target` or `strength` is a single value, it will be applied to all LoRAs.
|
||||
- Multiple LoRAs applied to the same target will be merged in order.
|
||||
</Note>
|
||||
|
||||
### Merge LoRA weights
|
||||
|
||||
Manually merges the currently set LoRA weights into the base model.
|
||||
|
||||
**Endpoint:** `POST /v1/merge_lora_weights`
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|:--|:--|:--|
|
||||
| `target` | string | Which transformer(s) to merge: `"all"` (default), `"transformer"`, `"transformer_2"`, `"critic"` |
|
||||
| `strength` | float | LoRA strength for merge (default: `1.0`) |
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/merge_lora_weights \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"strength": 0.8}'
|
||||
```
|
||||
|
||||
<Tip>
|
||||
`set_lora` automatically performs a merge, so this endpoint is typically only needed if you have manually unmerged but want to re-apply the same LoRA without calling `set_lora` again.
|
||||
</Tip>
|
||||
|
||||
### Unmerge LoRA weights
|
||||
|
||||
Unmerges the currently active LoRA weights from the base model, restoring it to its original state. Call this before setting a different LoRA.
|
||||
|
||||
**Endpoint:** `POST /v1/unmerge_lora_weights`
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/unmerge_lora_weights \
|
||||
-H "Content-Type: application/json"
|
||||
```
|
||||
|
||||
### List LoRA adapters
|
||||
|
||||
Returns loaded LoRA adapters and current application status per module.
|
||||
|
||||
**Endpoint:** `GET /v1/list_loras`
|
||||
|
||||
```bash
|
||||
curl -sS -X GET "http://localhost:30010/v1/list_loras"
|
||||
```
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"loaded_adapters": [
|
||||
{ "nickname": "lora_a", "path": "/weights/lora_a.safetensors" },
|
||||
{ "nickname": "lora_b", "path": "/weights/lora_b.safetensors" }
|
||||
],
|
||||
"active": {
|
||||
"transformer": [
|
||||
{
|
||||
"nickname": "lora2",
|
||||
"path": "tarn59/pixel_art_style_lora_z_image_turbo",
|
||||
"merged": true,
|
||||
"strength": 1.0
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Example: switching LoRAs
|
||||
|
||||
1. **Set LoRA A**
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/set_lora \
|
||||
-d '{"lora_nickname": "lora_a", "lora_path": "path/to/A"}'
|
||||
```
|
||||
|
||||
2. **Generate with LoRA A**
|
||||
|
||||
Run your image or video generation requests.
|
||||
|
||||
3. **Unmerge LoRA A**
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/unmerge_lora_weights
|
||||
```
|
||||
|
||||
4. **Set LoRA B**
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:30010/v1/set_lora \
|
||||
-d '{"lora_nickname": "lora_b", "lora_path": "path/to/B"}'
|
||||
```
|
||||
|
||||
5. **Generate with LoRA B**
|
||||
|
||||
Run your image or video generation requests with the new adapter.
|
||||
|
||||
---
|
||||
|
||||
## Output quality
|
||||
|
||||
Control output quality and compression for both image and video generation through the `output-quality` and `output-compression` parameters.
|
||||
|
||||
### Parameters
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|:--|:--|:--|
|
||||
| `output-quality` | string | Preset quality level. Default: `"default"` |
|
||||
| `output-compression` | integer | Direct compression level override (0-100). When provided, takes precedence over `output-quality` |
|
||||
|
||||
**Quality presets:**
|
||||
|
||||
| Preset | Compression value |
|
||||
|:--|:--|
|
||||
| `"maximum"` | 100 |
|
||||
| `"high"` | 90 |
|
||||
| `"medium"` | 55 |
|
||||
| `"low"` | 35 |
|
||||
| `"default"` | Auto (50 for video, 75 for image) |
|
||||
|
||||
<Warning>
|
||||
- When both `output-quality` and `output-compression` are provided, `output-compression` takes precedence.
|
||||
- Quality settings apply to JPEG and video formats. PNG uses lossless compression and ignores these settings.
|
||||
- Lower compression values (or `"low"` quality preset) produce smaller files but may show visible artifacts.
|
||||
</Warning>
|
||||
@@ -0,0 +1,405 @@
|
||||
---
|
||||
title: "Attention Backends"
|
||||
description: "Select and configure attention backends for SGLang diffusion pipelines."
|
||||
---
|
||||
|
||||
This document describes the attention backends available in sglang diffusion (`sglang.multimodal_gen`) and how to select them.
|
||||
|
||||
## Overview
|
||||
|
||||
Attention backends are defined by `AttentionBackendEnum` (`sglang.multimodal_gen.runtime.platforms.interface.AttentionBackendEnum`) and selected via the CLI flag `--attention-backend`.
|
||||
|
||||
Backend selection is performed by the shared attention layers (e.g. `LocalAttention` / `USPAttention` / `UlyssesAttention` in `sglang.multimodal_gen.runtime.layers.attention.layer`) and therefore applies to any model component using these layers (e.g. diffusion transformer / DiT and encoders).
|
||||
|
||||
When using the diffusers backend, `--attention-backend` is passed through to diffusers'
|
||||
`set_attention_backend` (e.g., `flash`, `_flash_3_hub`, `sage`, `xformers`, `native`).
|
||||
|
||||
- **CUDA**: prefers FlashAttention (FA3/FA4) when supported; otherwise falls back to PyTorch SDPA.
|
||||
- **ROCm**: uses FlashAttention when available; otherwise falls back to PyTorch SDPA.
|
||||
- **MPS**: always uses PyTorch SDPA.
|
||||
- **NPU**: always uses PyTorch SDPA.
|
||||
|
||||
## Backend options
|
||||
|
||||
For SGLang-native pipelines, the CLI accepts the lowercase names of `AttentionBackendEnum`. The table below lists the backends implemented by the built-in platforms. `fa3`/`fa4` are accepted as aliases for `fa`.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "26%"}} />
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "54%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>CLI value</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Enum value</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Notes</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`fa` / `fa3` / `fa4`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)", whiteSpace: "nowrap"}}>`FA`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>FlashAttention. `fa3/fa4` are normalized to `fa` during argument parsing (`ServerArgs.__post_init__`).</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`torch_sdpa`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)", whiteSpace: "nowrap"}}>`TORCH_SDPA`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>PyTorch `scaled_dot_product_attention`.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`sliding_tile_attn`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)", whiteSpace: "nowrap"}}>`SLIDING_TILE_ATTN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Sliding Tile Attention (STA). Requires `st_attn`. Configure via `--attention-backend-config`.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`sage_attn`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)", whiteSpace: "nowrap"}}>`SAGE_ATTN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Requires `sageattention`. Upstream SageAttention CUDA extensions target SM80/SM86/SM89/SM90/SM120 (compute capability 8.0/8.6/8.9/9.0/12.0); see upstream `setup.py`: https://github.com/thu-ml/SageAttention/blob/main/setup.py.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`sage_attn_3`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)", whiteSpace: "nowrap"}}>`SAGE_ATTN_3`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Requires SageAttention3 installed per upstream instructions.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`video_sparse_attn`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)", whiteSpace: "nowrap"}}>`VIDEO_SPARSE_ATTN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Requires `vsa`. Configure `sparsity` via `--attention-backend-config`.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`vmoba_attn`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)", whiteSpace: "nowrap"}}>`VMOBA_ATTN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Requires `kernel.attn.vmoba_attn.vmoba`. Configure via `--attention-backend-config`.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`aiter`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)", whiteSpace: "nowrap"}}>`AITER`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Requires `aiter`.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`sparse_video_gen_2_attn`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)", whiteSpace: "nowrap"}}>`SPARSE_VIDEO_GEN_2_ATTN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Requires `svg`. See installation instructions at https://github.com/svg-project/Sparse-VideoGen.</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Selection priority
|
||||
|
||||
The selection order in `runtime/layers/attention/selector.py` is:
|
||||
|
||||
1. `global_force_attn_backend(...)` / `global_force_attn_backend_context_manager(...)`
|
||||
2. CLI `--attention-backend` (`ServerArgs.attention_backend`)
|
||||
3. Auto selection (platform capability, dtype, and installed packages)
|
||||
|
||||
## Configuration
|
||||
|
||||
Some backends require additional configuration. You can pass these parameters via `--attention-backend-config`. This argument accepts:
|
||||
- A path to a JSON or YAML configuration file.
|
||||
- A JSON string (e.g., `'{"sparsity": 0.5}'`).
|
||||
- Key-value pairs (e.g., `"sparsity=0.5,enable_x=true"`).
|
||||
|
||||
### Supported Configuration Parameters
|
||||
|
||||
**Sliding Tile Attention (`sliding_tile_attn`)**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "24%"}} />
|
||||
<col style={{width: "14%"}} />
|
||||
<col style={{width: "44%"}} />
|
||||
<col style={{width: "18%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Default</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`mask_strategy_file_path`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`str`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>**Required.** Path to the mask strategy JSON file.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>-</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`sta_mode`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`str`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Mode of STA.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>``STA_inference``</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`skip_time_steps`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Number of steps to use full attention before switching to sparse attention.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`15`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Video Sparse Attention (`video_sparse_attn`)**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "16%"}} />
|
||||
<col style={{width: "46%"}} />
|
||||
<col style={{width: "18%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Default</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`sparsity`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`float`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Validation sparsity (0.0 - 1.0).</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`0.0`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**V-MoBA (`vmoba_attn`)**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "26%"}} />
|
||||
<col style={{width: "16%"}} />
|
||||
<col style={{width: "42%"}} />
|
||||
<col style={{width: "16%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Default</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`temporal_chunk_size`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Chunk size for temporal dimension.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>-</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`temporal_topk`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Top-K tokens to select in temporal dimension.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>-</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`spatial_chunk_size`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`list[int]`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Chunk size for spatial dimension (H, W).</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>-</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`spatial_topk`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Top-K tokens to select in spatial dimension.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>-</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`st_chunk_size`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`list[int]`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Chunk size for spatiotemporal dimension (T, H, W).</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>-</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`st_topk`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Top-K tokens to select in spatiotemporal dimension.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>-</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`moba_select_mode`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`str`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Selection mode (e.g., `threshold`).</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`threshold`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`moba_threshold`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`float`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Threshold value for selection.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`0.25`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`moba_threshold_type`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`str`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Type of thresholding (e.g., `query_head`).</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`query_head`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`first_full_step`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Number of initial steps to use full attention.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`12`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`first_full_layer`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Number of initial layers to use full attention.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`0`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`temporal_layer`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Number of temporal layers.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`1`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`spatial_layer`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Number of spatial layers.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`1`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`st_layer`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Number of spatiotemporal layers.</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`1`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Platform support matrix
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "22%"}} />
|
||||
<col style={{width: "7%"}} />
|
||||
<col style={{width: "7%"}} />
|
||||
<col style={{width: "7%"}} />
|
||||
<col style={{width: "7%"}} />
|
||||
<col style={{width: "50%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Backend</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>CUDA</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>ROCm</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>MPS</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>NPU</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Notes</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`fa`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>CUDA requires SM80+ and fp16/bf16. FlashAttention is only used when the required runtime is installed; otherwise it falls back to `torch_sdpa`.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`torch_sdpa`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most compatible option across platforms.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`sliding_tile_attn`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>CUDA-only. Requires `st_attn`. Configure via `--attention-backend-config`.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`sage_attn`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>CUDA-only (optional dependency).</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`sage_attn_3`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>CUDA-only (optional dependency).</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`video_sparse_attn`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>CUDA-only. Requires `vsa`. Configure `sparsity` via `--attention-backend-config`.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`vmoba_attn`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>CUDA-only. Requires `kernel.attn.vmoba_attn.vmoba`. Configure via `--attention-backend-config`.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`aiter`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Requires `aiter`.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`sparse_video_gen_2_attn`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>CUDA-only. Requires `svg`.</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Usage
|
||||
|
||||
### Select a backend via CLI
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path <MODEL_PATH_OR_ID> \
|
||||
--prompt "..." \
|
||||
--attention-backend fa
|
||||
```
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path <MODEL_PATH_OR_ID> \
|
||||
--prompt "..." \
|
||||
--attention-backend torch_sdpa
|
||||
```
|
||||
|
||||
### Using Sliding Tile Attention (STA)
|
||||
|
||||
```bash
|
||||
# Pass the mask strategy file path via config
|
||||
sglang generate \
|
||||
--model-path <MODEL_PATH_OR_ID> \
|
||||
--prompt "..." \
|
||||
--attention-backend sliding_tile_attn \
|
||||
--attention-backend-config "mask_strategy_file_path=/abs/path/to/mask_strategy.json"
|
||||
```
|
||||
|
||||
### Notes for ROCm / MPS
|
||||
|
||||
- ROCm: use `--attention-backend torch_sdpa` or `fa` depending on what is available in your environment.
|
||||
- MPS: the platform implementation always uses `torch_sdpa`.
|
||||
@@ -0,0 +1,433 @@
|
||||
---
|
||||
title: "Cache-DiT Acceleration"
|
||||
description: "Configure Cache-DiT acceleration for diffusion inference."
|
||||
---
|
||||
|
||||
SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to **1.69x inference speedup** with minimal quality loss.
|
||||
|
||||
## Overview
|
||||
|
||||
**Cache-DiT** uses intelligent caching strategies to skip redundant computation in the denoising loop:
|
||||
|
||||
- **DBCache (Dual Block Cache)**: Dynamically decides when to cache transformer blocks based on residual differences
|
||||
- **TaylorSeer**: Uses Taylor expansion for calibration to optimize caching decisions
|
||||
- **SCM (Step Computation Masking)**: Step-level caching control for additional speedup
|
||||
|
||||
## Basic Usage
|
||||
|
||||
Enable Cache-DiT by exporting the environment variable and using `sglang generate` or `sglang serve` :
|
||||
|
||||
```bash
|
||||
SGLANG_CACHE_DIT_ENABLED=true \
|
||||
sglang generate --model-path Qwen/Qwen-Image \
|
||||
--prompt "A beautiful sunset over the mountains"
|
||||
```
|
||||
|
||||
## Diffusers Backend
|
||||
|
||||
Cache-DiT supports loading acceleration configs from a custom YAML file. For
|
||||
diffusers pipelines (`diffusers` backend), pass the YAML/JSON path via `--cache-dit-config`. This
|
||||
flow requires cache-dit >= 1.2.0 (`cache_dit.load_configs`).
|
||||
|
||||
### Single GPU inference
|
||||
|
||||
Define a `cache.yaml` file that contains:
|
||||
|
||||
```yaml
|
||||
cache_config:
|
||||
max_warmup_steps: 8
|
||||
warmup_interval: 2
|
||||
max_cached_steps: -1
|
||||
max_continuous_cached_steps: 2
|
||||
Fn_compute_blocks: 1
|
||||
Bn_compute_blocks: 0
|
||||
residual_diff_threshold: 0.12
|
||||
enable_taylorseer: true
|
||||
taylorseer_order: 1
|
||||
```
|
||||
|
||||
Then apply the config with:
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--backend diffusers \
|
||||
--model-path Qwen/Qwen-Image \
|
||||
--cache-dit-config cache.yaml \
|
||||
--prompt "A beautiful sunset over the mountains"
|
||||
```
|
||||
|
||||
### Distributed inference
|
||||
|
||||
- 1D Parallelism
|
||||
|
||||
Define a parallelism only config yaml `parallel.yaml` file that contains:
|
||||
|
||||
```yaml
|
||||
parallelism_config:
|
||||
ulysses_size: auto
|
||||
parallel_kwargs:
|
||||
attention_backend: native
|
||||
extra_parallel_modules: ["text_encoder", "vae"]
|
||||
```
|
||||
|
||||
Then, apply the distributed inference acceleration config from yaml. `ulysses_size: auto` means that cache-dit will auto detect the `world_size` as the ulysses_size. Otherwise, you should manually set it as specific int number, e.g, 4.
|
||||
|
||||
Then apply the distributed config with: (Note: please add `--num-gpus N` to specify the number of gpus for distributed inference)
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--backend diffusers \
|
||||
--num-gpus 4 \
|
||||
--model-path Qwen/Qwen-Image \
|
||||
--cache-dit-config parallel.yaml \
|
||||
--prompt "A futuristic cityscape at sunset"
|
||||
```
|
||||
|
||||
- 2D Parallelism
|
||||
|
||||
You can also define a 2D parallelism config yaml `parallel_2d.yaml` file that contains:
|
||||
|
||||
```yaml
|
||||
parallelism_config:
|
||||
ulysses_size: auto
|
||||
tp_size: 2
|
||||
parallel_kwargs:
|
||||
attention_backend: native
|
||||
extra_parallel_modules: ["text_encoder", "vae"]
|
||||
```
|
||||
Then, apply the 2D parallelism config from yaml. Here `tp_size: 2` means using tensor parallelism with size 2. The `ulysses_size: auto` means that cache-dit will auto detect the `world_size // tp_size` as the ulysses_size.
|
||||
|
||||
- 3D Parallelism
|
||||
|
||||
You can also define a 3D parallelism config yaml `parallel_3d.yaml` file that contains:
|
||||
|
||||
```yaml
|
||||
parallelism_config:
|
||||
ulysses_size: 2
|
||||
ring_size: 2
|
||||
tp_size: 2
|
||||
parallel_kwargs:
|
||||
attention_backend: native
|
||||
extra_parallel_modules: ["text_encoder", "vae"]
|
||||
```
|
||||
Then, apply the 3D parallelism config from yaml. Here `ulysses_size: 2`, `ring_size: 2`, `tp_size: 2` means using ulysses parallelism with size 2, ring parallelism with size 2 and tensor parallelism with size 2.
|
||||
|
||||
### Hybrid Cache and Parallelism
|
||||
|
||||
Define a hybrid cache and parallel acceleration config yaml `hybrid.yaml` file that contains:
|
||||
|
||||
```yaml
|
||||
cache_config:
|
||||
max_warmup_steps: 8
|
||||
warmup_interval: 2
|
||||
max_cached_steps: -1
|
||||
max_continuous_cached_steps: 2
|
||||
Fn_compute_blocks: 1
|
||||
Bn_compute_blocks: 0
|
||||
residual_diff_threshold: 0.12
|
||||
enable_taylorseer: true
|
||||
taylorseer_order: 1
|
||||
parallelism_config:
|
||||
ulysses_size: auto
|
||||
parallel_kwargs:
|
||||
attention_backend: native
|
||||
extra_parallel_modules: ["text_encoder", "vae"]
|
||||
```
|
||||
|
||||
Then, apply the hybrid cache and parallel acceleration config from yaml.
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--backend diffusers \
|
||||
--num-gpus 4 \
|
||||
--model-path Qwen/Qwen-Image \
|
||||
--cache-dit-config hybrid.yaml \
|
||||
--prompt "A beautiful sunset over the mountains"
|
||||
```
|
||||
|
||||
## Advanced Configuration
|
||||
|
||||
### DBCache Parameters
|
||||
|
||||
DBCache controls block-level caching behavior:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "12%"}} />
|
||||
<col style={{width: "34%"}} />
|
||||
<col style={{width: "14%"}} />
|
||||
<col style={{width: "40%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Env Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Fn</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_FN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of first blocks to always compute</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Bn</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_BN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of last blocks to always compute</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>W</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_WARMUP`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Warmup steps before caching starts</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>R</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_RDT`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.24</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Residual difference threshold</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MC</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_MC`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>3</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Maximum continuous cached steps</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
### TaylorSeer Configuration
|
||||
|
||||
TaylorSeer improves caching accuracy using Taylor expansion:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "12%"}} />
|
||||
<col style={{width: "36%"}} />
|
||||
<col style={{width: "14%"}} />
|
||||
<col style={{width: "38%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Env Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Enable</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_TAYLORSEER`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>false</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable TaylorSeer calibrator</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Order</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_TS_ORDER`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Taylor expansion order (1 or 2)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
### Combined Configuration Example
|
||||
|
||||
DBCache and TaylorSeer are complementary strategies that work together, you can configure both sets of parameters
|
||||
simultaneously:
|
||||
|
||||
```bash
|
||||
SGLANG_CACHE_DIT_ENABLED=true \
|
||||
SGLANG_CACHE_DIT_FN=2 \
|
||||
SGLANG_CACHE_DIT_BN=1 \
|
||||
SGLANG_CACHE_DIT_WARMUP=4 \
|
||||
SGLANG_CACHE_DIT_RDT=0.4 \
|
||||
SGLANG_CACHE_DIT_MC=4 \
|
||||
SGLANG_CACHE_DIT_TAYLORSEER=true \
|
||||
SGLANG_CACHE_DIT_TS_ORDER=2 \
|
||||
sglang generate --model-path black-forest-labs/FLUX.1-dev \
|
||||
--prompt "A curious raccoon in a forest"
|
||||
```
|
||||
|
||||
### SCM (Step Computation Masking)
|
||||
|
||||
SCM provides step-level caching control for additional speedup. It decides which denoising steps to compute fully and
|
||||
which to use cached results.
|
||||
|
||||
**SCM Presets**
|
||||
|
||||
SCM is configured with presets:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "18%"}} />
|
||||
<col style={{width: "22%"}} />
|
||||
<col style={{width: "22%"}} />
|
||||
<col style={{width: "38%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Preset</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Compute Ratio</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Speed</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Quality</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`none`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>100%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Baseline</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Best</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`slow`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>~75%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>~1.3x</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>High</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`medium`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>~50%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>~2x</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Good</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`fast`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>~35%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>~3x</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Acceptable</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`ultra`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>~25%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>~4x</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Lower</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Usage**
|
||||
|
||||
```bash
|
||||
SGLANG_CACHE_DIT_ENABLED=true \
|
||||
SGLANG_CACHE_DIT_SCM_PRESET=medium \
|
||||
sglang generate --model-path Qwen/Qwen-Image \
|
||||
--prompt "A futuristic cityscape at sunset"
|
||||
```
|
||||
|
||||
**Custom SCM Bins**
|
||||
|
||||
For fine-grained control over which steps to compute vs cache:
|
||||
|
||||
```bash
|
||||
SGLANG_CACHE_DIT_ENABLED=true \
|
||||
SGLANG_CACHE_DIT_SCM_COMPUTE_BINS="8,3,3,2,2" \
|
||||
SGLANG_CACHE_DIT_SCM_CACHE_BINS="1,2,2,2,3" \
|
||||
sglang generate --model-path Qwen/Qwen-Image \
|
||||
--prompt "A futuristic cityscape at sunset"
|
||||
```
|
||||
|
||||
**SCM Policy**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "16%"}} />
|
||||
<col style={{width: "42%"}} />
|
||||
<col style={{width: "42%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Policy</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Env Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`dynamic`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_SCM_POLICY=dynamic`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Adaptive caching based on content (default)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`static`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_SCM_POLICY=static`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Fixed caching pattern</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Environment Variables
|
||||
|
||||
All Cache-DiT parameters can be configured via environment variables.
|
||||
See [Environment variables](./environment-variables) for the complete list.
|
||||
|
||||
## Supported Models
|
||||
|
||||
SGLang Diffusion x Cache-DiT supports almost all models originally supported in SGLang Diffusion:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "30%"}} />
|
||||
<col style={{width: "70%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model Family</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Example Models</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Wan</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Wan2.1, Wan2.2</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Flux</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>FLUX.1-dev, FLUX.2-dev</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Z-Image</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Z-Image-Turbo</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen-Image, Qwen-Image-Edit</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Hunyuan</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>HunyuanVideo</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Performance Tips
|
||||
|
||||
1. **Start with defaults**: The default parameters work well for most models
|
||||
2. **Use TaylorSeer**: It typically improves both speed and quality
|
||||
3. **Tune R threshold**: Lower values = better quality, higher values = faster
|
||||
4. **SCM for extra speed**: Use `medium` preset for good speed/quality balance
|
||||
5. **Warmup matters**: Higher warmup = more stable caching decisions
|
||||
|
||||
## Limitations
|
||||
|
||||
- **SGLang-native pipelines**: Distributed support (TP/SP) is not yet validated; Cache-DiT will be automatically
|
||||
disabled when `world_size > 1`.
|
||||
- **SCM minimum steps**: SCM requires >= 8 inference steps to be effective
|
||||
- **Model support**: Only models registered in Cache-DiT's BlockAdapterRegister are supported
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### SCM disabled for low step count
|
||||
|
||||
For models with < 8 inference steps (e.g., DMD distilled models), SCM will be automatically disabled. DBCache
|
||||
acceleration still works.
|
||||
|
||||
## References
|
||||
|
||||
- [Cache-DiT](https://github.com/vipshop/cache-dit)
|
||||
- [SGLang diffusion](../../sglang-diffusion/intro)
|
||||
@@ -0,0 +1,88 @@
|
||||
---
|
||||
title: "Caching Acceleration"
|
||||
description: "Compare caching acceleration strategies for diffusion models."
|
||||
---
|
||||
|
||||
SGLang provides multiple caching acceleration strategies for Diffusion Transformer (DiT) models. These strategies can significantly reduce inference time by skipping redundant computation.
|
||||
|
||||
## Overview
|
||||
|
||||
SGLang supports two complementary caching approaches:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "18%"}} />
|
||||
<col style={{width: "18%"}} />
|
||||
<col style={{width: "42%"}} />
|
||||
<col style={{width: "22%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Strategy</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Scope</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Mechanism</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Best For</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Cache-DiT</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Block-level</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Skip individual transformer blocks dynamically</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Advanced, higher speedup</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>TeaCache</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Timestep-level</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Skip entire denoising steps based on L1 similarity</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Simple, built-in</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
## Cache-DiT
|
||||
|
||||
[Cache-DiT](https://github.com/vipshop/cache-dit) provides block-level caching with
|
||||
advanced strategies like DBCache and TaylorSeer. It can achieve up to **1.69x speedup**.
|
||||
|
||||
See [Cache-DiT](./cache-dit) for detailed configuration.
|
||||
|
||||
### Quick Start
|
||||
|
||||
```bash
|
||||
SGLANG_CACHE_DIT_ENABLED=true \
|
||||
sglang generate --model-path Qwen/Qwen-Image \
|
||||
--prompt "A beautiful sunset over the mountains"
|
||||
```
|
||||
|
||||
### Key Features
|
||||
|
||||
- **DBCache**: Dynamic block-level caching based on residual differences
|
||||
- **TaylorSeer**: Taylor expansion-based calibration for optimized caching
|
||||
- **SCM**: Step-level computation masking for additional speedup
|
||||
|
||||
## TeaCache
|
||||
|
||||
TeaCache (Temporal similarity-based caching) accelerates diffusion inference by detecting when consecutive denoising steps are similar enough to skip computation entirely.
|
||||
|
||||
See [TeaCache](./tea-cache) for detailed documentation.
|
||||
|
||||
### Quick Overview
|
||||
|
||||
- Tracks L1 distance between modulated inputs across timesteps
|
||||
- When accumulated distance is below threshold, reuses cached residual
|
||||
- Supports CFG with separate positive/negative caches
|
||||
|
||||
### Supported Models
|
||||
|
||||
- Wan (wan2.1, wan2.2)
|
||||
- Hunyuan (HunyuanVideo)
|
||||
- Z-Image
|
||||
|
||||
For Flux and Qwen models, TeaCache is automatically disabled when CFG is enabled.
|
||||
|
||||
## References
|
||||
|
||||
- [Cache-DiT Repository](https://github.com/vipshop/cache-dit)
|
||||
- [TeaCache Paper](https://arxiv.org/abs/2411.14324)
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
title: "CI Performance Baselines"
|
||||
description: "Generate and update diffusion performance baselines used in CI."
|
||||
---
|
||||
|
||||
`python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py` starts a local diffusion server, issues requests for selected test cases, aggregates stage/denoise-step/E2E timings from the perf log, and writes the results back to the `scenarios` section of `perf_baselines.json`.
|
||||
|
||||
## Usage
|
||||
|
||||
Update a single case:
|
||||
|
||||
```bash
|
||||
python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --case qwen_image_t2i
|
||||
```
|
||||
|
||||
Select by regex:
|
||||
|
||||
```bash
|
||||
python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --match 'qwen_image_.*'
|
||||
```
|
||||
|
||||
Run all keys from the baseline file `scenarios`:
|
||||
|
||||
```bash
|
||||
python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --all-from-baseline
|
||||
```
|
||||
|
||||
Specify input/output paths and timeout:
|
||||
|
||||
```bash
|
||||
python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --baseline python/sglang/multimodal_gen/test/server/perf_baselines.json --out /tmp/perf_baselines.json --timeout 600
|
||||
```
|
||||
@@ -0,0 +1,140 @@
|
||||
---
|
||||
title: "Environment Variables"
|
||||
description: "Configure SGLang diffusion behavior with environment variables."
|
||||
---
|
||||
|
||||
These variables configure caching acceleration for Diffusion Transformer (DiT) models.
|
||||
SGLang supports multiple caching strategies - see [performance optimization documentation](./performance-optimization) for an overview.
|
||||
|
||||
See [Environment Variables](../references/environment_variables) for a list of all environment variables.
|
||||
|
||||
## Cache-DiT configuration
|
||||
|
||||
See [Cache-DiT documentation](./cache-dit) for detailed configuration.
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "42%"}} />
|
||||
<col style={{width: "16%"}} />
|
||||
<col style={{width: "42%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Environment Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_CACHE_DIT_ENABLED`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>false</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Enable Cache-DiT acceleration</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_CACHE_DIT_FN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>First N blocks to always compute</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_CACHE_DIT_BN`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Last N blocks to always compute</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_CACHE_DIT_WARMUP`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Warmup steps before caching</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_CACHE_DIT_RDT`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.24</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Residual difference threshold</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_CACHE_DIT_MC`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>3</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Max continuous cached steps</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_CACHE_DIT_TAYLORSEER`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>false</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Enable TaylorSeer calibrator</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_CACHE_DIT_TS_ORDER`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>TaylorSeer order (1 or 2)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_CACHE_DIT_SCM_PRESET`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>none</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>SCM preset (none/slow/medium/fast/ultra)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_CACHE_DIT_SCM_POLICY`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>dynamic</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>SCM caching policy</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_CACHE_DIT_SCM_COMPUTE_BINS`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>not set</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Custom SCM compute bins</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_CACHE_DIT_SCM_CACHE_BINS`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>not set</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Custom SCM cache bins</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Cloud Storage
|
||||
|
||||
These variables configure S3-compatible cloud storage for automatically uploading generated images and videos.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "35%"}} />
|
||||
<col style={{width: "16%"}} />
|
||||
<col style={{width: "49%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Environment Variable</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Default</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_CLOUD_STORAGE_TYPE`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>not set</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Set to `s3` to enable cloud storage</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_S3_BUCKET_NAME`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>not set</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The name of the S3 bucket</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_S3_ENDPOINT_URL`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>not set</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Custom endpoint URL (for MinIO, OSS, etc.)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_S3_REGION_NAME`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>us-east-1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>AWS region name</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_S3_ACCESS_KEY_ID`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>not set</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>AWS Access Key ID</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_S3_SECRET_ACCESS_KEY`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>not set</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>AWS Secret Access Key</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
@@ -0,0 +1,65 @@
|
||||
---
|
||||
title: SGLang Diffusion
|
||||
description: Accelerated image and video generation with diffusion models.
|
||||
---
|
||||
|
||||
SGLang Diffusion is an inference framework for accelerated image and video generation using diffusion models. It provides an end-to-end unified pipeline with optimized kernels and an efficient scheduler loop.
|
||||
|
||||
## Key features
|
||||
|
||||
* **Broad model support:** Wan series, FastWan series, Hunyuan, Qwen-Image, Qwen-Image-Edit, Flux, Z-Image, GLM-Image, and more
|
||||
* **Fast inference:** optimized kernels, efficient scheduler loop, and Cache-DiT acceleration
|
||||
* **Ease of use:** OpenAI-compatible API, CLI, and Python SDK
|
||||
* **Multi-platform:** NVIDIA GPUs (H100, H200, A100, B200, 4090), AMD GPUs (MI300X, MI325X), and Ascend NPU (A2, A3)
|
||||
|
||||
## Quick start
|
||||
|
||||
1. **Install SGLang Diffusion**
|
||||
|
||||
```bash
|
||||
uv pip install "sglang[diffusion]" --prerelease=allow
|
||||
```
|
||||
|
||||
See the [installation guide](./installation) for more installation methods and ROCm-specific instructions.
|
||||
|
||||
2. **Run a one-off generation**
|
||||
|
||||
```bash
|
||||
sglang generate --model-path Qwen/Qwen-Image \
|
||||
--prompt "A beautiful sunset over the mountains" \
|
||||
--save-output
|
||||
```
|
||||
|
||||
3. **Serve with the OpenAI-compatible API**
|
||||
|
||||
```bash
|
||||
sglang serve --model-path Qwen/Qwen-Image --port 30010
|
||||
```
|
||||
|
||||
## CLI quick reference
|
||||
|
||||
### Generate (one-off generation)
|
||||
|
||||
```bash
|
||||
sglang generate --model-path <MODEL> --prompt "<PROMPT>" --save-output
|
||||
```
|
||||
|
||||
### Serve (HTTP server)
|
||||
|
||||
```bash
|
||||
sglang serve --model-path <MODEL> --port 30010
|
||||
```
|
||||
|
||||
### Enable Cache-DiT acceleration
|
||||
|
||||
```bash
|
||||
SGLANG_CACHE_DIT_ENABLED=true sglang generate --model-path <MODEL> --prompt "<PROMPT>"
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
* [SGLang GitHub](https://github.com/sgl-project/sglang)
|
||||
* [Cache-DiT](https://github.com/vipshop/cache-dit)
|
||||
* [FastVideo](https://github.com/hao-ai-lab/FastVideo)
|
||||
* [xDiT](https://github.com/xdit-project/xDiT)
|
||||
* [Diffusers](https://github.com/huggingface/diffusers)
|
||||
@@ -0,0 +1,110 @@
|
||||
---
|
||||
title: Install SGLang Diffusion
|
||||
description: Install SGLang Diffusion on NVIDIA, AMD, MUSA, and Ascend platforms.
|
||||
---
|
||||
|
||||
You can install SGLang Diffusion using one of the methods below.
|
||||
|
||||
## Standard installation (NVIDIA GPUs)
|
||||
|
||||
**Platform:** NVIDIA GPUs (CUDA)
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Pip or uv">
|
||||
Use `uv` for faster installation:
|
||||
|
||||
```bash
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
uv pip install "sglang[diffusion]" --prerelease=allow
|
||||
```
|
||||
</Tab>
|
||||
|
||||
<Tab title="From source">
|
||||
```bash
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
pip install --upgrade pip
|
||||
pip install -e "python[diffusion]"
|
||||
```
|
||||
|
||||
Or with `uv`:
|
||||
|
||||
```bash
|
||||
uv pip install -e "python[diffusion]" --prerelease=allow
|
||||
```
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
The Docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from the [Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/Dockerfile). Replace `<secret>` below with your HuggingFace Hub [token](https://huggingface.co/docs/hub/en/security-tokens).
|
||||
|
||||
```bash
|
||||
docker run --gpus all \
|
||||
--shm-size 32g \
|
||||
-p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<secret>" \
|
||||
--ipc=host \
|
||||
lmsysorg/sglang:dev \
|
||||
zsh -c '\
|
||||
echo "Installing diffusion dependencies..." && \
|
||||
pip install -e "python[diffusion]" && \
|
||||
echo "Starting SGLang-Diffusion..." && \
|
||||
sglang generate \
|
||||
--model-path black-forest-labs/FLUX.1-dev \
|
||||
--prompt "A logo With Bold Large text: SGL Diffusion" \
|
||||
--save-output \
|
||||
'
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
## Platform-specific installs
|
||||
|
||||
Use the tab that matches your accelerator.
|
||||
|
||||
<Tabs>
|
||||
<Tab title="ROCm (AMD GPUs)">
|
||||
**Platform:** AMD Instinct GPUs (ROCm)
|
||||
|
||||
For AMD Instinct GPUs (for example, MI300X), use the ROCm-enabled Docker image:
|
||||
|
||||
```bash
|
||||
docker run --device=/dev/kfd --device=/dev/dri --ipc=host \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env HF_TOKEN=<secret> \
|
||||
lmsysorg/sglang:v0.5.9-rocm700-mi30x \
|
||||
sglang generate --model-path black-forest-labs/FLUX.1-dev --prompt "A logo With Bold Large text: SGL Diffusion" --save-output
|
||||
```
|
||||
|
||||
For detailed ROCm system configuration and installation from source, see [AMD GPUs](../hardware-platforms/amd-gpus).
|
||||
</Tab>
|
||||
|
||||
<Tab title="MUSA (Moore Threads GPUs)">
|
||||
**Platform:** Moore Threads GPUs (MUSA)
|
||||
|
||||
For Moore Threads GPUs (MTGPU) with the MUSA software stack:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
pip install --upgrade pip
|
||||
rm -f python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
|
||||
pip install -e "python[all_musa]"
|
||||
```
|
||||
</Tab>
|
||||
|
||||
<Tab title="Ascend NPU">
|
||||
**Platform:** Ascend NPU
|
||||
|
||||
For Ascend NPU, follow the [NPU installation guide](../hardware-platforms/ascend-npus/SGLang-installation-with-NPUs-support).
|
||||
|
||||
Quick test:
|
||||
|
||||
```bash
|
||||
sglang generate --model-path black-forest-labs/FLUX.1-dev \
|
||||
--prompt "A logo With Bold Large text: SGL Diffusion" \
|
||||
--save-output
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
@@ -0,0 +1,105 @@
|
||||
---
|
||||
title: "Performance Optimization"
|
||||
description: "Optimize SGLang diffusion performance with caching, kernels, and profiling."
|
||||
---
|
||||
|
||||
SGLang-Diffusion provides multiple performance optimization strategies to accelerate inference. This section covers all available performance tuning options.
|
||||
|
||||
## Overview
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "22%"}} />
|
||||
<col style={{width: "18%"}} />
|
||||
<col style={{width: "60%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Optimization</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Cache-DiT</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Caching</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Block-level caching with DBCache, TaylorSeer, and SCM</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>TeaCache</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Caching</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Timestep-level caching using L1 similarity</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Attention Backends</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Kernel</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Optimized attention implementations (FlashAttention, SageAttention, etc.)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Profiling</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Diagnostics</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>PyTorch Profiler and Nsight Systems guidance</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Caching Strategies
|
||||
|
||||
SGLang supports two complementary caching approaches:
|
||||
|
||||
### Cache-DiT
|
||||
|
||||
[Cache-DiT](https://github.com/vipshop/cache-dit) provides block-level caching with advanced strategies. It can achieve up to **1.69x speedup**.
|
||||
|
||||
**Quick Start:**
|
||||
```bash
|
||||
SGLANG_CACHE_DIT_ENABLED=true \
|
||||
sglang generate --model-path Qwen/Qwen-Image \
|
||||
--prompt "A beautiful sunset over the mountains"
|
||||
```
|
||||
|
||||
**Key Features:**
|
||||
- **DBCache**: Dynamic block-level caching based on residual differences
|
||||
- **TaylorSeer**: Taylor expansion-based calibration for optimized caching
|
||||
- **SCM**: Step-level computation masking for additional speedup
|
||||
|
||||
See [Cache-DiT documentation](./cache-dit) for detailed configuration.
|
||||
|
||||
### TeaCache
|
||||
|
||||
TeaCache (Temporal similarity-based caching) accelerates diffusion inference by detecting when consecutive denoising steps are similar enough to skip computation entirely.
|
||||
|
||||
**Quick Overview:**
|
||||
- Tracks L1 distance between modulated inputs across timesteps
|
||||
- When accumulated distance is below threshold, reuses cached residual
|
||||
- Supports CFG with separate positive/negative caches
|
||||
|
||||
**Supported Models:** Wan (wan2.1, wan2.2), Hunyuan (HunyuanVideo), Z-Image
|
||||
|
||||
See [TeaCache documentation](./tea-cache) for detailed configuration.
|
||||
|
||||
## Attention Backends
|
||||
|
||||
Different attention backends offer varying performance characteristics depending on your hardware and model:
|
||||
|
||||
- **FlashAttention**: Fastest on NVIDIA GPUs with fp16/bf16
|
||||
- **SageAttention**: Alternative optimized implementation
|
||||
- **xformers**: Memory-efficient attention
|
||||
- **SDPA**: PyTorch native scaled dot-product attention
|
||||
|
||||
See [Attention backends](./attention-backends) for platform support and configuration options.
|
||||
|
||||
## Profiling
|
||||
|
||||
To diagnose performance bottlenecks, SGLang-Diffusion supports profiling tools:
|
||||
|
||||
- **PyTorch Profiler**: Built-in Python profiling
|
||||
- **Nsight Systems**: GPU kernel-level analysis
|
||||
|
||||
See [Profiling guide](./profiling) for detailed instructions.
|
||||
|
||||
## References
|
||||
|
||||
- [Cache-DiT Repository](https://github.com/vipshop/cache-dit)
|
||||
- [TeaCache Paper](https://arxiv.org/abs/2411.14324)
|
||||
@@ -0,0 +1,139 @@
|
||||
---
|
||||
title: "Profiling"
|
||||
description: "Profile SGLang diffusion workloads with PyTorch Profiler and Nsight Systems."
|
||||
---
|
||||
|
||||
This guide covers profiling techniques for multimodal generation pipelines in SGLang.
|
||||
|
||||
## PyTorch Profiler
|
||||
|
||||
PyTorch Profiler provides detailed kernel execution time, call stack, and GPU utilization metrics.
|
||||
|
||||
### Denoising Stage Profiling
|
||||
|
||||
Profile the denoising stage with sampled timesteps (default: 5 steps after 1 warmup step):
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path Qwen/Qwen-Image \
|
||||
--prompt "A Logo With Bold Large Text: SGL Diffusion" \
|
||||
--seed 0 \
|
||||
--profile
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `--profile`: Enable profiling for the denoising stage
|
||||
- `--num-profiled-timesteps N`: Number of timesteps to profile after warmup (default: 5)
|
||||
- Smaller values reduce trace file size
|
||||
- Example: `--num-profiled-timesteps 10` profiles 10 steps after 1 warmup step
|
||||
|
||||
### Full Pipeline Profiling
|
||||
|
||||
Profile all pipeline stages (text encoding, denoising, VAE decoding, etc.):
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path Qwen/Qwen-Image \
|
||||
--prompt "A Logo With Bold Large Text: SGL Diffusion" \
|
||||
--seed 0 \
|
||||
--profile \
|
||||
--profile-all-stages
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `--profile-all-stages`: Used with `--profile`, profile all pipeline stages instead of just denoising
|
||||
|
||||
### Output Location
|
||||
|
||||
By default, trace files are saved in the ./logs/ directory.
|
||||
|
||||
The exact output file path will be shown in the console output, for example:
|
||||
|
||||
```bash
|
||||
[mm-dd hh:mm:ss] Saved profiler traces to: /sgl-workspace/sglang/logs/mocked_fake_id_for_offline_generate-5_steps-global-rank0.trace.json.gz
|
||||
```
|
||||
|
||||
### View Traces
|
||||
|
||||
Load and visualize trace files at:
|
||||
- https://ui.perfetto.dev/ (recommended)
|
||||
- chrome://tracing (Chrome only)
|
||||
|
||||
For large trace files, reduce `--num-profiled-timesteps` or avoid using `--profile-all-stages`.
|
||||
|
||||
|
||||
### `--perf-dump-path` (Stage/Step Timing Dump)
|
||||
|
||||
Besides profiler traces, you can also dump a lightweight JSON report that contains:
|
||||
- stage-level timing breakdown for the full pipeline
|
||||
- step-level timing breakdown for the denoising stage (per diffusion step)
|
||||
|
||||
This is useful to quickly identify which stage dominates end-to-end latency, and whether denoising steps have uniform runtimes (and if not, which step has an abnormal spike).
|
||||
|
||||
The dumped JSON contains a `denoise_steps_ms` field formatted as an array of objects, each with a `step` key (the step index) and a `duration_ms` key.
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path <MODEL_PATH_OR_ID> \
|
||||
--prompt "<PROMPT>" \
|
||||
--perf-dump-path perf.json
|
||||
```
|
||||
|
||||
## Nsight Systems
|
||||
|
||||
Nsight Systems provides low-level CUDA profiling with kernel details, register usage, and memory access patterns.
|
||||
|
||||
### Installation
|
||||
|
||||
See the [SGLang profiling guide](../developer_guide/benchmark_and_profiling#profile-with-nsight) for installation instructions.
|
||||
|
||||
### Basic Profiling
|
||||
|
||||
Profile the entire pipeline execution:
|
||||
|
||||
```bash
|
||||
nsys profile \
|
||||
--trace-fork-before-exec=true \
|
||||
--cuda-graph-trace=node \
|
||||
--force-overwrite=true \
|
||||
-o QwenImage \
|
||||
sglang generate \
|
||||
--model-path Qwen/Qwen-Image \
|
||||
--prompt "A Logo With Bold Large Text: SGL Diffusion" \
|
||||
--seed 0
|
||||
```
|
||||
|
||||
### Targeted Stage Profiling
|
||||
|
||||
Use `--delay` and `--duration` to capture specific stages and reduce file size:
|
||||
|
||||
```bash
|
||||
nsys profile \
|
||||
--trace-fork-before-exec=true \
|
||||
--cuda-graph-trace=node \
|
||||
--force-overwrite=true \
|
||||
--delay 10 \
|
||||
--duration 30 \
|
||||
-o QwenImage_denoising \
|
||||
sglang generate \
|
||||
--model-path Qwen/Qwen-Image \
|
||||
--prompt "A Logo With Bold Large Text: SGL Diffusion" \
|
||||
--seed 0
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `--delay N`: Wait N seconds before starting capture (skip initialization overhead)
|
||||
- `--duration N`: Capture for N seconds (focus on specific stages)
|
||||
- `--force-overwrite`: Overwrite existing output files
|
||||
|
||||
## Notes
|
||||
|
||||
- **Reduce trace size**: Use `--num-profiled-timesteps` with smaller values or `--delay`/`--duration` with Nsight Systems
|
||||
- **Stage-specific analysis**: Use `--profile` alone for denoising stage, add `--profile-all-stages` for full pipeline
|
||||
- **Multiple runs**: Profile with different prompts and resolutions to identify bottlenecks across workloads
|
||||
|
||||
## FAQ
|
||||
|
||||
- If you are profiling `sglang generate` with Nsight Systems and find that the generated profiler file did not capture any CUDA kernels, you can resolve this issue by increasing the model's inference steps to extend the execution time.
|
||||
@@ -0,0 +1,350 @@
|
||||
---
|
||||
title: "Supported Models"
|
||||
description: "Check model compatibility across diffusion optimizations and backends."
|
||||
---
|
||||
|
||||
The table below shows every supported model and the optimizations supported for them.
|
||||
|
||||
Key:
|
||||
|
||||
- `Yes` = Compatible
|
||||
- `No` = Incompatible
|
||||
- `N/A` = Not applicable
|
||||
|
||||
## Models x Optimization
|
||||
|
||||
The `HuggingFace Model ID` can be passed directly to `from_pretrained()` methods, and sglang-diffusion will use the optimal default parameters when initializing and generating videos.
|
||||
|
||||
### Video Generation Models
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "17%"}} />
|
||||
<col style={{width: "22%"}} />
|
||||
<col style={{width: "8%"}} />
|
||||
<col style={{width: "6%"}} />
|
||||
<col style={{width: "8%"}} />
|
||||
<col style={{width: "6%"}} />
|
||||
<col style={{width: "8%"}} />
|
||||
<col style={{width: "8%"}} />
|
||||
<col style={{width: "9%"}} />
|
||||
<col style={{width: "8%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model Name</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>HuggingFace Model ID</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Resolutions</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>TeaCache</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Sliding Tile Attn</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Sage Attn</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Video Sparse Attention (VSA)</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Sparse Linear Attention (SLA)</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Sage Sparse Linear Attention (SageSLA)</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Sparse Video Gen 2 (SVG2)</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>FastWan2.1 T2V 1.3B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`FastVideo/FastWan2.1-T2V-1.3B-Diffusers`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>480p</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>FastWan2.2 TI2V 5B Full Attn</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`FastVideo/FastWan2.2-TI2V-5B-FullAttn-Diffusers`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>720p</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Wan2.2 TI2V 5B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Wan-AI/Wan2.2-TI2V-5B-Diffusers`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>720p</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Wan2.2 T2V A14B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Wan-AI/Wan2.2-T2V-A14B-Diffusers`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>480p, 720p</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Wan2.2 I2V A14B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Wan-AI/Wan2.2-I2V-A14B-Diffusers`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>480p, 720p</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>HunyuanVideo</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`hunyuanvideo-community/HunyuanVideo`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>720x1280, 544x960</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>FastHunyuan</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`FastVideo/FastHunyuan-diffusers`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>720x1280, 544x960</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Wan2.1 T2V 1.3B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Wan-AI/Wan2.1-T2V-1.3B-Diffusers`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>480p</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Wan2.1 T2V 14B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Wan-AI/Wan2.1-T2V-14B-Diffusers`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>480p, 720p</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Wan2.1 I2V 480P</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Wan-AI/Wan2.1-I2V-14B-480P-Diffusers`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>480p</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Wan2.1 I2V 720P</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Wan-AI/Wan2.1-I2V-14B-720P-Diffusers`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>720p</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>TurboWan2.1 T2V 1.3B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`IPostYellow/TurboWan2.1-T2V-1.3B-Diffusers`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>480p</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>TurboWan2.1 T2V 14B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`IPostYellow/TurboWan2.1-T2V-14B-Diffusers`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>480p</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>TurboWan2.1 T2V 14B 720P</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`IPostYellow/TurboWan2.1-T2V-14B-720P-Diffusers`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>720p</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>TurboWan2.2 I2V A14B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`IPostYellow/TurboWan2.2-I2V-A14B-Diffusers`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>720p</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<Note>
|
||||
1. Wan2.2 TI2V 5B has known quality issues for some I2V workloads.
|
||||
2. SageSLA is based on SpargeAttn. Install SpargeAttn first with `pip install git+https://github.com/thu-ml/SpargeAttn.git --no-build-isolation`.
|
||||
</Note>
|
||||
|
||||
### Image Generation Models
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "22%"}} />
|
||||
<col style={{width: "46%"}} />
|
||||
<col style={{width: "32%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model Name</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>HuggingFace Model ID</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Resolutions</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>FLUX.1-dev</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`black-forest-labs/FLUX.1-dev`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Any resolution</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>FLUX.2-dev</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`black-forest-labs/FLUX.2-dev`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Any resolution</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>FLUX.2-Klein</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`black-forest-labs/FLUX.2-klein-4B`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Any resolution</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Z-Image-Turbo</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Tongyi-MAI/Z-Image-Turbo`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Any resolution</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GLM-Image</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`zai-org/GLM-Image`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Any resolution</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen Image</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Qwen/Qwen-Image`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Any resolution</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen Image 2512</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Qwen/Qwen-Image-2512`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Any resolution</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen Image Edit</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Qwen/Qwen-Image-Edit`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Any resolution</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Verified LoRA Examples
|
||||
|
||||
This section lists example LoRAs that have been explicitly tested and verified with each base model in the **SGLang Diffusion** pipeline.
|
||||
|
||||
<Info>
|
||||
LoRAs that are not listed here are not necessarily incompatible.
|
||||
In practice, most standard LoRAs are expected to work, especially those following common Diffusers or SD-style conventions.
|
||||
The entries below simply reflect configurations that have been manually validated by the SGLang team.
|
||||
</Info>
|
||||
|
||||
### Verified LoRAs by Base Model
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "80%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Base Model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Supported LoRAs</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Wan2.2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lightx2v/Wan2.2-Distill-Loras`<br />`Cseti/wan2.2-14B-Arcane_Jinx-lora-v1`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Wan2.1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lightx2v/Wan2.1-Distill-Loras`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Z-Image-Turbo</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`tarn59/pixel_art_style_lora_z_image_turbo`<br />`wcde/Z-Image-Turbo-DeJPEG-Lora`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen-Image</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lightx2v/Qwen-Image-Lightning`<br />`flymy-ai/qwen-image-realism-lora`<br />`prithivMLmods/Qwen-Image-HeadshotX`<br />`starsfriday/Qwen-Image-EVA-LoRA`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen-Image-Edit</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`ostris/qwen_image_edit_inpainting`<br />`lightx2v/Qwen-Image-Edit-2511-Lightning`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Flux</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`dvyio/flux-lora-simple-illustration`<br />`XLabs-AI/flux-furry-lora`<br />`XLabs-AI/flux-RealismLora`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## Special requirements
|
||||
|
||||
### Sliding Tile Attention
|
||||
|
||||
- Currently, only Hopper GPUs (H100s) are supported.
|
||||
@@ -0,0 +1,143 @@
|
||||
---
|
||||
title: "TeaCache Acceleration"
|
||||
description: "Configure TeaCache for temporal similarity-based diffusion acceleration."
|
||||
---
|
||||
|
||||
> **Note**: This is one of two caching strategies available in SGLang.
|
||||
> For an overview of all caching options, see [SGLang diffusion overview](../../sglang-diffusion/intro).
|
||||
|
||||
TeaCache (Temporal similarity-based caching) accelerates diffusion inference by detecting when consecutive denoising steps are similar enough to skip computation entirely.
|
||||
|
||||
## Overview
|
||||
|
||||
TeaCache works by:
|
||||
1. Tracking the L1 distance between modulated inputs across consecutive timesteps
|
||||
2. Accumulating the rescaled L1 distance over steps
|
||||
3. When accumulated distance is below a threshold, reusing the cached residual
|
||||
4. Supporting CFG (Classifier-Free Guidance) with separate positive/negative caches
|
||||
|
||||
## How It Works
|
||||
|
||||
### L1 Distance Tracking
|
||||
|
||||
At each denoising step, TeaCache computes the relative L1 distance between the current and previous modulated inputs:
|
||||
|
||||
```text
|
||||
rel_l1 = |current - previous|.mean() / |previous|.mean()
|
||||
```
|
||||
|
||||
This distance is then rescaled using polynomial coefficients and accumulated:
|
||||
|
||||
```text
|
||||
accumulated += poly(coefficients)(rel_l1)
|
||||
```
|
||||
|
||||
### Cache Decision
|
||||
|
||||
- If `accumulated >= threshold`: Force computation, reset accumulator
|
||||
- If `accumulated < threshold`: Skip computation, use cached residual
|
||||
|
||||
### CFG Support
|
||||
|
||||
For models that support CFG cache separation (Wan, Hunyuan, Z-Image), TeaCache maintains separate caches for positive and negative branches:
|
||||
- `previous_modulated_input` / `previous_residual` for positive branch
|
||||
- `previous_modulated_input_negative` / `previous_residual_negative` for negative branch
|
||||
|
||||
For models that don't support CFG separation (Flux, Qwen), TeaCache is automatically disabled when CFG is enabled.
|
||||
|
||||
## Configuration
|
||||
|
||||
TeaCache is configured via `TeaCacheParams` in the sampling parameters:
|
||||
|
||||
```python
|
||||
from sglang.multimodal_gen.configs.sample.teacache import TeaCacheParams
|
||||
|
||||
params = TeaCacheParams(
|
||||
teacache_thresh=0.1, # Threshold for accumulated L1 distance
|
||||
coefficients=[1.0, 0.0, 0.0], # Polynomial coefficients for L1 rescaling
|
||||
)
|
||||
```
|
||||
|
||||
### Parameters
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "28%"}} />
|
||||
<col style={{width: "14%"}} />
|
||||
<col style={{width: "58%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`teacache_thresh`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>float</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Threshold for accumulated L1 distance. Lower = more caching, faster but potentially lower quality</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`coefficients`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>list[float]</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Polynomial coefficients for L1 rescaling. Model-specific tuning</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
### Model-Specific Configurations
|
||||
|
||||
Different models may have different optimal configurations. The coefficients are typically tuned per-model to balance speed and quality.
|
||||
|
||||
## Supported Models
|
||||
|
||||
TeaCache is built into the following model families:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "34%"}} />
|
||||
<col style={{width: "28%"}} />
|
||||
<col style={{width: "38%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model Family</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>CFG Cache Separation</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Notes</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Wan (wan2.1, wan2.2)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Full support</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Hunyuan (HunyuanVideo)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>To be supported</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Z-Image</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>To be supported</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Flux</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>To be supported</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>To be supported</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
## References
|
||||
|
||||
- [TeaCache: Accelerating Diffusion Models with Temporal Similarity](https://arxiv.org/abs/2411.14324)
|
||||
Reference in New Issue
Block a user