diff --git a/docs/diffusion/api/cli.md b/docs/diffusion/api/cli.md index a4caaca7d..8ea3134c6 100644 --- a/docs/diffusion/api/cli.md +++ b/docs/diffusion/api/cli.md @@ -83,6 +83,7 @@ Use `sglang generate --help` and `sglang serve --help` for the full argument lis ### Sampling and output - `--prompt {PROMPT}` and `--negative-prompt {PROMPT}` +- `--image-path {PATH} [{PATH} ...]`: input image(s) for image-to-video or image-to-image generation - `--num-inference-steps {STEPS}` and `--seed {SEED}` - `--height {HEIGHT}`, `--width {WIDTH}`, `--num-frames {N}`, `--fps {FPS}` - `--output-path {PATH}`, `--output-file-name {NAME}`, `--save-output`, `--return-frames` diff --git a/docs/diffusion/api/openai_api.md b/docs/diffusion/api/openai_api.md index 30d8d5312..8d18c4959 100644 --- a/docs/diffusion/api/openai_api.md +++ b/docs/diffusion/api/openai_api.md @@ -154,7 +154,7 @@ curl -sS -L "http://localhost:30010/v1/images//content" \ The server implements a subset of the OpenAI Videos API under the `/v1/videos` namespace. -**Create a video** +**Create a video (text-to-video)** **Endpoint:** `POST /v1/videos` @@ -184,6 +184,33 @@ curl -sS -X POST "http://localhost:30010/v1/videos" \ }' ``` +**Create a video (image-to-video)** + +For I2V or TI2V models (e.g., Wan2.1 I2V, LTX-2.3 two-stage), pass an input image via multipart form upload or a reference URL. + +**Curl Example (multipart form upload):** + +```bash +curl -sS -X POST "http://localhost:30010/v1/videos" \ + -H "Authorization: Bearer sk-proj-1234567890" \ + -F "prompt=A cat playing a piano" \ + -F "input_reference=@input_image.png" \ + -F "size=1280x720" +``` + +**Curl Example (reference URL):** + +```bash +curl -sS -X POST "http://localhost:30010/v1/videos" \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer sk-proj-1234567890" \ + -d '{ + "prompt": "A cat playing a piano", + "reference_url": "https://example.com/input_image.png", + "size": "1280x720" + }' +``` + **List videos** **Endpoint:** `GET /v1/videos` diff --git a/docs/diffusion/compatibility_matrix.md b/docs/diffusion/compatibility_matrix.md index de1083c01..81389332b 100644 --- a/docs/diffusion/compatibility_matrix.md +++ b/docs/diffusion/compatibility_matrix.md @@ -33,6 +33,10 @@ default parameters when initializing and generating videos. | TurboWan2.1 T2V 14B | `IPostYellow/TurboWan2.1-T2V-14B-Diffusers` | 480p | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ⭕ | | TurboWan2.1 T2V 14B 720P | `IPostYellow/TurboWan2.1-T2V-14B-720P-Diffusers` | 720p | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ⭕ | | TurboWan2.2 I2V A14B | `IPostYellow/TurboWan2.2-I2V-A14B-Diffusers` | 720p | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ⭕ | +| Wan2.1 Fun 1.3B InP | `weizhou03/Wan2.1-Fun-1.3B-InP-Diffusers` | 480p | ✅ | ✅ | ✅ | ⭕ | ❌ | ❌ | ✅ | +| Helios Base | `BestWishYsh/Helios-Base` | 720p | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | +| Helios Mid | `BestWishYsh/Helios-Mid` | 720p | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | +| Helios Distilled | `BestWishYsh/Helios-Distilled` | 720p | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | LTX-2 (one and two stages) | `Lightricks/LTX-2` | 768×512
1536×1024 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | LTX-2.3 (one and two stages) | `Lightricks/LTX-2.3` | 768×512
1536×1024 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | @@ -40,23 +44,41 @@ default parameters when initializing and generating videos. 1. Wan2.2 TI2V 5B has some quality issues when performing I2V generation. We are working on fixing this issue. 2. SageSLA is based on SpargeAttn. Install it first with `pip install git+https://github.com/thu-ml/SpargeAttn.git --no-build-isolation` -3. LTX-2 two-stage generation uses `--pipeline-class-name LTX2TwoStagePipeline`. The spatial upsampler and distilled LoRA are auto-resolved from the model snapshot by default, and can still be overridden with `--spatial-upsampler-path` and `--distilled-lora-path`. 3. LTX-2 and LTX-2.3 two-stage generation uses `--pipeline-class-name LTX2TwoStagePipeline`. The spatial upsampler and distilled LoRA are auto-resolved from the model snapshot by default, and can still be overridden with `--spatial-upsampler-path` and `--distilled-lora-path`. - For LTX models, the `Resolutions` column uses output video `width×height` semantics, matching `sglang generate --width ... --height ...`. ### Image Generation Models -| Model Name | HuggingFace Model ID | -|:---------------------|:------------------------------------| -| FLUX.1-dev | `black-forest-labs/FLUX.1-dev` | -| FLUX.2-dev | `black-forest-labs/FLUX.2-dev` | -| FLUX.2-Klein | `black-forest-labs/FLUX.2-klein-4B` | -| Z-Image-Turbo | `Tongyi-MAI/Z-Image-Turbo` | -| GLM-Image | `zai-org/GLM-Image` | -| Qwen Image | `Qwen/Qwen-Image` | -| Qwen Image 2512 | `Qwen/Qwen-Image-2512` | -| Qwen Image Edit | `Qwen/Qwen-Image-Edit` | -| Qwen Image Edit 2511 | `Qwen/Qwen-Image-Edit-2511` | +| Model Name | HuggingFace Model ID | +|:--------------------------|:---------------------------------------------------------| +| FLUX.1-dev | `black-forest-labs/FLUX.1-dev` | +| FLUX.2-dev | `black-forest-labs/FLUX.2-dev` | +| FLUX.2-dev-NVFP4 | `black-forest-labs/FLUX.2-dev-NVFP4` | +| FLUX.2-Klein-4B | `black-forest-labs/FLUX.2-klein-4B` | +| FLUX.2-Klein-9B | `black-forest-labs/FLUX.2-klein-9B` | +| Z-Image | `Tongyi-MAI/Z-Image` | +| Z-Image-Turbo | `Tongyi-MAI/Z-Image-Turbo` | +| GLM-Image | `zai-org/GLM-Image` | +| Qwen Image | `Qwen/Qwen-Image` | +| Qwen Image 2512 | `Qwen/Qwen-Image-2512` | +| Qwen Image Edit | `Qwen/Qwen-Image-Edit` | +| Qwen Image Edit 2509 | `Qwen/Qwen-Image-Edit-2509` | +| Qwen Image Edit 2511 | `Qwen/Qwen-Image-Edit-2511` | +| Qwen Image Layered | `Qwen/Qwen-Image-Layered` | +| SD3 Medium | `stabilityai/stable-diffusion-3-medium-diffusers` | +| SD3.5 Medium | `stabilityai/stable-diffusion-3.5-medium-diffusers` | +| SD3.5 Large | `stabilityai/stable-diffusion-3.5-large-diffusers` | +| Hunyuan3D-2 | `tencent/Hunyuan3D-2` | +| SANA 1.5 1.6B | `Efficient-Large-Model/SANA1.5_1.6B_1024px_diffusers` | +| SANA 1.5 4.8B | `Efficient-Large-Model/SANA1.5_4.8B_1024px_diffusers` | +| SANA 1600M 1024px | `Efficient-Large-Model/Sana_1600M_1024px_diffusers` | +| SANA 600M 1024px | `Efficient-Large-Model/Sana_600M_1024px_diffusers` | +| SANA 1600M 512px | `Efficient-Large-Model/Sana_1600M_512px_diffusers` | +| SANA 600M 512px | `Efficient-Large-Model/Sana_600M_512px_diffusers` | +| FireRed-Image-Edit 1.0 | `FireRedTeam/FireRed-Image-Edit-1.0` | +| FireRed-Image-Edit 1.1 | `FireRedTeam/FireRed-Image-Edit-1.1` | +| ERNIE-Image | `baidu/ERNIE-Image` | +| ERNIE-Image-Turbo | `baidu/ERNIE-Image-Turbo` | ## Supported Components diff --git a/docs/diffusion/environment_variables.md b/docs/diffusion/environment_variables.md index 7f418ad57..745c84af2 100644 --- a/docs/diffusion/environment_variables.md +++ b/docs/diffusion/environment_variables.md @@ -1,11 +1,42 @@ # Environment Variables -## Apple MPS +## Runtime + +| Environment Variable | Default | Description | +|----------------------|---------|-------------| +| `SGLANG_DIFFUSION_TARGET_DEVICE` | `cuda` | Target device for inference (`cuda`, `rocm`, `xpu`, `npu`, `musa`, `mps`, `cpu`) | +| `SGLANG_DIFFUSION_ATTENTION_BACKEND` | not set | Override attention backend via env var (e.g. `fa`, `torch_sdpa`, `sage_attn`) | +| `SGLANG_DIFFUSION_ATTENTION_CONFIG` | not set | Path to attention backend configuration file (JSON/YAML) | +| `SGLANG_DIFFUSION_STAGE_LOGGING` | false | Enable per-stage timing logs | +| `SGLANG_DIFFUSION_SERVER_DEV_MODE` | false | Enable dev-only HTTP endpoints for debugging | +| `SGLANG_DIFFUSION_TORCH_PROFILER_DIR` | not set | Directory for torch profiler traces (absolute path). Enables profiling when set | +| `SGLANG_DIFFUSION_CACHE_ROOT` | `~/.cache/sgl_diffusion` | Root directory for cache files | +| `SGLANG_DIFFUSION_CONFIG_ROOT` | `~/.config/sgl_diffusion` | Root directory for configuration files | +| `SGLANG_DIFFUSION_LOGGING_LEVEL` | `INFO` | Default logging level | +| `SGLANG_DIFFUSION_WORKER_MULTIPROC_METHOD` | `fork` | Multiprocess context for workers (`fork` or `spawn`) | +| `SGLANG_USE_RUNAI_MODEL_STREAMER` | true | Use Run:AI model streamer for model loading | + +## Platform-Specific + +### Apple MPS | Environment Variable | Default | Description | |----------------------|---------|--------------------------------------------------------------| | `SGLANG_USE_MLX` | not set | Set to `1` to enable MLX fused Metal kernels for norm ops on MPS | +### ROCm (AMD GPUs) + +| Environment Variable | Default | Description | +|----------------------|---------|-------------| +| `SGLANG_USE_ROCM_VAE` | false | Use AITer GroupNorm in VAE for improved performance on ROCm | +| `SGLANG_USE_ROCM_CUDNN_BENCHMARK` | false | Enable MIOpen auto-tuning for VAE conv layers on ROCm | + +### Quantization + +| Environment Variable | Default | Description | +|----------------------|---------|-------------| +| `SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND` | not set | FlashInfer FP4 GEMM backend for generic NVFP4 fallback | + ## Caching Acceleration These variables configure caching acceleration for Diffusion Transformer (DiT) models. @@ -30,6 +61,20 @@ See [cache-dit documentation](performance/cache/cache_dit.md) for detailed confi | `SGLANG_CACHE_DIT_SCM_COMPUTE_BINS` | not set | Custom SCM compute bins | | `SGLANG_CACHE_DIT_SCM_CACHE_BINS` | not set | Custom SCM cache bins | +### Cache-DiT Secondary Transformer + +For dual-transformer models (e.g., Wan2.2 with high/low-noise experts), these variables configure caching for the secondary transformer. Each falls back to its primary counterpart if not set. + +| Environment Variable | Default | Description | +|-------------------------------------|---------|------------------------------------------| +| `SGLANG_CACHE_DIT_SECONDARY_FN` | (from primary) | First N blocks to always compute | +| `SGLANG_CACHE_DIT_SECONDARY_BN` | (from primary) | Last N blocks to always compute | +| `SGLANG_CACHE_DIT_SECONDARY_WARMUP` | (from primary) | Warmup steps before caching | +| `SGLANG_CACHE_DIT_SECONDARY_RDT` | (from primary) | Residual difference threshold | +| `SGLANG_CACHE_DIT_SECONDARY_MC` | (from primary) | Max continuous cached steps | +| `SGLANG_CACHE_DIT_SECONDARY_TAYLORSEER` | (from primary) | Enable TaylorSeer calibrator | +| `SGLANG_CACHE_DIT_SECONDARY_TS_ORDER` | (from primary) | TaylorSeer order (1 or 2) | + ## Cloud Storage These variables configure S3-compatible cloud storage for automatically uploading generated images and videos. diff --git a/docs/diffusion/index.md b/docs/diffusion/index.md index 5c467522c..e0790d9e7 100644 --- a/docs/diffusion/index.md +++ b/docs/diffusion/index.md @@ -7,7 +7,7 @@ SGLang Diffusion is a high-performance inference framework for image and video g - Broad model support across Wan, Hunyuan, Qwen-Image, FLUX, Z-Image, GLM-Image, and more - Fast inference with `sgl-kernel`, JIT kernels, scheduler improvements, and caching acceleration - Multiple interfaces: `sglang generate`, `sglang serve`, and an OpenAI-compatible API -- Multi-platform support for NVIDIA, AMD, Ascend, Apple Silicon, and Moore Threads +- Multi-platform support for NVIDIA, AMD, Intel XPU, Ascend, Apple Silicon, and Moore Threads ## Quick Start diff --git a/docs/diffusion/installation.md b/docs/diffusion/installation.md index 8c0fa1d11..46fbab063 100644 --- a/docs/diffusion/installation.md +++ b/docs/diffusion/installation.md @@ -82,6 +82,14 @@ rm -f python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.t pip install -e "python[all_musa]" ``` +## Platform-Specific: Intel XPU + +For Intel Data Center GPU Max or Arc GPUs, follow the [XPU installation guide](../platforms/xpu.md) to set up the base environment, then install diffusion dependencies: + +```bash +pip install -e "python[diffusion]" +``` + ## Platform-Specific: Ascend NPU For Ascend NPU, please follow the [NPU installation guide](../platforms/ascend/ascend_npu.md). diff --git a/docs/diffusion/performance/attention_backends.md b/docs/diffusion/performance/attention_backends.md index 55d6da270..0c93a130b 100644 --- a/docs/diffusion/performance/attention_backends.md +++ b/docs/diffusion/performance/attention_backends.md @@ -13,6 +13,8 @@ When using the diffusers backend, `--attention-backend` is passed through to dif - **CUDA**: prefers FlashAttention (FA3/FA4) when supported; otherwise falls back to PyTorch SDPA. - **ROCm**: uses FlashAttention when available; otherwise falls back to PyTorch SDPA. +- **Intel XPU**: uses XPU Flash Attention backend (fp16/bf16, head sizes 64/96/128/192/256); otherwise falls back to PyTorch SDPA. +- **MUSA**: uses FlashAttention when available; otherwise falls back to PyTorch SDPA. - **MPS**: always uses PyTorch SDPA. - **NPU**: for ring attention uses FA otherwise uses PyTorch SDPA. @@ -31,6 +33,8 @@ For SGLang-native pipelines, the CLI accepts the lowercase names of `AttentionBa | `vmoba_attn` | `VMOBA_ATTN` | Requires `kernel.attn.vmoba_attn.vmoba`. Configure via `--attention-backend-config`. | | `aiter` | `AITER` | Requires `aiter`. | | `aiter_sage` | `AITER_SAGE` | Requires `aiter`. | +| `sla_attn` | `SLA_ATTN` | Sparse Linear Attention. Requires `SpargeAttn`. Install with `pip install git+https://github.com/thu-ml/SpargeAttn.git --no-build-isolation`. | +| `sage_sla_attn` | `SAGE_SLA_ATTN` | SageAttention + Sparse Linear Attention. Requires `SpargeAttn` (same install as SLA). | | `sparse_video_gen_2_attn` | `SPARSE_VIDEO_GEN_2_ATTN` | Requires `svg`. See installation instructions at https://github.com/svg-project/Sparse-VideoGen. | ## Selection priority @@ -85,18 +89,20 @@ Some backends require additional configuration. You can pass these parameters vi ## Platform support matrix -| Backend | CUDA | ROCm | MPS | NPU | Notes | -|---|---:|---:|---:|---:|---| -| `fa` | ✅ | ✅ | ❌ | ✅ | CUDA requires SM80+ and fp16/bf16. FlashAttention is only used when the required runtime is installed; otherwise it falls back to `torch_sdpa`. No extra installations are required for NPU | -| `torch_sdpa` | ✅ | ✅ | ✅ | ✅ | Most compatible option across platforms. | -| `sliding_tile_attn` | ✅ | ❌ | ❌ | ❌ | CUDA-only. Requires `st_attn`. Configure via `--attention-backend-config`. | -| `sage_attn` | ✅ | ❌ | ❌ | ❌ | CUDA-only (optional dependency). | -| `sage_attn_3` | ✅ | ❌ | ❌ | ❌ | CUDA-only (optional dependency). | -| `video_sparse_attn` | ✅ | ❌ | ❌ | ❌ | CUDA-only. Requires `vsa`. Configure `sparsity` via `--attention-backend-config`. | -| `vmoba_attn` | ✅ | ❌ | ❌ | ❌ | CUDA-only. Requires `kernel.attn.vmoba_attn.vmoba`. Configure via `--attention-backend-config`. | -| `aiter` | ❌ | ✅ | ❌ | ❌ | Requires `aiter`. | -| `aiter_sage` | ❌ | ✅ | ❌ | ❌ | Requires `aiter`. | -| `sparse_video_gen_2_attn` | ✅ | ❌ | ❌ | ❌ | CUDA-only. Requires `svg`. | +| Backend | CUDA | ROCm | XPU | MUSA | MPS | NPU | Notes | +|---|---:|---:|---:|---:|---:|---:|---| +| `fa` | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | CUDA requires SM80+ and fp16/bf16. XPU uses its own flash attention backend. FlashAttention is only used when the required runtime is installed; otherwise it falls back to `torch_sdpa`. No extra installations are required for NPU | +| `torch_sdpa` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | Most compatible option across platforms. | +| `sliding_tile_attn` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | CUDA-only. Requires `st_attn`. Configure via `--attention-backend-config`. | +| `sage_attn` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | CUDA-only (optional dependency). | +| `sage_attn_3` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | CUDA-only (optional dependency). | +| `video_sparse_attn` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | CUDA-only. Requires `vsa`. Configure `sparsity` via `--attention-backend-config`. | +| `sla_attn` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | CUDA-only. Requires `SpargeAttn`. | +| `sage_sla_attn` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | CUDA-only. Requires `SpargeAttn`. | +| `vmoba_attn` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | CUDA-only. Requires `kernel.attn.vmoba_attn.vmoba`. Configure via `--attention-backend-config`. | +| `aiter` | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | Requires `aiter`. | +| `aiter_sage` | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | Requires `aiter`. | +| `sparse_video_gen_2_attn` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | CUDA-only. Requires `svg`. | ## Usage