From 6425df5c8a25812efb1574667ec2d821859f1caa Mon Sep 17 00:00:00 2001 From: Mick Date: Wed, 25 Mar 2026 16:01:32 +0800 Subject: [PATCH] [diffusion] doc: consolidate documentation (#21373) --- docs/diffusion/api/cli.md | 335 ++++++------------ docs/diffusion/ci_perf.md | 2 + docs/diffusion/contributing.md | 12 + docs/diffusion/development.md | 5 + docs/diffusion/environment_variables.md | 2 + docs/diffusion/index.md | 96 ++--- docs/diffusion/installation.md | 2 +- docs/diffusion/performance/cache/cache_dit.md | 2 +- docs/diffusion/performance/cache/index.md | 13 +- docs/diffusion/performance/cache/teacache.md | 2 +- docs/diffusion/performance/index.md | 68 +--- docs/diffusion/quantization.md | 4 +- docs/diffusion/reference.md | 11 + docs/diffusion/usage.md | 17 + docs/index.rst | 9 +- 15 files changed, 204 insertions(+), 376 deletions(-) create mode 100644 docs/diffusion/development.md create mode 100644 docs/diffusion/reference.md create mode 100644 docs/diffusion/usage.md diff --git a/docs/diffusion/api/cli.md b/docs/diffusion/api/cli.md index a66711484..5ab0f00ce 100644 --- a/docs/diffusion/api/cli.md +++ b/docs/diffusion/api/cli.md @@ -1,121 +1,81 @@ -# SGLang diffusion CLI Inference +# SGLang Diffusion CLI -The SGLang-diffusion CLI provides a quick way to access the inference pipeline for image and video generation. +Use the CLI for one-off generation with `sglang generate` or to start a persistent HTTP server with `sglang serve`. -## Prerequisites +## Quick Start -- A working SGLang diffusion installation and the `sglang` CLI available in `$PATH`. +### Generate +```bash +sglang generate \ + --model-path Qwen/Qwen-Image \ + --prompt "A beautiful sunset over the mountains" \ + --save-output +``` -## Supported Arguments +### Serve +```bash +sglang serve \ + --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ + --num-gpus 4 \ + --ulysses-degree 2 \ + --ring-degree 2 \ + --port 30010 +``` -### Server Arguments +For request and response examples, see [OpenAI-Compatible API](openai_api.md). -- `--model-path {MODEL_PATH}`: Path to the model or model ID -- `--lora-path {LORA_PATH}`: Path to a LoRA adapter (local path or HuggingFace model ID). If not specified, LoRA will not be applied. -- `--lora-nickname {NAME}`: Nickname for the LoRA adapter. (default: `default`). -- `--num-gpus {NUM_GPUS}`: Number of GPUs to use -- `--tp-size {TP_SIZE}`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster) -- `--sp-degree {SP_SIZE}`: Sequence parallelism size (typically should match the number of GPUs) -- `--ulysses-degree {ULYSSES_DEGREE}`: The degree of DeepSpeed-Ulysses-style SP in USP -- `--ring-degree {RING_DEGREE}`: The degree of ring attention-style SP in USP -- `--attention-backend {BACKEND}`: Attention backend to use. For SGLang-native pipelines use `fa`, `torch_sdpa`, `sage_attn`, etc. For diffusers pipelines use diffusers backend names like `flash`, `_flash_3_hub`, `sage`, `xformers`. -- `--attention-backend-config {CONFIG}`: Configuration for the attention backend. Can be a JSON string (e.g., '{"k": "v"}'), a path to a JSON/YAML file, or key=value pairs (e.g., "k=v,k2=v2"). -- `--cache-dit-config {PATH}`: Path to a Cache-DiT YAML/JSON config (diffusers backend only) -- `--dit-precision {DTYPE}`: Precision for the DiT model (currently supports fp32, fp16, and bf16). +```{tip} +Use `sglang generate --help` and `sglang serve --help` for the full argument list. The CLI help output is the source of truth for exhaustive flags. +``` -### Quantized Transformers +## Common Options + +### Model and runtime + +- `--model-path {MODEL}`: model path or Hugging Face model ID +- `--lora-path {PATH}` and `--lora-nickname {NAME}`: load a LoRA adapter +- `--num-gpus {N}`: number of GPUs to use +- `--tp-size {N}`: tensor parallelism size, mainly for encoders +- `--sp-degree {N}`: sequence parallelism size +- `--ulysses-degree {N}` and `--ring-degree {N}`: USP parallelism controls +- `--attention-backend {BACKEND}`: attention backend for native SGLang pipelines +- `--attention-backend-config {CONFIG}`: attention backend configuration + +### Sampling and output + +- `--prompt {PROMPT}` and `--negative-prompt {PROMPT}` +- `--num-inference-steps {STEPS}` and `--seed {SEED}` +- `--height {HEIGHT}`, `--width {WIDTH}`, `--num-frames {N}`, `--fps {FPS}` +- `--output-path {PATH}`, `--output-file-name {NAME}`, `--save-output`, `--return-frames` + +For frame interpolation and upscaling, see [Post-Processing](post_processing.md). + +### Quantized transformers For quantized transformer checkpoints, prefer: -- `--model-path` for the base model (the pipeline) +- `--model-path` for the base pipeline - `--transformer-path` for a quantized `transformers` transformer component folder - `--transformer-weights-path` for a quantized safetensors file, directory, or repo -See [Quantization](../quantization.md) for the supported quantization families and examples. +See [Quantization](../quantization.md) for supported quantization families and examples. +## Configuration Files -### Sampling Parameters - -- `--prompt {PROMPT}`: Text description for the video you want to generate -- `--num-inference-steps {STEPS}`: Number of denoising steps -- `--negative-prompt {PROMPT}`: Negative prompt to guide generation away from certain concepts -- `--seed {SEED}`: Random seed for reproducible generation - - -**Image/Video Configuration** - -- `--height {HEIGHT}`: Height of the generated output -- `--width {WIDTH}`: Width of the generated output -- `--num-frames {NUM_FRAMES}`: Number of frames to generate -- `--fps {FPS}`: Frames per second for the saved output, if this is a video-generation task - - -**Post-Processing** (frame interpolation & upscaling) - -SGLang diffusion supports optional post-processing steps — frame interpolation -(RIFE) for smoother video and upscaling (Real-ESRGAN) for higher resolution. -See the dedicated **[Post-Processing](post_processing.md)** page for full -details, supported models, and examples. - -**Output Options** - -- `--output-path {PATH}`: Directory to save the generated video -- `--save-output`: Whether to save the image/video to disk -- `--return-frames`: Whether to return the raw frames - -### Using Configuration Files - -Instead of specifying all parameters on the command line, you can use a configuration file: +Use `--config` to load JSON or YAML configuration. Command-line flags override values from the config file. ```bash -sglang generate --config {CONFIG_FILE_PATH} +sglang generate --config config.yaml ``` -The configuration file should be in JSON or YAML format with the same parameter names as the CLI options. Command-line arguments take precedence over settings in the configuration file, allowing you to override specific values while keeping the rest from the configuration file. - -Example configuration file (config.json): - -```json -{ - "model_path": "FastVideo/FastHunyuan-diffusers", - "prompt": "A beautiful woman in a red dress walking down a street", - "output_path": "outputs/", - "num_gpus": 2, - "sp_size": 2, - "tp_size": 1, - "num_frames": 45, - "height": 720, - "width": 1280, - "num_inference_steps": 6, - "seed": 1024, - "fps": 24, - "precision": "bf16", - "vae_precision": "fp16", - "vae_tiling": true, - "vae_sp": true, - "vae_config": { - "load_encoder": false, - "load_decoder": true, - "tile_sample_min_height": 256, - "tile_sample_min_width": 256 - }, - "text_encoder_precisions": [ - "fp16", - "fp16" - ], - "mask_strategy_file_path": null, - "enable_torch_compile": false -} -``` - -Or using YAML format (config.yaml): +Example: ```yaml -model_path: "FastVideo/FastHunyuan-diffusers" -prompt: "A beautiful woman in a red dress walking down a street" -output_path: "outputs/" +model_path: FastVideo/FastHunyuan-diffusers +prompt: A beautiful woman in a red dress walking down a street +output_path: outputs/ num_gpus: 2 sp_size: 2 tp_size: 1 @@ -125,163 +85,95 @@ width: 1280 num_inference_steps: 6 seed: 1024 fps: 24 -precision: "bf16" -vae_precision: "fp16" +precision: bf16 +vae_precision: fp16 vae_tiling: true vae_sp: true -vae_config: - load_encoder: false - load_decoder: true - tile_sample_min_height: 256 - tile_sample_min_width: 256 -text_encoder_precisions: - - "fp16" - - "fp16" -mask_strategy_file_path: null enable_torch_compile: false ``` +## Generate -To see all the options, you can use the `--help` flag: +`sglang generate` runs a single generation job and exits when the job finishes. ```bash -sglang generate --help +sglang generate \ + --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ + --text-encoder-cpu-offload \ + --pin-cpu-memory \ + --num-gpus 4 \ + --ulysses-degree 2 \ + --ring-degree 2 \ + --prompt "A curious raccoon" \ + --save-output \ + --output-path outputs \ + --output-file-name "a-curious-raccoon.mp4" ``` +```{note} +HTTP server-only arguments are ignored by `sglang generate`. +``` + +For diffusers pipelines, Cache-DiT can be enabled with `SGLANG_CACHE_DIT_ENABLED=true` or `--cache-dit-config`. See [Cache-DiT](../performance/cache/cache_dit.md). + ## Serve -Launch the SGLang diffusion HTTP server and interact with it using the OpenAI SDK and curl. - -### Start the server - -Use the following command to launch the server: +`sglang serve` starts the HTTP server and keeps the model loaded for repeated requests. ```bash -SERVER_ARGS=( - --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers - --text-encoder-cpu-offload - --pin-cpu-memory - --num-gpus 4 - --ulysses-degree=2 - --ring-degree=2 -) - -sglang serve "${SERVER_ARGS[@]}" +sglang serve \ + --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ + --text-encoder-cpu-offload \ + --pin-cpu-memory \ + --num-gpus 4 \ + --ulysses-degree 2 \ + --ring-degree 2 \ + --port 30010 ``` -- **--model-path**: Which model to load. The example uses `Wan-AI/Wan2.1-T2V-1.3B-Diffusers`. -- **--port**: HTTP port to listen on (the default here is `30010`). +### Cloud Storage -For detailed API usage, including Image, Video Generation and LoRA management, please refer to the [OpenAI API Documentation](openai_api.md). - -### Cloud Storage Support - -SGLang diffusion supports automatically uploading generated images and videos to S3-compatible cloud storage (e.g., AWS S3, MinIO, Alibaba Cloud OSS, Tencent Cloud COS). - -When enabled, the server follows a **Generate -> Upload -> Delete** workflow: -1. The artifact is generated to a temporary local file. -2. The file is immediately uploaded to the configured S3 bucket in a background thread. -3. Upon successful upload, the local file is deleted. -4. The API response returns the public URL of the uploaded object. - -**Configuration** - -Cloud storage is enabled via environment variables. Note that `boto3` must be installed separately (`pip install boto3`) to use this feature. +SGLang Diffusion can upload generated images and videos to S3-compatible object storage after generation. ```bash -# Enable S3 storage export SGLANG_CLOUD_STORAGE_TYPE=s3 export SGLANG_S3_BUCKET_NAME=my-bucket export SGLANG_S3_ACCESS_KEY_ID=your-access-key export SGLANG_S3_SECRET_ACCESS_KEY=your-secret-key - -# Optional: Custom endpoint for MinIO/OSS/COS export SGLANG_S3_ENDPOINT_URL=https://minio.example.com ``` -See [Environment Variables Documentation](../environment_variables.md) for more details. - -## Generate - -Run a one-off generation task without launching a persistent server. - -To use it, pass both server arguments and sampling parameters in one command, after the `generate` subcommand, for example: - -```bash -SERVER_ARGS=( - --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers - --text-encoder-cpu-offload - --pin-cpu-memory - --num-gpus 4 - --ulysses-degree=2 - --ring-degree=2 -) - -SAMPLING_ARGS=( - --prompt "A curious raccoon" - --save-output - --output-path outputs - --output-file-name "A curious raccoon.mp4" -) - -sglang generate "${SERVER_ARGS[@]}" "${SAMPLING_ARGS[@]}" - -# Or, users can set `SGLANG_CACHE_DIT_ENABLED` env as `true` to enable cache acceleration -SGLANG_CACHE_DIT_ENABLED=true sglang generate "${SERVER_ARGS[@]}" "${SAMPLING_ARGS[@]}" -``` - -Once the generation task has finished, the server will shut down automatically. - -> [!NOTE] -> The HTTP server-related arguments are ignored in this subcommand. +See [Environment Variables](../environment_variables.md) for the full set of storage options. ## Component Path Overrides -SGLang diffusion allows you to override any pipeline component (e.g., `vae`, `transformer`, `text_encoder`) by specifying a custom checkpoint path. This is useful for: - -### Example: FLUX.2-dev with Tiny AutoEncoder - -You can override **any** component by using `---path`, where `` matches the key in the model's `model_index.json`: - -For example, replace the default VAE with a distilled tiny autoencoder for ~3x faster decoding: +Override individual pipeline components such as `vae`, `transformer`, or `text_encoder` with `---path`. ```bash sglang serve \ - --model-path=black-forest-labs/FLUX.2-dev \ - # with a Huggingface Repo ID - --vae-path=fal/FLUX.2-Tiny-AutoEncoder - # or use a local path - --vae-path=~/.cache/huggingface/hub/models--fal--FLUX.2-Tiny-AutoEncoder/snapshots/.../vae + --model-path black-forest-labs/FLUX.2-dev \ + --vae-path fal/FLUX.2-Tiny-AutoEncoder ``` -**Important:** -- The component key must match the one in your model's `model_index.json` (e.g., `vae`). -- The path must: - - either be a Huggingface Repo ID (e.g., fal/FLUX.2-Tiny-AutoEncoder) - - or point to a **complete component folder**, containing `config.json` and safetensors files - +The component key must match the key in the model's `model_index.json`, and the path must be either a Hugging Face repo ID or a complete component directory. ## Diffusers Backend -SGLang diffusion supports a **diffusers backend** that allows you to run any diffusers-compatible model through SGLang's infrastructure using vanilla diffusers pipelines. This is useful for running models without native SGLang implementations or models with custom pipeline classes. +Use `--backend diffusers` to force vanilla diffusers pipelines when no native SGLang implementation exists or when a model requires a custom pipeline class. -### Arguments +### Key Options | Argument | Values | Description | |----------|--------|-------------| -| `--backend` | `auto` (default), `sglang`, `diffusers` | `auto`: prefer native SGLang, fallback to diffusers. `sglang`: force native (fails if unavailable). `diffusers`: force vanilla diffusers pipeline. | -| `--diffusers-attention-backend` | `flash`, `_flash_3_hub`, `sage`, `xformers`, `native` | Attention backend for diffusers pipelines. See [diffusers attention backends](https://huggingface.co/docs/diffusers/main/en/optimization/attention_backends). | -| `--trust-remote-code` | flag | Required for models with custom pipeline classes (e.g., Ovis). | -| `--vae-tiling` | flag | Enable VAE tiling for large image support (decodes tile-by-tile). | -| `--vae-slicing` | flag | Enable VAE slicing for lower memory usage (decodes slice-by-slice). | -| `--dit-precision` | `fp16`, `bf16`, `fp32` | Precision for the diffusion transformer. | -| `--vae-precision` | `fp16`, `bf16`, `fp32` | Precision for the VAE. | -| `--enable-torch-compile` | flag | Enable `torch.compile` for diffusers pipelines. | -| `--cache-dit-config` | `{PATH}` | Path to a Cache-DiT YAML/JSON config file for accelerating diffusers pipelines with Cache-DiT. | +| `--backend` | `auto`, `sglang`, `diffusers` | Choose native SGLang, force native, or force diffusers | +| `--diffusers-attention-backend` | `flash`, `_flash_3_hub`, `sage`, `xformers`, `native` | Attention backend for diffusers pipelines | +| `--trust-remote-code` | flag | Required for models with custom pipeline classes | +| `--vae-tiling` and `--vae-slicing` | flag | Lower memory usage for VAE decode | +| `--dit-precision` and `--vae-precision` | `fp16`, `bf16`, `fp32` | Precision controls | +| `--enable-torch-compile` | flag | Enable `torch.compile` | +| `--cache-dit-config` | `{PATH}` | Cache-DiT config for diffusers pipelines | -### Example: Running Ovis-Image-7B - -[Ovis-Image-7B](https://huggingface.co/AIDC-AI/Ovis-Image-7B) is a 7B text-to-image model optimized for high-quality text rendering. +### Example ```bash sglang generate \ @@ -298,25 +190,4 @@ sglang generate \ --output-file-name ovis_garden.png ``` -### Extra Diffusers Arguments - -For pipeline-specific parameters not exposed via CLI, use `diffusers_kwargs` in a config file: - -```json -{ - "model_path": "AIDC-AI/Ovis-Image-7B", - "backend": "diffusers", - "prompt": "A beautiful landscape", - "diffusers_kwargs": { - "cross_attention_kwargs": {"scale": 0.5} - } -} -``` - -```bash -sglang generate --config config.json -``` - -### Cache-DiT Acceleration - -Users who use the diffusers backend can also leverage Cache-DiT acceleration and load custom cache configs from a YAML file to boost performance of diffusers pipelines. See the [Cache-DiT Acceleration](https://docs.sglang.io/diffusion/performance/cache/cache_dit.html) documentation for details. +For pipeline-specific arguments not exposed in the CLI, pass `diffusers_kwargs` in a config file. diff --git a/docs/diffusion/ci_perf.md b/docs/diffusion/ci_perf.md index 088c5be56..f8bb2316b 100644 --- a/docs/diffusion/ci_perf.md +++ b/docs/diffusion/ci_perf.md @@ -1,3 +1,5 @@ +# CI Performance + ## Perf Baseline Generation Script `python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py` starts a local diffusion server, issues requests for selected test cases, aggregates stage/denoise-step/E2E timings from the perf log, and writes the results back to the `scenarios` section of `perf_baselines.json`. diff --git a/docs/diffusion/contributing.md b/docs/diffusion/contributing.md index 7de656100..9b960aec9 100644 --- a/docs/diffusion/contributing.md +++ b/docs/diffusion/contributing.md @@ -2,6 +2,18 @@ This guide outlines the requirements for contributing to the SGLang Diffusion module (`sglang.multimodal_gen`). +## Contributor Guides + +- [Support New Models](support_new_models.md): implementation guide for adding new diffusion pipelines +- [CI Performance](ci_perf.md): update and regenerate perf baselines + +```{toctree} +:maxdepth: 1 + +support_new_models +ci_perf +``` + ## On AI-Assisted ("Vibe Coding") PRs Vibe-coded PRs are welcome — we judge code quality, not how it was produced. The bar is the same for all PRs: diff --git a/docs/diffusion/development.md b/docs/diffusion/development.md new file mode 100644 index 000000000..afed2fb8d --- /dev/null +++ b/docs/diffusion/development.md @@ -0,0 +1,5 @@ +# Development + +This page collects lower-level development material for SGLang Diffusion. + +- [Contributing](contributing.md): contribution workflow, adding new models, and CI perf baselines diff --git a/docs/diffusion/environment_variables.md b/docs/diffusion/environment_variables.md index c66bf79f6..7f418ad57 100644 --- a/docs/diffusion/environment_variables.md +++ b/docs/diffusion/environment_variables.md @@ -1,3 +1,5 @@ +# Environment Variables + ## Apple MPS | Environment Variable | Default | Description | diff --git a/docs/diffusion/index.md b/docs/diffusion/index.md index ca80e8b25..db384c090 100644 --- a/docs/diffusion/index.md +++ b/docs/diffusion/index.md @@ -1,100 +1,48 @@ # SGLang Diffusion -SGLang Diffusion is an inference framework for accelerated image and video generation using diffusion models. It provides an end-to-end unified pipeline with optimized kernels and an efficient scheduler loop. +SGLang Diffusion is a high-performance inference framework for image and video generation. It provides native SGLang pipelines, diffusers backend support, an OpenAI-compatible server, and an optimized kernel stack built on both precompiled `sgl-kernel` operators and JIT kernels for key inference paths. ## Key Features -- **Broad Model Support**: Wan series, FastWan series, Hunyuan, Qwen-Image, Qwen-Image-Edit, Flux, Z-Image, GLM-Image, and more -- **Fast Inference**: Optimized kernels, efficient scheduler loop, and Cache-DiT acceleration -- **Ease of Use**: OpenAI-compatible API, CLI, and Python SDK -- **Multi-Platform**: - - NVIDIA GPUs (H100, H200, A100, B200, 4090) - - AMD GPUs (MI300X, MI325X) - - Ascend NPU (A2, A3) - - Apple Silicon (M-series via MPS) - - Moore Threads GPUs (MTT S5000) - ---- +- Broad model support across Wan, Hunyuan, Qwen-Image, FLUX, Z-Image, GLM-Image, and more +- Fast inference with `sgl-kernel`, JIT kernels, scheduler improvements, and caching acceleration +- Multiple interfaces: `sglang generate`, `sglang serve`, and an OpenAI-compatible API +- Multi-platform support for NVIDIA, AMD, Ascend, Apple Silicon, and Moore Threads ## Quick Start -### Installation - ```bash uv pip install "sglang[diffusion]" --prerelease=allow ``` -See [Installation Guide](installation.md) for more installation methods and ROCm-specific instructions. - -### Basic Usage - -Generate an image with the CLI: - ```bash sglang generate --model-path Qwen/Qwen-Image \ - --prompt "A beautiful sunset over the mountains" \ - --save-output + --prompt "A beautiful sunset over the mountains" \ + --save-output ``` -Or start a server with the OpenAI-compatible API: - ```bash sglang serve --model-path Qwen/Qwen-Image --port 30010 ``` ---- +## Start Here -## Documentation +- [Installation](installation.md): install SGLang Diffusion and platform dependencies +- [Compatibility Matrix](compatibility_matrix.md): check model and optimization support +- [CLI](api/cli.md): run one-off generation jobs or launch a persistent server +- [OpenAI-Compatible API](api/openai_api.md): send image and video requests to the HTTP server +- [Attention Backends](performance/attention_backends.md): choose the best backend for your model and hardware +- [Caching Acceleration](performance/cache/index.md): use Cache-DiT or TeaCache to reduce denoising cost +- [Quantization](quantization.md): load quantized transformer checkpoints +- [Contributing](contributing.md): contribution workflow, adding new models, and CI perf baselines -### Getting Started +## Additional Documentation -- **[Installation](installation.md)** - Install SGLang Diffusion via pip, uv, Docker, or from source -- **[Compatibility Matrix](compatibility_matrix.md)** - Supported models and optimization compatibility - -### Usage - -- **[CLI Documentation](api/cli.md)** - Command-line interface for `sglang generate` and `sglang serve` -- **[Quantization](quantization.md)** - Quantized transformer checkpoint usage and supported quantization families -- **[OpenAI API](api/openai_api.md)** - OpenAI-compatible API for image/video generation and LoRA management -- **[Post-Processing](api/post_processing.md)** - Frame interpolation (RIFE) and upscaling (Real-ESRGAN) - -### Performance Optimization - -- **[Performance Overview](performance/index.md)** - Overview of all performance optimization strategies -- **[Attention Backends](performance/attention_backends.md)** - Available attention backends (FlashAttention, SageAttention, etc.) -- **[Caching Strategies](performance/cache/)** - Cache-DiT and TeaCache acceleration -- **[Profiling](performance/profiling.md)** - Profiling techniques with PyTorch Profiler and Nsight Systems - -### Reference - -- **[Environment Variables](environment_variables.md)** - Configuration via environment variables -- **[Support New Models](support_new_models.md)** - Guide for adding new diffusion models -- **[Contributing](contributing.md)** - Contribution guidelines and commit message conventions -- **[CI Performance](ci_perf.md)** - Performance baseline generation script - ---- - -## CLI Quick Reference - -### Generate (one-off generation) - -```bash -sglang generate --model-path --prompt "" --save-output -``` - -### Serve (HTTP server) - -```bash -sglang serve --model-path --port 30010 -``` - -### Enable Cache-DiT acceleration - -```bash -SGLANG_CACHE_DIT_ENABLED=true sglang generate --model-path --prompt "" -``` - ---- +- [Post-Processing](api/post_processing.md): frame interpolation and upscaling +- [Performance Overview](performance/index.md): overview of attention, caching, and profiling +- [Environment Variables](environment_variables.md): platform, caching, storage, and debugging configuration +- [Support New Models](support_new_models.md): implementation guide for new diffusion pipelines +- [CI Performance](ci_perf.md): performance baseline generation ## References diff --git a/docs/diffusion/installation.md b/docs/diffusion/installation.md index 2e4c23cc8..9531d70d2 100644 --- a/docs/diffusion/installation.md +++ b/docs/diffusion/installation.md @@ -1,6 +1,6 @@ # Install SGLang-Diffusion -You can install SGLang-Diffusion using one of the methods below. +You can install SGLang-Diffusion using one of the methods below. The standard installation already includes SGLang's optimized kernel stack, including both `sgl-kernel` and JIT kernels used by diffusion workloads. ## Standard Installation (NVIDIA GPUs) diff --git a/docs/diffusion/performance/cache/cache_dit.md b/docs/diffusion/performance/cache/cache_dit.md index 60d9ccbfa..9f804ce54 100644 --- a/docs/diffusion/performance/cache/cache_dit.md +++ b/docs/diffusion/performance/cache/cache_dit.md @@ -1,4 +1,4 @@ -# Cache-DiT Acceleration +# Cache-DiT SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to **1.69x inference speedup** with minimal quality loss. diff --git a/docs/diffusion/performance/cache/index.md b/docs/diffusion/performance/cache/index.md index d1f8e61d0..c7f8f53ef 100644 --- a/docs/diffusion/performance/cache/index.md +++ b/docs/diffusion/performance/cache/index.md @@ -1,6 +1,6 @@ -# Caching Acceleration for Diffusion Models +# Caching Acceleration -SGLang provides multiple caching acceleration strategies for Diffusion Transformer (DiT) models. These strategies can significantly reduce inference time by skipping redundant computation. +SGLang provides two complementary caching strategies for Diffusion Transformer (DiT) models. Both reduce denoising cost by skipping redundant computation, but they operate at different levels. ## Overview @@ -11,8 +11,6 @@ SGLang supports two complementary caching approaches: | **Cache-DiT** | Block-level | Skip individual transformer blocks dynamically | Advanced, higher speedup | | **TeaCache** | Timestep-level | Skip entire denoising steps based on L1 similarity | Simple, built-in | - - ## Cache-DiT [Cache-DiT](https://github.com/vipshop/cache-dit) provides block-level caching with @@ -54,6 +52,13 @@ See [teacache.md](teacache.md) for detailed documentation. For Flux and Qwen models, TeaCache is automatically disabled when CFG is enabled. +```{toctree} +:maxdepth: 1 + +cache_dit +teacache +``` + ## References - [Cache-DiT Repository](https://github.com/vipshop/cache-dit) diff --git a/docs/diffusion/performance/cache/teacache.md b/docs/diffusion/performance/cache/teacache.md index 7960437c7..dd9691c43 100644 --- a/docs/diffusion/performance/cache/teacache.md +++ b/docs/diffusion/performance/cache/teacache.md @@ -1,4 +1,4 @@ -# TeaCache Acceleration +# TeaCache > **Note**: This is one of two caching strategies available in SGLang. > For an overview of all caching options, see [caching](../index.md). diff --git a/docs/diffusion/performance/index.md b/docs/diffusion/performance/index.md index f61c4e93c..4a3c06440 100644 --- a/docs/diffusion/performance/index.md +++ b/docs/diffusion/performance/index.md @@ -1,71 +1,35 @@ -# Performance Optimization +# Performance -SGLang-Diffusion provides multiple performance optimization strategies to accelerate inference. This section covers all available performance tuning options. +This section covers the main performance levers for SGLang Diffusion: attention backends, caching acceleration, and profiling. ## Overview | Optimization | Type | Description | |--------------|------|-------------| | **Cache-DiT** | Caching | Block-level caching with DBCache, TaylorSeer, and SCM | -| **TeaCache** | Caching | Timestep-level caching using L1 similarity | +| **TeaCache** | Caching | Timestep-level caching based on temporal similarity | | **Attention Backends** | Kernel | Optimized attention implementations (FlashAttention, SageAttention, etc.) | | **Profiling** | Diagnostics | PyTorch Profiler and Nsight Systems guidance | -## Caching Strategies +## Start Here -SGLang supports two complementary caching approaches: +- Use [Attention Backends](attention_backends.md) to choose the best backend for your model and hardware. +- Use [Caching Acceleration](cache/index.md) to reduce denoising cost with Cache-DiT or TeaCache. +- Use [Profiling](profiling.md) when you need to diagnose a bottleneck rather than guess. -### Cache-DiT +## Caching at a Glance -[Cache-DiT](https://github.com/vipshop/cache-dit) provides block-level caching with advanced strategies. It can achieve up to **1.69x speedup**. +- [Cache-DiT](cache/cache_dit.md) is block-level caching for diffusers pipelines and higher speedup-oriented tuning. +- [TeaCache](cache/teacache.md) is timestep-level caching built into SGLang model families. -**Quick Start:** -```bash -SGLANG_CACHE_DIT_ENABLED=true \ -sglang generate --model-path Qwen/Qwen-Image \ - --prompt "A beautiful sunset over the mountains" +```{toctree} +:maxdepth: 1 + +attention_backends +cache/index +profiling ``` -**Key Features:** -- **DBCache**: Dynamic block-level caching based on residual differences -- **TaylorSeer**: Taylor expansion-based calibration for optimized caching -- **SCM**: Step-level computation masking for additional speedup - -See [Cache-DiT Documentation](cache/cache_dit.md) for detailed configuration. - -### TeaCache - -TeaCache (Temporal similarity-based caching) accelerates diffusion inference by detecting when consecutive denoising steps are similar enough to skip computation entirely. - -**Quick Overview:** -- Tracks L1 distance between modulated inputs across timesteps -- When accumulated distance is below threshold, reuses cached residual -- Supports CFG with separate positive/negative caches - -**Supported Models:** Wan (wan2.1, wan2.2), Hunyuan (HunyuanVideo), Z-Image - -See [TeaCache Documentation](cache/teacache.md) for detailed configuration. - -## Attention Backends - -Different attention backends offer varying performance characteristics depending on your hardware and model: - -- **FlashAttention**: Fastest on NVIDIA GPUs with fp16/bf16 -- **SageAttention**: Alternative optimized implementation -- **xformers**: Memory-efficient attention -- **SDPA**: PyTorch native scaled dot-product attention - -See [Attention Backends](attention_backends.md) for platform support and configuration options. - -## Profiling - -To diagnose performance bottlenecks, SGLang-Diffusion supports profiling tools: - -- **PyTorch Profiler**: Built-in Python profiling -- **Nsight Systems**: GPU kernel-level analysis - -See [Profiling Guide](profiling.md) for detailed instructions. - ## References - [Cache-DiT Repository](https://github.com/vipshop/cache-dit) diff --git a/docs/diffusion/quantization.md b/docs/diffusion/quantization.md index 871e78b40..ab1b634a8 100644 --- a/docs/diffusion/quantization.md +++ b/docs/diffusion/quantization.md @@ -141,8 +141,7 @@ Recommended auto-detected flow: sglang generate \ --model-path Qwen/Qwen-Image \ --transformer-weights-path /path/to/svdq-int4_r32-qwen-image.safetensors \ - --prompt "change the raccoon to a cute cat" \ - --attention-backend torch_sdpa \ + --prompt "a beautiful sunset" \ --save-output ``` @@ -156,7 +155,6 @@ sglang generate \ --quantization-precision int4 \ --quantization-rank 128 \ --prompt "a beautiful sunset" \ - --attention-backend torch_sdpa \ --save-output ``` diff --git a/docs/diffusion/reference.md b/docs/diffusion/reference.md new file mode 100644 index 000000000..2005a91c7 --- /dev/null +++ b/docs/diffusion/reference.md @@ -0,0 +1,11 @@ +# Reference + +Reference material for environment-based configuration and runtime behavior. + +- [Environment Variables](environment_variables.md): platform, caching, cloud storage, and debugging variables + +```{toctree} +:maxdepth: 1 + +environment_variables +``` diff --git a/docs/diffusion/usage.md b/docs/diffusion/usage.md new file mode 100644 index 000000000..78b0a545d --- /dev/null +++ b/docs/diffusion/usage.md @@ -0,0 +1,17 @@ +# Usage + +Use this section for day-to-day inference workflows with SGLang Diffusion. + +- [CLI](api/cli.md): run one-off jobs with `sglang generate` or start a server with `sglang serve` +- [OpenAI-Compatible API](api/openai_api.md): request format, endpoints, and SDK examples +- [Post-Processing](api/post_processing.md): frame interpolation and upscaling +- [Quantization](quantization.md): quantized transformer checkpoints and supported quantization families + +```{toctree} +:maxdepth: 1 + +api/cli +api/openai_api +api/post_processing +quantization +``` diff --git a/docs/index.rst b/docs/index.rst index 6be8c0d28..e61b40ef4 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -85,18 +85,11 @@ Its core features include: diffusion/installation diffusion/compatibility_matrix diffusion/api/cli - diffusion/quantization diffusion/api/openai_api - diffusion/performance/index diffusion/performance/attention_backends - diffusion/performance/profiling diffusion/performance/cache/index - diffusion/performance/cache/cache_dit - diffusion/performance/cache/teacache - diffusion/support_new_models + diffusion/quantization diffusion/contributing - diffusion/ci_perf - diffusion/environment_variables .. toctree:: :maxdepth: 1