sglang/python/sglang/multimodal_gen

SGLang diffusion is an inference framework for accelerated image/video generation.

SGLang diffusion features an end-to-end unified pipeline for accelerating diffusion models. It is designed to be modular and extensible, allowing users to easily add new models and optimizations.

Key Features

SGLang Diffusion has the following features:

• Broad model support: Wan, FastWan, FLUX, Qwen-Image / Qwen-Image 2.1, LongCat-Image, Z-Image, Ideogram 4, Krea-2, Cosmos3, LTX-2/LTX-2.3/LTX-2.5, MiniMax-H3, FastH3, VDN-H3, LingBot Video MoE, LingBot World, SANA-Video/SANA-WM, JoyEcho, MOVA, GLM-Image, ERNIE-Image, Hunyuan3D, and more
• Fast inference speed: empowered by optimized sgl-kernel kernels, scheduler/runtime improvements, caching acceleration, and native diffusion hot-path optimizations
• Ease of use: OpenAI-compatible api, CLI, and python sdk support
• Multi-platform support:
  • NVIDIA GPUs (H100, H200, A100, B200, 4090, 5090)
  • AMD GPUs (MI300X, MI325X, MI355X)
  • Intel XPUs
  • Ascend NPU (A2, A3)
  • Apple Silicon (M-series via MPS)
  • Moore Threads GPUs (MTT S5000)

AMD/ROCm Support

SGLang Diffusion supports AMD Instinct GPUs through ROCm. On AMD platforms, we use the Triton attention backend and leverage AITER kernels for optimized layernorm and other operations. See the installation guide for setup instructions.

Moore Threads/MUSA Support

SGLang Diffusion supports Moore Threads GPUs (MTGPU) through the MUSA software stack. On MUSA platforms, we use FlashAttention (FA3) when available; also supports Sage Attention when installed; otherwise falls back to the Torch SDPA backend. See the installation guide for setup instructions.

Apple MPS Support

SGLang Diffusion supports Apple Silicon (M-series) via the MPS backend. Since Triton is Linux-only, Triton kernels are replaced with PyTorch-native fallbacks on MPS. See the installation guide for setup instructions.

Getting Started

uv pip install 'sglang[diffusion]' --prerelease=allow

For more installation methods (e.g. pypi, uv, docker, ROCm/AMD, MUSA/Moore Threads), check the installation guide.

Inference

Here's a minimal example to generate a video using the default settings:

from sglang.multimodal_gen import DiffGenerator

def main():
    # Create a diff generator from a pre-trained model
    generator = DiffGenerator.from_pretrained(
        model_path="Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
        num_gpus=1,  # Adjust based on your hardware
    )

    # Generate the video
    video = generator.generate(
        sampling_params_kwargs=dict(
            prompt="A curious raccoon peers through a vibrant field of yellow sunflowers, its eyes wide with interest.",
            return_frames=True,  # Also return frames from this call (defaults to False)
            output_path="my_videos/",  # Controls where videos are saved
            save_output=True
        )
    )

if __name__ == '__main__':
    main()

Or, more simply, with the CLI:

sglang generate --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
    --component-residency text_encoder=component-offload --pin-cpu-memory \
    --prompt "A curious raccoon" \
    --save-output

Qwen-Image 2.1

The native QwenImage21Pipeline supports text-to-image and reference-image conditioning with Qwen3-VL, a single-stream block-causal DiT, and the 64-channel VAE. Use an authorized checkpoint directory:

sglang generate --model-path /models/qwen-image-2.1 --model-id Qwen-Image-2.1 \
  --prompt "A capybara reading a book by candlelight" \
  --height 1024 --width 1024 --num-inference-steps 40 --guidance-scale 1 \
  --seed 0 --save-output

Add --image-path /path/to/input.png for editing. Dimensions must be multiples of 32. Full-checkpoint generation and editing have been tested on H200; see the model cookbook for component requirements and optimization boundaries. Compatible text-to-image requests support opt-in dynamic batching with --batching-max-size 2 when serving. Image edits are not merged across requests; use n for multiple outputs within an edit request. Batching can improve offload throughput, but changes floating-point rounding and is not always faster with resident weights.

Component residency

Use --component-residency COMPONENT=MODE to choose one runtime mode for each loaded component:

• resident keeps the complete component on the accelerator.
• component-offload stores the complete component on CPU between uses.
• snapshot-offload keeps a CPU weight snapshot while the complete component runs on the accelerator, avoiding a weight copy back to CPU after each use.
• layerwise-offload streams the component's declared layers from CPU.

COMPONENT can be an exact model_index.json key or one of all, dit, text_encoder, image_encoder, and vae. Exact keys override groups, and groups override all. Existing options such as --dit-cpu-offload, --text-encoder-cpu-offload, --image-encoder-cpu-offload, --vae-cpu-offload, and --cpu-offload-components remain supported. See the CLI reference for precedence and compatibility details.

LoRA support

Apply LoRA adapters via --lora-path:

sglang generate \
  --model-path Qwen/Qwen-Image-Edit-2511 \
  --lora-path prithivMLmods/Qwen-Image-Edit-2511-Anime \
  --prompt "Transform into anime." \
  --image-path "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png" \
  --save-output

For more usage examples (e.g. OpenAI compatible API, server mode), check the CLI reference.

Contributing

All contributions are welcome. The contribution guide is available here.

Acknowledgement

We learnt and reused code from the following projects:

• FastVideo. The major components of this repo are based on a fork of FastVideo on Sept. 24, 2025.
• xDiT. We used the parallelism library from it.
• diffusers We used the pipeline design from it.