diff --git a/docs_new/cookbook/diffusion/intro.mdx b/docs_new/cookbook/diffusion/intro.mdx index b96c2e8a8..59554072a 100644 --- a/docs_new/cookbook/diffusion/intro.mdx +++ b/docs_new/cookbook/diffusion/intro.mdx @@ -96,12 +96,6 @@ Realtime models keep a session alive and generate chunk by chunk with causal sta - @@ -112,3 +106,5 @@ Realtime models keep a session alive and generate chunk by chunk with causal sta img="/cards/logos/sana.png" /> + +Use the sidebar group for LingBot World family variants. The overview links the newer LingBot World 2.0 recipe directly. diff --git a/docs_new/docs/sglang-diffusion/api/cli.mdx b/docs_new/docs/sglang-diffusion/api/cli.mdx index 2f3d96e00..3d7ae504f 100644 --- a/docs_new/docs/sglang-diffusion/api/cli.mdx +++ b/docs_new/docs/sglang-diffusion/api/cli.mdx @@ -78,12 +78,16 @@ Use `sglang generate --help` and `sglang serve --help` for the full argument lis - `--lora-path {PATH}` and `--lora-nickname {NAME}`: load a LoRA adapter - `--lora-merge-mode {auto|merge|dynamic}`: choose how LoRA is applied. `auto` statically merges regular weights and uses dynamic LoRA for FSDP-sharded weights to avoid full-gather peaks. - `--num-gpus {N}`: number of GPUs to use -- `--performance-mode {manual|auto|speed|memory}` / `--mode`: preset for latency/throughput and memory defaults. `auto` is the default and keeps safe offload defaults, using FSDP only for validated DiT-offload replacement paths; use `manual` to keep performance-related server args under explicit user control. Explicit offload, FSDP, and parallelism flags take precedence in all modes. +- `--performance-mode {manual|auto|speed|memory}` / `--mode`: preset for latency/throughput and memory defaults. `auto` is the default and keeps safe offload defaults, using FSDP only for validated DiT-offload replacement paths; `speed` also enables `--enable-torch-compile` by default unless you explicitly disable it. Use `manual` to keep performance-related server args under explicit user control. Explicit offload, FSDP, and parallelism flags take precedence in all modes. - `--tp-size {N}`: tensor parallelism size, mainly for encoders - `--sp-degree {N}`: sequence parallelism size - `--ulysses-degree {N}` and `--ring-degree {N}`: USP parallelism controls - `--enable-cfg-parallel {true|false}`: enable or explicitly disable CFG parallelism - `--warmup-mode {off|request|server}`: control startup warmup for `sglang serve`; `off` skips warmup, `request` primes the request path, and `server` runs a full synthetic server warmup before serving traffic +- `--enable-torch-compile {true|false}`: compile native diffusion hot paths. When no warmup mode is configured, this also enables server warmup so first real requests do not pay compile latency. +- `--offload-during-compile {true|false}`: when compile warmup is active, temporarily layerwise-offload DiT weights and move resident non-DiT components off-device so `max-autotune` fits on tighter-memory GPUs; the configured serving residency is restored before real traffic. Skipped under existing layerwise offload, Cache-DiT, or FSDP. +- `--enable-breakable-cuda-graph {true|false}`: capture supported DiT forwards as breakable CUDA graph segments to reduce launch overhead. Requires `--warmup-resolutions` for every served resolution because each resolution is captured separately. +- `--bcg-text-buckets {N...}`: prompt-length padding buckets for breakable CUDA graph capture/replay reuse. - `--attention-backend {BACKEND}`: attention backend for native SGLang and diffusers pipelines - `--component-attention-backends {MAP}`: per-component attention backend overrides, for example `text_encoder=torch_sdpa,transformer=fa` - `--attention-backend-config {CONFIG}`: attention backend configuration @@ -174,6 +178,8 @@ HTTP server-only arguments are ignored by `sglang generate`. For diffusers pipelines, Cache-DiT can be enabled with `SGLANG_CACHE_DIT_ENABLED=true` or `--cache-dit-config`. See [Cache-DiT](../cache_dit). +For supported image pipelines, breakable CUDA graph can be enabled with `--enable-breakable-cuda-graph`, but you must declare every served resolution in `--warmup-resolutions` so warmup captures matching graph signatures. + ### Layerwise Offload Use layerwise offload when a large component does not fit comfortably in GPU memory. By default, `--dit-layerwise-offload` only applies to legacy DiT components. Use `--layerwise-offload-components` to select pipeline component names explicitly (`--layerwise-offload-modules` is accepted as an alias): diff --git a/docs_new/docs/sglang-diffusion/deployment_cookbook.mdx b/docs_new/docs/sglang-diffusion/deployment_cookbook.mdx index 454a22bc1..f77333cb8 100644 --- a/docs_new/docs/sglang-diffusion/deployment_cookbook.mdx +++ b/docs_new/docs/sglang-diffusion/deployment_cookbook.mdx @@ -76,7 +76,7 @@ Base the decision on available memory on the selected GPU(s). `speed` - Favors GPU-resident execution for lower latency and higher throughput. Disables CPU offload when unset; may OOM. + Favors GPU-resident execution for lower latency and higher throughput. Disables CPU offload when unset and enables `torch.compile` by default unless explicitly disabled; may OOM. `memory` @@ -85,10 +85,14 @@ Base the decision on available memory on the selected GPU(s). -`auto` checks selected GPU memory before applying FSDP. In multi-GPU runs it uses the least available memory across selected GPUs, and only turns on FSDP automatically when doing so can replace DiT offload. Text encoder, image encoder, and other component residency still follow the offload policy unless the model marks a high-memory resident path as safe. When the model default uses CFG and the user did not set a parallelism policy, `auto` may also enable CFG parallelism. `speed` intentionally does not check memory; it is the mode for users who prefer latency/throughput and accept OOM risk. +`auto` checks selected GPU memory before applying FSDP. In multi-GPU runs it uses the least available memory across selected GPUs, and only turns on FSDP automatically when doing so can replace DiT offload. Text encoder, image encoder, and other component residency still follow the offload policy unless the model marks a high-memory resident path as safe. When the model default uses CFG and the user did not set a parallelism policy, `auto` may also enable CFG parallelism. `speed` intentionally does not check memory; it is the mode for users who prefer latency/throughput and accept OOM risk, and it will turn on `torch.compile` automatically unless `--enable-torch-compile false` is set explicitly. The modes tune residency for native pipeline components declared to the component residency manager. Today this covers the major DiT, text/image encoder, VAE, vocoder, and upsampler components; DiT can use layerwise offload when supported, while text encoders use either resident execution or component CPU offload. Do not assume text-encoder layerwise offload unless a model implements and validates it. +When `torch.compile` is enabled, `--offload-during-compile` stays on by default. During compile warmup it temporarily offloads the DiT and evicts resident non-DiT components so `max-autotune` fits on tighter-memory GPUs, then restores the configured serving residency before real traffic. + +Breakable CUDA graph is a separate manual opt-in for supported image pipelines. If you enable `--enable-breakable-cuda-graph`, declare every served resolution in `--warmup-resolutions` so warmup captures matching graph signatures. + > [!NOTE] > The preset is intentionally coarse. A future continuous value such as `0.0` to `1.0` could express the speed-memory tradeoff more precisely, but it would need model-specific memory models and clearer user expectations. Until then, use the preset plus explicit flags for overrides. diff --git a/python/sglang/multimodal_gen/README.md b/python/sglang/multimodal_gen/README.md index 682739dca..ae2418c11 100644 --- a/python/sglang/multimodal_gen/README.md +++ b/python/sglang/multimodal_gen/README.md @@ -9,12 +9,13 @@ SGLang diffusion features an end-to-end unified pipeline for accelerating diffus ## Key Features SGLang Diffusion has the following features: - - Broad model support: Wan series, FastWan series, Hunyuan, LTX-2, Qwen-Image, Qwen-Image-Edit, Flux, Z-Image, GLM-Image - - Fast inference speed: enpowered by highly optimized kernel from sgl-kernel and efficient scheduler loop + - Broad model support: Wan, FastWan, FLUX, Qwen-Image, Z-Image, Ideogram 4, Krea-2, Cosmos3, LTX-2/LTX-2.3, LingBot World, SANA-WM, JoyEcho, MOVA, GLM-Image, ERNIE-Image, Hunyuan3D, and more + - Fast inference speed: empowered by optimized `sgl-kernel` kernels, scheduler/runtime improvements, caching acceleration, and native diffusion hot-path optimizations - Ease of use: OpenAI-compatible api, CLI, and python sdk support - Multi-platform support: - - NVIDIA GPUs (H100, H200, A100, B200, 4090) - - AMD GPUs (MI300X, MI325X) + - NVIDIA GPUs (H100, H200, A100, B200, 4090, 5090) + - AMD GPUs (MI300X, MI325X, MI355X) + - Intel XPUs - Ascend NPU (A2, A3) - Apple Silicon (M-series via MPS) - Moore Threads GPUs (MTT S5000) diff --git a/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/lingbot_world/lingbot_world_causal_denoising.py b/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/lingbot_world/lingbot_world_causal_denoising.py index 44615e2c8..680f8c27c 100644 --- a/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/lingbot_world/lingbot_world_causal_denoising.py +++ b/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/lingbot_world/lingbot_world_causal_denoising.py @@ -360,7 +360,7 @@ class LingBotWorldCausalDMDDenoisingStage(CausalDMDDenoisingStage): if sample_frames is None else int(sample_frames) * int(self.num_token_per_frame) ) - logger.info( + logger.debug( "LingBot interactive KV window: session_id=%s request_id=%s " "chunk_idx=%s mode=%s window_frames=%s sample_frames=%s " "cache_frames=%s sink_frames=%s current_frames=%s sample_tokens=%s "