diff --git a/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx b/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx index 96e7fe8c3..c9a1755bf 100644 --- a/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx +++ b/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx @@ -545,6 +545,7 @@ listed hardware and topology; it is not inherited by a similar GPU family. | Feature | Validation status | Notes | | --- | --- | --- | | Ulysses sequence parallelism | Verified: 8× B200, 4× H200, 4× H100, and Ulysses1/2/4/8 on MI300X and MI355X | Use `--ulysses-degree`; Ring is not compatible with H3's packed multi-segment attention. | +| SageAttention | Supported | Use `--attention-backend sage_attn` to select the native packed varlen path; install the SageAttention dependency first. | | Tensor parallelism | Verified: B200 TP2 + Ulysses4; H100 TP2 + Ulysses2 and TP4 + Ulysses1 | `--tp-size` may be combined with Ulysses when the TP-local head count remains divisible by the Ulysses degree. On 4×H100, TP2 + Ulysses2 is the measured speed default. | | FSDP inference | Verified: 4× B200 and 4× H100 + Ulysses4 | Preserves H3's mixed BF16/FP32 parameter policy. B200 completed the exact eager comparison; H100 completed consecutive real requests at about 57 GB peak memory per GPU. | | Resident components | Verified: B200, H200, 4×H100 with TP, and 1/2/4/8× MI300X and MI355X | This is the recommended single-request latency path when the complete workload fits. | diff --git a/docs/docs.json b/docs/docs.json index acc5a8977..624077363 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -1578,7 +1578,8 @@ "tag": "approx", "pages": [ "docs/sglang-diffusion/cache_dit", - "docs/sglang-diffusion/teacache" + "docs/sglang-diffusion/teacache", + "docs/sglang-diffusion/spectrum" ] }, "docs/sglang-diffusion/progressive_resolution", diff --git a/docs/docs/sglang-diffusion/api/cli.mdx b/docs/docs/sglang-diffusion/api/cli.mdx index 030a9b0d3..4a96e2d3b 100644 --- a/docs/docs/sglang-diffusion/api/cli.mdx +++ b/docs/docs/sglang-diffusion/api/cli.mdx @@ -81,7 +81,7 @@ Use `sglang generate --help` and `sglang serve --help` for the full argument lis - `--lora-merge-mode {auto|merge|dynamic}`: choose how LoRA is applied. `auto` statically merges regular weights and uses dynamic LoRA for FSDP-sharded weights to avoid full-gather peaks. - `--num-gpus {N}`: number of GPUs to use - `--performance-mode {manual|auto|speed|memory}` / `--mode`: preset for latency/throughput and memory defaults. `auto` is the default and keeps safe offload defaults, using FSDP only for validated DiT-offload replacement paths; `speed` also enables `--enable-torch-compile` unless the model-specific deployment config opts out or you explicitly disable it. Use `manual` to keep performance-related server args under explicit user control. Explicit offload, FSDP, and parallelism flags take precedence in all modes. -- `--tp-size {N}`: tensor parallelism size, mainly for encoders +- `--tp-size {N}`: tensor parallelism size. Depending on the pipeline, it can shard the DiT, one or more encoders, or both. - `--sp-degree {N}`: sequence parallelism size - `--dp-size {N}` (alias `--data-parallel-size`): number of data-parallel replicas. Each replica is a full copy of the engine on `num_gpus / N` GPUs with its own ingress; generation requests round-robin across replicas, realtime sessions stick to the replica holding their state, and control operations (weights, LoRA, memory occupation, shutdown) apply to every replica. Combines with the other parallelism axes (`num_gpus = dp × cfg × tp × sp`); monolithic serving only. - `--ulysses-degree {N}` and `--ring-degree {N}`: USP parallelism controls @@ -183,7 +183,7 @@ sglang generate \ HTTP server-only arguments are ignored by `sglang generate`. -For supported pipelines, Cache-DiT can be enabled with `SGLANG_CACHE_DIT_ENABLED=true` or `--cache-dit-config`. See [Cache-DiT](../cache_dit). +For supported native pipelines, set `SGLANG_CACHE_DIT_ENABLED=true` to enable Cache-DiT. For the diffusers backend, use `--backend diffusers --cache-dit-config ...`. See [Cache-DiT](../cache_dit). For supported image pipelines, breakable CUDA graph can be enabled with `--enable-breakable-cuda-graph`, but you must declare every served resolution in `--warmup-resolutions` so warmup captures matching graph signatures. diff --git a/docs/docs/sglang-diffusion/attention_backends.mdx b/docs/docs/sglang-diffusion/attention_backends.mdx index 87117753c..09f0e3423 100644 --- a/docs/docs/sglang-diffusion/attention_backends.mdx +++ b/docs/docs/sglang-diffusion/attention_backends.mdx @@ -14,7 +14,7 @@ Backend selection is performed by the shared attention layers (e.g. `LocalAttent When using the diffusers backend, `--attention-backend` is passed through to diffusers' `set_attention_backend` (e.g., `flash`, `_flash_3_hub`, `sage`, `xformers`, `native`). -- **CUDA**: prefers FlashAttention (FA3/FA4) when supported; otherwise falls back to PyTorch SDPA. +- **CUDA**: prefers FlashAttention (FA3/FA4) when supported; otherwise falls back to PyTorch SDPA. On SM100/B200, dense non-causal fp16/bf16 native attention prefers cuDNN SDPA and falls back to FA4 if cuDNN has no compatible kernel. - **ROCm**: uses FlashAttention when available; otherwise falls back to PyTorch SDPA. - **Intel XPU**: uses XPU Flash Attention backend (fp16/bf16, head sizes 64/96/128/192/256); otherwise falls back to PyTorch SDPA. - **MUSA**: uses FlashAttention when available; also supports Sage Attention when installed; otherwise falls back to PyTorch SDPA. diff --git a/docs/docs/sglang-diffusion/caching-acceleration.mdx b/docs/docs/sglang-diffusion/caching-acceleration.mdx index 7c9060432..bdcdd041f 100644 --- a/docs/docs/sglang-diffusion/caching-acceleration.mdx +++ b/docs/docs/sglang-diffusion/caching-acceleration.mdx @@ -3,11 +3,11 @@ title: "Caching Acceleration" description: "Compare caching acceleration strategies for diffusion models." tag: "approx" --- -SGLang provides two complementary caching strategies for Diffusion Transformer (DiT) models. Both reduce denoising cost by skipping redundant computation, but they operate at different levels. +SGLang provides three complementary caching strategies for Diffusion Transformer (DiT) models. All reduce denoising cost by skipping redundant computation, but they operate at different levels. ## Overview -SGLang supports two complementary caching approaches: +SGLang supports three complementary caching approaches: @@ -37,6 +37,12 @@ SGLang supports two complementary caching approaches: + + + + + +
Skip entire denoising steps based on L1 similarity Simple, built-in
SpectrumTimestep-levelForecast DiT features to skip selected denoising stepsExperimental, model-validated tuning
@@ -86,6 +92,14 @@ See [TeaCache](./teacache) for detailed documentation. For Flux and Qwen models, TeaCache is automatically disabled when CFG is enabled. +## Spectrum + +Spectrum forecasts DiT features and skips selected denoising steps. It is +approximate and currently applies only to selected native implementation paths. + +See [Spectrum Acceleration](./spectrum) for supported model families, +constraints, and request controls. + ## References diff --git a/docs/docs/sglang-diffusion/compatibility_matrix.mdx b/docs/docs/sglang-diffusion/compatibility_matrix.mdx index d73e782c0..8b90d91b6 100644 --- a/docs/docs/sglang-diffusion/compatibility_matrix.mdx +++ b/docs/docs/sglang-diffusion/compatibility_matrix.mdx @@ -119,7 +119,7 @@ Rows are grouped when a family shares the same runtime path or optimization supp MiniMax-H3
MiniMaxAI/MiniMax-H3
T2VA / FL2VA / Ref2VA, 768p at 24 fps with synchronized audio - Cache-DiTOnline FP8 + Cache-DiTSageOnline FP8 Wan2.1 Fun @@ -560,7 +560,7 @@ Optimization columns are abbreviated to keep the matrix readable: 768p · 24 fps ❌ ❌ - ❌ + ✅ ❌ ❌ ❌ diff --git a/docs/docs/sglang-diffusion/index.mdx b/docs/docs/sglang-diffusion/index.mdx index 06aeafe67..31276783c 100644 --- a/docs/docs/sglang-diffusion/index.mdx +++ b/docs/docs/sglang-diffusion/index.mdx @@ -34,7 +34,7 @@ sglang serve --model-path Qwen/Qwen-Image --port 30010 - [CLI](/docs/sglang-diffusion/api/cli): run one-off generation jobs or launch a persistent server - [OpenAI-Compatible API](/docs/sglang-diffusion/api/openai_api): send image and video requests to the HTTP server - [Performance Overview](/docs/sglang-diffusion/performance-optimization): choose speed, memory, parallelism, caching, and quality-tradeoff levers -- [Caching Acceleration](/docs/sglang-diffusion/caching-acceleration): use Cache-DiT or TeaCache to reduce denoising cost +- [Caching Acceleration](/docs/sglang-diffusion/caching-acceleration): use Cache-DiT, TeaCache, or Spectrum to reduce denoising cost - [Quantization](/docs/sglang-diffusion/quantization): load quantized transformer checkpoints - [Contributing](/docs/sglang-diffusion/contributing): contribution workflow, adding new models, and CI perf baselines diff --git a/docs/docs/sglang-diffusion/spectrum.mdx b/docs/docs/sglang-diffusion/spectrum.mdx new file mode 100644 index 000000000..55ca7fc7c --- /dev/null +++ b/docs/docs/sglang-diffusion/spectrum.mdx @@ -0,0 +1,54 @@ +--- +title: "Spectrum Acceleration" +description: "Approximate request-scoped denoising-step acceleration." +tag: "approx" +--- + +Spectrum forecasts DiT features and skips selected denoising steps. It is an +approximation: validate visual or video quality and latency on the exact model, +shape, hardware, and sampling settings you plan to deploy. + +## Quick start + +```bash +sglang generate \ + --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ + --prompt "A paper boat floating through a misty mountain lake" \ + --enable-spectrum \ + --save-output +``` + +## Scope and constraints + +- Available only on native FLUX.1, Wan, HunyuanVideo, and SD3 implementation + paths. It is not a `--backend diffusers` feature and does not currently + cover FLUX.2. +- The request control is available through `sglang generate` and Python + sampling parameters. It is not a `sglang serve` or OpenAI-server request + option yet. +- Spectrum and `--enable-teacache` are mutually exclusive. +- Start with the defaults. `--debug` adds shadow-prediction validation work, so + its latency is not representative of normal Spectrum execution. + +## Advanced controls + +Use `--enable-spectrum` explicitly. Providing any Spectrum override also +enables it, but that implicit behavior is intended for scripts rather than new +commands. + +| Flag | Default | Purpose | +| --- | --- | --- | +| `--spectrum-window-size` | `2.0` | Initial step-skipping window | +| `--spectrum-flex-window` | `0.75` | Window growth after a real forward | +| `--spectrum-warmup-steps` | `5` | Initial exact DiT forwards | +| `--spectrum-m` | `4` | Chebyshev basis count | +| `--spectrum-lam` | `0.1` | Ridge regularization | +| `--spectrum-tau-num-steps` | `50` | Chebyshev time horizon | +| `--history-size` | `100` | Recent feature-history capacity | +| `--taylor-order` | `1` | Local predictor order (`1`, `2`, or `3`) | +| `--w` | `1.0` | Chebyshev/Taylor blend weight | + + +These controls trade speed against output fidelity. Change one control at a +time and retain a lossless baseline for comparison. + diff --git a/python/sglang/multimodal_gen/configs/sample/spectrum.py b/python/sglang/multimodal_gen/configs/sample/spectrum.py index b3cf0d7ed..675706e01 100644 --- a/python/sglang/multimodal_gen/configs/sample/spectrum.py +++ b/python/sglang/multimodal_gen/configs/sample/spectrum.py @@ -1,6 +1,7 @@ # SPDX-License-Identifier: Apache-2.0 from __future__ import annotations +import math from dataclasses import dataclass from sglang.multimodal_gen.configs.sample.sampling_params import CacheParams @@ -67,6 +68,45 @@ class SpectrumParams(CacheParams): tau_num_steps: int = 50 taylor_order: int = 1 + def __post_init__(self) -> None: + finite_numbers = { + "window_size": self.window_size, + "flex_window": self.flex_window, + "w": self.w, + "lam": self.lam, + } + for name, value in finite_numbers.items(): + if ( + isinstance(value, bool) + or not isinstance(value, (int, float)) + or not math.isfinite(value) + ): + raise ValueError(f"Spectrum {name} must be a finite number.") + + if self.window_size <= 0: + raise ValueError("Spectrum window_size must be greater than zero.") + if self.flex_window < 0: + raise ValueError("Spectrum flex_window must be non-negative.") + if not 0 <= self.w <= 1: + raise ValueError("Spectrum w must be between zero and one.") + if self.lam < 0: + raise ValueError("Spectrum lam must be non-negative.") + + non_negative_ints = {"warmup_steps": self.warmup_steps} + positive_ints = { + "m": self.m, + "history_size": self.history_size, + "tau_num_steps": self.tau_num_steps, + } + for name, value in non_negative_ints.items(): + if isinstance(value, bool) or not isinstance(value, int) or value < 0: + raise ValueError(f"Spectrum {name} must be a non-negative integer.") + for name, value in positive_ints.items(): + if isinstance(value, bool) or not isinstance(value, int) or value <= 0: + raise ValueError(f"Spectrum {name} must be a positive integer.") + if self.taylor_order not in (1, 2, 3): + raise ValueError("Spectrum taylor_order must be one of 1, 2, or 3.") + def get_total_forward_steps( self, num_inference_steps: int, do_cfg: bool, separate_cfg_branches: bool ) -> int: diff --git a/python/sglang/multimodal_gen/test/unit/test_sampling_params.py b/python/sglang/multimodal_gen/test/unit/test_sampling_params.py index f8a5b0f3e..58f3cf19e 100644 --- a/python/sglang/multimodal_gen/test/unit/test_sampling_params.py +++ b/python/sglang/multimodal_gen/test/unit/test_sampling_params.py @@ -29,6 +29,7 @@ from sglang.multimodal_gen.configs.sample.sampling_params import ( SamplingParams, _json_safe, ) +from sglang.multimodal_gen.configs.sample.spectrum import SpectrumParams from sglang.multimodal_gen.configs.sample.teacache import TeaCacheParams from sglang.multimodal_gen.configs.sample.wan import ( FastWanT2V480PConfig, @@ -105,6 +106,29 @@ class TestSamplingParamsValidate(unittest.TestCase): ): SamplingParams(enable_teacache=True, enable_spectrum=True) + def test_spectrum_params_reject_invalid_controls(self): + invalid_controls = ( + {"window_size": 0}, + {"flex_window": -0.1}, + {"w": 1.1}, + {"lam": -0.1}, + {"warmup_steps": -1}, + {"m": 0}, + {"history_size": 0}, + {"tau_num_steps": 0}, + {"taylor_order": 4}, + ) + for kwargs in invalid_controls: + with self.assertRaises(ValueError): + SpectrumParams(**kwargs) + + def test_spectrum_dict_is_validated_when_sampling_params_constructs_it(self): + with self.assertRaisesRegex(ValueError, "history_size"): + SamplingParams( + enable_spectrum=True, + spectrum_params={"history_size": 0}, + ) + class TestSamplingParamsSubclass(unittest.TestCase): def test_glm_image_rounds_resolution_up_to_multiple_of_32(self):