[diffusion] refactor: validate and document spectrum controls (#33851)
This commit is contained in:
Mick
@@ -545,6 +545,7 @@ listed hardware and topology; it is not inherited by a similar GPU family.
|
||||
| Feature | Validation status | Notes |
|
||||
| --- | --- | --- |
|
||||
| Ulysses sequence parallelism | Verified: 8× B200, 4× H200, 4× H100, and Ulysses1/2/4/8 on MI300X and MI355X | Use `--ulysses-degree`; Ring is not compatible with H3's packed multi-segment attention. |
|
||||
| SageAttention | Supported | Use `--attention-backend sage_attn` to select the native packed varlen path; install the SageAttention dependency first. |
|
||||
| Tensor parallelism | Verified: B200 TP2 + Ulysses4; H100 TP2 + Ulysses2 and TP4 + Ulysses1 | `--tp-size` may be combined with Ulysses when the TP-local head count remains divisible by the Ulysses degree. On 4×H100, TP2 + Ulysses2 is the measured speed default. |
|
||||
| FSDP inference | Verified: 4× B200 and 4× H100 + Ulysses4 | Preserves H3's mixed BF16/FP32 parameter policy. B200 completed the exact eager comparison; H100 completed consecutive real requests at about 57 GB peak memory per GPU. |
|
||||
| Resident components | Verified: B200, H200, 4×H100 with TP, and 1/2/4/8× MI300X and MI355X | This is the recommended single-request latency path when the complete workload fits. |
|
||||
|
||||
+2
-1
@@ -1578,7 +1578,8 @@
|
||||
"tag": "approx",
|
||||
"pages": [
|
||||
"docs/sglang-diffusion/cache_dit",
|
||||
"docs/sglang-diffusion/teacache"
|
||||
"docs/sglang-diffusion/teacache",
|
||||
"docs/sglang-diffusion/spectrum"
|
||||
]
|
||||
},
|
||||
"docs/sglang-diffusion/progressive_resolution",
|
||||
|
||||
@@ -81,7 +81,7 @@ Use `sglang generate --help` and `sglang serve --help` for the full argument lis
|
||||
- `--lora-merge-mode {auto|merge|dynamic}`: choose how LoRA is applied. `auto` statically merges regular weights and uses dynamic LoRA for FSDP-sharded weights to avoid full-gather peaks.
|
||||
- `--num-gpus {N}`: number of GPUs to use
|
||||
- `--performance-mode {manual|auto|speed|memory}` / `--mode`: preset for latency/throughput and memory defaults. `auto` is the default and keeps safe offload defaults, using FSDP only for validated DiT-offload replacement paths; `speed` also enables `--enable-torch-compile` unless the model-specific deployment config opts out or you explicitly disable it. Use `manual` to keep performance-related server args under explicit user control. Explicit offload, FSDP, and parallelism flags take precedence in all modes.
|
||||
- `--tp-size {N}`: tensor parallelism size, mainly for encoders
|
||||
- `--tp-size {N}`: tensor parallelism size. Depending on the pipeline, it can shard the DiT, one or more encoders, or both.
|
||||
- `--sp-degree {N}`: sequence parallelism size
|
||||
- `--dp-size {N}` (alias `--data-parallel-size`): number of data-parallel replicas. Each replica is a full copy of the engine on `num_gpus / N` GPUs with its own ingress; generation requests round-robin across replicas, realtime sessions stick to the replica holding their state, and control operations (weights, LoRA, memory occupation, shutdown) apply to every replica. Combines with the other parallelism axes (`num_gpus = dp × cfg × tp × sp`); monolithic serving only.
|
||||
- `--ulysses-degree {N}` and `--ring-degree {N}`: USP parallelism controls
|
||||
@@ -183,7 +183,7 @@ sglang generate \
|
||||
HTTP server-only arguments are ignored by `sglang generate`.
|
||||
</Note>
|
||||
|
||||
For supported pipelines, Cache-DiT can be enabled with `SGLANG_CACHE_DIT_ENABLED=true` or `--cache-dit-config`. See [Cache-DiT](../cache_dit).
|
||||
For supported native pipelines, set `SGLANG_CACHE_DIT_ENABLED=true` to enable Cache-DiT. For the diffusers backend, use `--backend diffusers --cache-dit-config ...`. See [Cache-DiT](../cache_dit).
|
||||
|
||||
For supported image pipelines, breakable CUDA graph can be enabled with `--enable-breakable-cuda-graph`, but you must declare every served resolution in `--warmup-resolutions` so warmup captures matching graph signatures.
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@ Backend selection is performed by the shared attention layers (e.g. `LocalAttent
|
||||
When using the diffusers backend, `--attention-backend` is passed through to diffusers'
|
||||
`set_attention_backend` (e.g., `flash`, `_flash_3_hub`, `sage`, `xformers`, `native`).
|
||||
|
||||
- **CUDA**: prefers FlashAttention (FA3/FA4) when supported; otherwise falls back to PyTorch SDPA.
|
||||
- **CUDA**: prefers FlashAttention (FA3/FA4) when supported; otherwise falls back to PyTorch SDPA. On SM100/B200, dense non-causal fp16/bf16 native attention prefers cuDNN SDPA and falls back to FA4 if cuDNN has no compatible kernel.
|
||||
- **ROCm**: uses FlashAttention when available; otherwise falls back to PyTorch SDPA.
|
||||
- **Intel XPU**: uses XPU Flash Attention backend (fp16/bf16, head sizes 64/96/128/192/256); otherwise falls back to PyTorch SDPA.
|
||||
- **MUSA**: uses FlashAttention when available; also supports Sage Attention when installed; otherwise falls back to PyTorch SDPA.
|
||||
|
||||
@@ -3,11 +3,11 @@ title: "Caching Acceleration"
|
||||
description: "Compare caching acceleration strategies for diffusion models."
|
||||
tag: "approx"
|
||||
---
|
||||
SGLang provides two complementary caching strategies for Diffusion Transformer (DiT) models. Both reduce denoising cost by skipping redundant computation, but they operate at different levels.
|
||||
SGLang provides three complementary caching strategies for Diffusion Transformer (DiT) models. All reduce denoising cost by skipping redundant computation, but they operate at different levels.
|
||||
|
||||
## Overview
|
||||
|
||||
SGLang supports two complementary caching approaches:
|
||||
SGLang supports three complementary caching approaches:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
@@ -37,6 +37,12 @@ SGLang supports two complementary caching approaches:
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Skip entire denoising steps based on L1 similarity</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Simple, built-in</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Spectrum</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Timestep-level</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Forecast DiT features to skip selected denoising steps</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Experimental, model-validated tuning</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
@@ -86,6 +92,14 @@ See [TeaCache](./teacache) for detailed documentation.
|
||||
|
||||
For Flux and Qwen models, TeaCache is automatically disabled when CFG is enabled.
|
||||
|
||||
## Spectrum
|
||||
|
||||
Spectrum forecasts DiT features and skips selected denoising steps. It is
|
||||
approximate and currently applies only to selected native implementation paths.
|
||||
|
||||
See [Spectrum Acceleration](./spectrum) for supported model families,
|
||||
constraints, and request controls.
|
||||
|
||||
|
||||
## References
|
||||
|
||||
|
||||
@@ -119,7 +119,7 @@ Rows are grouped when a family shares the same runtime path or optimization supp
|
||||
<td>MiniMax-H3</td>
|
||||
<td><div className="sgd-id-list"><code>MiniMaxAI/MiniMax-H3</code></div></td>
|
||||
<td>T2VA / FL2VA / Ref2VA, 768p at 24 fps with synchronized audio</td>
|
||||
<td><span className="sgd-chip">Cache-DiT</span><span className="sgd-chip">Online FP8</span></td>
|
||||
<td><span className="sgd-chip">Cache-DiT</span><span className="sgd-chip">Sage</span><span className="sgd-chip">Online FP8</span></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Wan2.1 Fun</td>
|
||||
@@ -560,7 +560,7 @@ Optimization columns are abbreviated to keep the matrix readable:
|
||||
<td style={{padding: "9px 8px", backgroundColor: "rgba(255,255,255,0.02)"}}>768p · 24 fps</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
|
||||
@@ -34,7 +34,7 @@ sglang serve --model-path Qwen/Qwen-Image --port 30010
|
||||
- [CLI](/docs/sglang-diffusion/api/cli): run one-off generation jobs or launch a persistent server
|
||||
- [OpenAI-Compatible API](/docs/sglang-diffusion/api/openai_api): send image and video requests to the HTTP server
|
||||
- [Performance Overview](/docs/sglang-diffusion/performance-optimization): choose speed, memory, parallelism, caching, and quality-tradeoff levers
|
||||
- [Caching Acceleration](/docs/sglang-diffusion/caching-acceleration): use Cache-DiT or TeaCache to reduce denoising cost
|
||||
- [Caching Acceleration](/docs/sglang-diffusion/caching-acceleration): use Cache-DiT, TeaCache, or Spectrum to reduce denoising cost
|
||||
- [Quantization](/docs/sglang-diffusion/quantization): load quantized transformer checkpoints
|
||||
- [Contributing](/docs/sglang-diffusion/contributing): contribution workflow, adding new models, and CI perf baselines
|
||||
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
---
|
||||
title: "Spectrum Acceleration"
|
||||
description: "Approximate request-scoped denoising-step acceleration."
|
||||
tag: "approx"
|
||||
---
|
||||
|
||||
Spectrum forecasts DiT features and skips selected denoising steps. It is an
|
||||
approximation: validate visual or video quality and latency on the exact model,
|
||||
shape, hardware, and sampling settings you plan to deploy.
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
|
||||
--prompt "A paper boat floating through a misty mountain lake" \
|
||||
--enable-spectrum \
|
||||
--save-output
|
||||
```
|
||||
|
||||
## Scope and constraints
|
||||
|
||||
- Available only on native FLUX.1, Wan, HunyuanVideo, and SD3 implementation
|
||||
paths. It is not a `--backend diffusers` feature and does not currently
|
||||
cover FLUX.2.
|
||||
- The request control is available through `sglang generate` and Python
|
||||
sampling parameters. It is not a `sglang serve` or OpenAI-server request
|
||||
option yet.
|
||||
- Spectrum and `--enable-teacache` are mutually exclusive.
|
||||
- Start with the defaults. `--debug` adds shadow-prediction validation work, so
|
||||
its latency is not representative of normal Spectrum execution.
|
||||
|
||||
## Advanced controls
|
||||
|
||||
Use `--enable-spectrum` explicitly. Providing any Spectrum override also
|
||||
enables it, but that implicit behavior is intended for scripts rather than new
|
||||
commands.
|
||||
|
||||
| Flag | Default | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `--spectrum-window-size` | `2.0` | Initial step-skipping window |
|
||||
| `--spectrum-flex-window` | `0.75` | Window growth after a real forward |
|
||||
| `--spectrum-warmup-steps` | `5` | Initial exact DiT forwards |
|
||||
| `--spectrum-m` | `4` | Chebyshev basis count |
|
||||
| `--spectrum-lam` | `0.1` | Ridge regularization |
|
||||
| `--spectrum-tau-num-steps` | `50` | Chebyshev time horizon |
|
||||
| `--history-size` | `100` | Recent feature-history capacity |
|
||||
| `--taylor-order` | `1` | Local predictor order (`1`, `2`, or `3`) |
|
||||
| `--w` | `1.0` | Chebyshev/Taylor blend weight |
|
||||
|
||||
<Warning>
|
||||
These controls trade speed against output fidelity. Change one control at a
|
||||
time and retain a lossless baseline for comparison.
|
||||
</Warning>
|
||||
@@ -1,6 +1,7 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from dataclasses import dataclass
|
||||
|
||||
from sglang.multimodal_gen.configs.sample.sampling_params import CacheParams
|
||||
@@ -67,6 +68,45 @@ class SpectrumParams(CacheParams):
|
||||
tau_num_steps: int = 50
|
||||
taylor_order: int = 1
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
finite_numbers = {
|
||||
"window_size": self.window_size,
|
||||
"flex_window": self.flex_window,
|
||||
"w": self.w,
|
||||
"lam": self.lam,
|
||||
}
|
||||
for name, value in finite_numbers.items():
|
||||
if (
|
||||
isinstance(value, bool)
|
||||
or not isinstance(value, (int, float))
|
||||
or not math.isfinite(value)
|
||||
):
|
||||
raise ValueError(f"Spectrum {name} must be a finite number.")
|
||||
|
||||
if self.window_size <= 0:
|
||||
raise ValueError("Spectrum window_size must be greater than zero.")
|
||||
if self.flex_window < 0:
|
||||
raise ValueError("Spectrum flex_window must be non-negative.")
|
||||
if not 0 <= self.w <= 1:
|
||||
raise ValueError("Spectrum w must be between zero and one.")
|
||||
if self.lam < 0:
|
||||
raise ValueError("Spectrum lam must be non-negative.")
|
||||
|
||||
non_negative_ints = {"warmup_steps": self.warmup_steps}
|
||||
positive_ints = {
|
||||
"m": self.m,
|
||||
"history_size": self.history_size,
|
||||
"tau_num_steps": self.tau_num_steps,
|
||||
}
|
||||
for name, value in non_negative_ints.items():
|
||||
if isinstance(value, bool) or not isinstance(value, int) or value < 0:
|
||||
raise ValueError(f"Spectrum {name} must be a non-negative integer.")
|
||||
for name, value in positive_ints.items():
|
||||
if isinstance(value, bool) or not isinstance(value, int) or value <= 0:
|
||||
raise ValueError(f"Spectrum {name} must be a positive integer.")
|
||||
if self.taylor_order not in (1, 2, 3):
|
||||
raise ValueError("Spectrum taylor_order must be one of 1, 2, or 3.")
|
||||
|
||||
def get_total_forward_steps(
|
||||
self, num_inference_steps: int, do_cfg: bool, separate_cfg_branches: bool
|
||||
) -> int:
|
||||
|
||||
@@ -29,6 +29,7 @@ from sglang.multimodal_gen.configs.sample.sampling_params import (
|
||||
SamplingParams,
|
||||
_json_safe,
|
||||
)
|
||||
from sglang.multimodal_gen.configs.sample.spectrum import SpectrumParams
|
||||
from sglang.multimodal_gen.configs.sample.teacache import TeaCacheParams
|
||||
from sglang.multimodal_gen.configs.sample.wan import (
|
||||
FastWanT2V480PConfig,
|
||||
@@ -105,6 +106,29 @@ class TestSamplingParamsValidate(unittest.TestCase):
|
||||
):
|
||||
SamplingParams(enable_teacache=True, enable_spectrum=True)
|
||||
|
||||
def test_spectrum_params_reject_invalid_controls(self):
|
||||
invalid_controls = (
|
||||
{"window_size": 0},
|
||||
{"flex_window": -0.1},
|
||||
{"w": 1.1},
|
||||
{"lam": -0.1},
|
||||
{"warmup_steps": -1},
|
||||
{"m": 0},
|
||||
{"history_size": 0},
|
||||
{"tau_num_steps": 0},
|
||||
{"taylor_order": 4},
|
||||
)
|
||||
for kwargs in invalid_controls:
|
||||
with self.assertRaises(ValueError):
|
||||
SpectrumParams(**kwargs)
|
||||
|
||||
def test_spectrum_dict_is_validated_when_sampling_params_constructs_it(self):
|
||||
with self.assertRaisesRegex(ValueError, "history_size"):
|
||||
SamplingParams(
|
||||
enable_spectrum=True,
|
||||
spectrum_params={"history_size": 0},
|
||||
)
|
||||
|
||||
|
||||
class TestSamplingParamsSubclass(unittest.TestCase):
|
||||
def test_glm_image_rounds_resolution_up_to_multiple_of_32(self):
|
||||
|
||||
Reference in New Issue
Block a user