minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 10 Packages Projects Releases Wiki Activity

[Diffusion] Add cumulative extra-high quality tier (#37422)

Browse Source
This commit is contained in:
Xiaoyu Zhang
2026-09-02 10:26:13 +08:00
committed by GitHub
parent 26f760d5c0
commit 1aa8299d1d
34 changed files with 449 additions and 215 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/diffusion/LingBot-World/LingBot-World-2.0.mdx
+1 -1
View File
@@ -84,7 +84,7 @@ Send this MessagePack map immediately after the WebSocket opens.
| `num_inference_steps` | integer | No | Denoising steps per chunk. LingBot defaults to `4` when omitted. |
| `guidance_scale` | number | No | Classifier-free guidance scale. Realtime LingBot commonly uses `1`. |
| `negative_prompt` | string | No | Negative prompt passed to the diffusion pipeline. |
| `quality` | `"lossless"`, `"high"` | No | `lossless` keeps FP32 VAE decode. `high` uses the validated BF16 decode path for lower per-chunk latency. |
| `quality` | `"lossless"`, `"extra-high"`, `"high"` | No | `lossless` keeps FP32 VAE decode. `extra-high` keeps FP32 decode and enables only any eligible request-gated kernel fusions. `high` includes those fusions and uses the validated BF16 decode path for lower per-chunk latency. |
| `max_chunks` | integer | No | Stop after this many chunks. Omit for a continuous session. |
| `realtime_causal_sink_size` | integer | No | Number of sink frames/tokens retained in the causal attention window. |
| `realtime_causal_kv_cache_num_frames` | integer | No | Number of recent frames retained in the causal KV cache window. |
docs/cookbook/diffusion/LingBot-World/LingBot-World.mdx
+1 -1
View File
@@ -79,7 +79,7 @@ Send this MessagePack map immediately after the WebSocket opens.
| `num_inference_steps` | integer | No | Denoising steps per chunk. LingBot defaults to `4` when omitted. |
| `guidance_scale` | number | No | Classifier-free guidance scale. Realtime LingBot commonly uses `1`. |
| `negative_prompt` | string | No | Negative prompt passed to the diffusion pipeline. |
| `quality` | `"lossless"`, `"high"` | No | `lossless` keeps FP32 VAE decode. `high` uses the validated BF16 decode path for lower per-chunk latency. |
| `quality` | `"lossless"`, `"extra-high"`, `"high"` | No | `lossless` keeps FP32 VAE decode. `extra-high` keeps FP32 decode and enables only any eligible request-gated kernel fusions. `high` includes those fusions and uses the validated BF16 decode path for lower per-chunk latency. |
| `max_chunks` | integer | No | Stop after this many chunks. Omit for a continuous session. |
| `realtime_causal_sink_size` | integer | No | Number of sink frames/tokens retained in the causal attention window. |
| `realtime_causal_kv_cache_num_frames` | integer | No | Number of recent frames retained in the causal KV cache window. |
docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx
+25 -6
View File
@@ -673,17 +673,22 @@ done
### Choose the quality level
`quality` is a request-scoped sampling parameter with two validated levels:
`quality` is a cumulative request-scoped optimization parameter with three
levels:
- `"lossless"` (default): the exact reference path. Output is bit-exact
against the reference implementation and the CI ground truth.
- `"extra-high"`: includes the global fusion-only tier but does not enable
Cache-DiT or another approximate optimization. MiniMax-H3 currently has no
request-gated fusion site, so its denoise path is the same as `lossless`.
- `"high"`: the audited accelerated path. Quality is guaranteed (the audited
Cache-DiT configuration measures SSIM 0.931 / PSNR 28.16 dB against
`lossless`), but output is no longer bit-identical to the reference.
One resident server serves both levels; a `quality: "high"` request mounts
its audited Cache-DiT policy at the batch boundary, and a later
`quality: "lossless"` request removes the hooks before denoising.
One resident server serves all three levels; a `quality: "high"` request
mounts its audited Cache-DiT policy at the batch boundary, and a later
`quality: "lossless"` or `quality: "extra-high"` request removes the hooks
before denoising.
Start the validated server once:
@@ -720,6 +725,19 @@ omitting the field is equivalent.
</Tab>
<Tab title="extra-high">
The global fusion-only tier. It does not enable MiniMax-H3 Cache-DiT and
currently follows the same H3 denoise path as `lossless`.
```json Request field
{
"quality": "extra-high"
}
```
</Tab>
<Tab title="high">
The audited accelerated path. Use it when you can trade bit-exactness for
@@ -740,6 +758,7 @@ The measured trade-off is:
| `quality` | Mean <br />inference <br />latency | Speedup | SSIM vs <br />lossless | PSNR vs <br />lossless | Expected <br />trade-off |
| --- | ---: | ---: | ---: | ---: | --- |
| `lossless` | 75.10 s | 1.00× | 1.000 | exact | Native reference path |
| `extra-high` | Not separately measured | — | Same H3 denoise path | Same H3 denoise path | Fusion-only tier; no H3-specific request-gated site yet |
| `high` | 53.70 s | 1.40× | 0.931 | 28.16 dB | Smallest same-seed visual change |
These numbers use 1344×768, 124-frame, 24 fps T2VA with 50 inference steps,
@@ -766,8 +785,8 @@ name, for example `sglang generate --quality high`.
For manually tuned Cache-DiT experiments outside that validated path, omit
the request `quality` field and set `--enable-cache-dit` or the
process-wide `SGLANG_CACHE_DIT_*` defaults. An explicit `quality`
(including `"lossless"`) takes H3 off the generic Cache-DiT path. The
process-wide `SGLANG_CACHE_DIT_*` defaults. Any explicit `quality`, including
`"lossless"` and `"extra-high"`, takes H3 off the generic Cache-DiT path. The
24 GB layerwise recipe above can use the same switch; skipped blocks are
not streamed.
docs/docs/sglang-diffusion/api/cli.mdx
+12 -12
View File
@@ -64,7 +64,7 @@ sglang serve \
--port 30010
```
For request and response examples, see [OpenAI-Compatible API](./openai_api).
For request and response examples, see [OpenAI-Compatible API](/docs/sglang-diffusion/api/openai_api).
<Tip>
Use `sglang generate --help` and `sglang serve --help` for the full argument list. The CLI help output is the source of truth for exhaustive flags.
@@ -97,16 +97,16 @@ Use `sglang generate --help` and `sglang serve --help` for the full argument lis
- `--warmup-mode {off|request|server}`: control startup warmup for `sglang serve`; `off` skips warmup, `request` primes the request path, and `server` runs a full synthetic server warmup before serving traffic
- `--enable-torch-compile {true|false}`: compile native diffusion hot paths. When no warmup mode is configured, this also enables server warmup so first real requests do not pay compile latency.
- `--offload-during-compile {true|false}`: when compile warmup is active, temporarily layerwise-offload DiT weights and move resident non-DiT components off-device so `max-autotune` fits on tighter-memory GPUs; the configured serving residency is restored before real traffic. Skipped under existing layerwise offload, Cache-DiT, or FSDP.
- `--enable-breakable-cuda-graph {true|false}`: capture supported DiT forwards as breakable CUDA graph segments to reduce launch overhead. Requires `--warmup-resolutions` for every served resolution because each resolution is captured separately. A `quality=high` request is rejected when it would mount request-scoped DiT fusions that were not present during lossless graph capture; VAE-only high-quality paths remain compatible.
- `--enable-breakable-cuda-graph {true|false}`: capture supported DiT forwards as breakable CUDA graph segments to reduce launch overhead. Requires `--warmup-resolutions` for every served resolution because each resolution is captured separately. An `extra-high` or `high` request is rejected when it would mount request-scoped DiT fusions that were not present during lossless graph capture; VAE-only request-gated paths remain compatible.
- `--bcg-text-buckets {N...}`: prompt-length padding buckets for breakable CUDA graph capture/replay reuse.
- `--attention-backend {BACKEND}`: attention backend for native SGLang and diffusers pipelines
- `--component-attention-backends {MAP}`: per-component attention backend overrides, for example `text_encoder=torch_sdpa,transformer=fa`
- `--attention-backend-config {CONFIG}`: attention backend configuration
- `--srt-encoder-url {HTTPADDRESS}`: address of SGLang srt server with AR model for GLM-Image like models. See [Models with AR Stage](../models_with_ar).
- `--srt-encoder-url {HTTPADDRESS}`: address of SGLang srt server with AR model for GLM-Image like models. See [Models with AR Stage](/docs/sglang-diffusion/models_with_ar).
- `--srt-encoder-timeout {SECONDS}`: Timeout in seconds for HTTP requests to the SGLang encoder server
- `--srt-encoder-connection-timeout {SECONDS}`: TCP connection timeout in seconds for SGLang encoder server
- `--scheduler-rpc-timeout {SECONDS}`: optional end-to-end deadline for an internal scheduler RPC, including scheduler queue time. It is unset by default so valid long-running and queued video jobs are not failed by the transport layer. Set it only when the deployment requires a bounded request deadline; caller cancellation and server shutdown remain effective without it.
- `--pe-server-url {HTTPADDRESS}`: url of SGLang server hosting the PE model (e.g., for ERNIE-Image). See [Models with Prompt Enhancement](../models_with_pe).
- `--pe-server-url {HTTPADDRESS}`: url of SGLang server hosting the PE model (e.g., for ERNIE-Image). See [Models with Prompt Enhancement](/docs/sglang-diffusion/models_with_pe).
### Sampling and output
@@ -114,11 +114,11 @@ Use `sglang generate --help` and `sglang serve --help` for the full argument lis
- `--image-path {PATH} [{PATH} ...]`: input image(s) for image-to-video or image-to-image generation
- `--num-inference-steps {STEPS}` and `--seed {SEED}`
- `--num-outputs-per-prompt {N}` / `--num-outputs {N}`: generate multiple outputs for each prompt. A scalar seed expands as `seed + output_index`.
- `--quality {lossless,high}`: request-level quality. `lossless` (default) keeps the exact reference path, bit-exact against the reference implementation; `high` opts into the model-owned validated accelerated path, whose quality stays guaranteed but is not bit-exact. Support and validated deployment constraints are model-specific.
- `--quality {lossless,extra-high,high}`: cumulative request-level optimization tier. `lossless` (default) keeps the selected deployment's reference path and all unconditional bit-exact replacements. `extra-high` adds only request-gated DiT/VAE kernel fusions; the tier does not itself enable sparse, caching, or other approximate paths. `high` includes the complete `extra-high` set and may also enable model-owned approximate optimizations. Separately configured quantization, attention, or caching options still apply. Support and validation constraints are model-specific.
- `--height {HEIGHT}`, `--width {WIDTH}`, `--num-frames {N}`, `--fps {FPS}`
- `--output-path {PATH}`, `--output-file-name {NAME}`, `--save-output`, `--return-frames`
For frame interpolation and upscaling, see [Post-Processing](./post_processing).
For frame interpolation and upscaling, see [Post-Processing](/docs/sglang-diffusion/api/post_processing).
### Quantization
@@ -162,7 +162,7 @@ The same contract applies to every weighted component: path routing is generic,
while quantized materialization is capability-based. Native auxiliary loaders
whose current materializer expects plain state dicts reject unsupported
quantization metadata before model construction. See
[Quantized Component Repositories](../quantization#quantized-component-repositories)
[Quantized Component Repositories](/docs/sglang-diffusion/quantization#quantized-component-repositories)
for the current component matrix. A model cookbook is the source of truth for
published, model-specific checkpoint examples; for example, all H3 sources and
their exact overlays are kept in one
@@ -191,8 +191,8 @@ For supported realtime causal video models, `--kv-cache-quant {off|int4|int2}`
compresses completed KV-cache chunks independently of transformer weight
quantization. It is lossy and disabled by default.
See [Realtime and Causal Video Models](../realtime_models) for the runtime and
model scope, and [Quantization](../quantization) for supported quantization
See [Realtime and Causal Video Models](/docs/sglang-diffusion/realtime_models) for the runtime and
model scope, and [Quantization](/docs/sglang-diffusion/quantization) for supported quantization
families and examples.
### Request logging
@@ -263,7 +263,7 @@ sglang generate \
HTTP server-only arguments are ignored by `sglang generate`.
</Note>
For supported native pipelines, set `SGLANG_CACHE_DIT_ENABLED=true` to enable Cache-DiT. It can run with DiT layerwise offload; it cannot run with FSDP. For the diffusers backend, use `--backend diffusers --cache-dit-config ...`. See [Cache-DiT](../cache_dit).
For supported native pipelines, set `SGLANG_CACHE_DIT_ENABLED=true` to enable Cache-DiT. It can run with DiT layerwise offload; it cannot run with FSDP. For the diffusers backend, use `--backend diffusers --cache-dit-config ...`. See [Cache-DiT](/docs/sglang-diffusion/cache_dit).
For supported image pipelines, breakable CUDA graph can be enabled with `--enable-breakable-cuda-graph`, but you must declare every served resolution in `--warmup-resolutions` so warmup captures matching graph signatures.
@@ -394,7 +394,7 @@ export SGLANG_S3_SECRET_ACCESS_KEY=your-secret-key
export SGLANG_S3_ENDPOINT_URL=https://minio.example.com
```
See [Environment Variables](../environment_variables) for the full set of storage options.
See [Environment Variables](/docs/sglang-diffusion/environment_variables) for the full set of storage options.
## Component Path Overrides
@@ -416,7 +416,7 @@ selected loader must support that serialized format. Native plain-state loaders
fail closed; library-managed components inherit the corresponding Transformers
or Diffusers support. The transformer-specific `--quantization` flag does not
select the format of component checkpoints; their own metadata does. See
[Quantized Component Repositories](../quantization#quantized-component-repositories).
[Quantized Component Repositories](/docs/sglang-diffusion/quantization#quantized-component-repositories).
## Component Attention Backend Overrides
docs/docs/sglang-diffusion/api/openai_api.mdx
+1 -1
View File
@@ -129,7 +129,7 @@ The server implements an OpenAI-compatible Images API under the `/v1/images` nam
#### Request quality
`quality` selects a model-owned sampling level when that model advertises one: use `lossless` for the reference path or `high` for a validated accelerated path. Omit it (or send OpenAI's default `auto`) to keep the runtime default. It is distinct from `output_quality`, which controls only output-file compression. The same extension is accepted by image edits and video requests.
`quality` selects a cumulative request-level optimization tier: `lossless` keeps the selected deployment's reference path and all unconditional bit-exact replacements; `extra-high` additionally enables only request-gated DiT/VAE kernel fusions; `high` includes the full `extra-high` set and may also enable model-owned sparse, caching, lower-precision, or other approximate paths. The tier does not override separately configured quantization, attention, or caching options. Omit it (or send OpenAI's default `auto`) to keep the `lossless` runtime default. It is distinct from `output_quality`, which controls only output-file compression. The same extension is accepted by image edits and video requests.
**Python Example (b64_json response):**
docs/docs/sglang-diffusion/fused_kernels.mdx
+58 -23
View File
@@ -8,22 +8,38 @@ Diffusion transformers and VAEs spend a large share of their non-GEMM time on sh
This page is an inventory: what each kernel fuses, what its numerical contract is, and which models use it. It is not a lever you tune — most of these kernels are on by default and require no flag. The one switch is `--quality`, described below.
## Two numerical contracts
## Numerical contracts and quality tiers
Multi-step denoising amplifies a per-step rounding difference into visible quality loss, so "close enough" and "bit-exact" are different products here. Every kernel in the package falls into one of two classes.
Multi-step denoising amplifies a per-step rounding difference into visible quality loss, so "close enough" and "bit-exact" are different products here. The `quality` switch distinguishes unconditional bit-exact replacements from non-bit-exact eager-chain fusions:
**Bit-exact — mounted unconditionally.** The kernel reproduces every rounding boundary of the eager chain, so `torch.equal` holds against the reference. Some go quite far to get there: the fused LayerNorm+modulate kernel replicates PyTorch's `vectorized_layer_norm_kernel` down to its Welford update order, guarded reciprocal, and warp-fold tree; the fused RMSNorm+scale/shift kernel replicates FlashInfer's CuTe-DSL `RMSNormKernel` fragment order and `shfl.bfly` fold. Because the dispatch they replicate can change underneath them, each one still verifies itself against the live eager chain on first sight and falls back permanently on any mismatch.
**Not bit-exact — request-gated.** These differ from eager only at half-precision rounding-order level, but that is enough to matter, so they are mounted only for `quality="high"` requests, at batch boundaries, all-or-nothing per transformer. The default `quality="lossless"` runs the unmodified reference chain.
**Not bit-exact — request-gated.** These differ from eager only at half-precision rounding-order level, but that is enough to matter, so they are mounted only for `quality="extra-high"` and `quality="high"` requests, at batch boundaries, all-or-nothing per transformer. The default `quality="lossless"` runs the unmodified reference chain.
**Model/checkpoint-native.** Generic close-contract kernels, sparse operators, and FP8/NVFP4 producers can be part of a model implementation or a separately selected deployment path. They are documented in the inventory, but `quality` does not select or undo those choices.
<Note>
A plain fp32 single-pass norm fusion looks harmless and is not. On ERNIE-Image it moved the 50-step trajectory to 18.83 dB PSNR at `quality=high`, which is what motivated the bit-exact rewrite of that path.
A plain fp32 single-pass norm fusion looks harmless and is not. On ERNIE-Image it moved the 50-step trajectory to 18.83 dB PSNR, which is what motivated the bit-exact rewrite of that path.
</Note>
The quality levels are cumulative:
| `quality` | Included optimization set |
| --- | --- |
| `lossless` | The selected deployment's reference path plus every unconditional bit-exact replacement |
| `extra-high` | Everything in `lossless`, plus request-gated DiT and VAE kernel fusions; this level does not itself enable sparse, caching, or another approximate path |
| `high` | Everything in `extra-high`, plus any model-owned high-only optimization, such as an audited Cache-DiT policy or lower-precision VAE decode |
If a model has no eligible request-gated fusion, `extra-high` can execute the same path as `lossless`. Likewise, `high` adds only the model-specific high-only paths that the active pipeline implements.
<Note>
`quality` is not a master precision switch. A quantized checkpoint, an explicitly selected approximate attention backend, or an independently enabled cache remains active at every quality tier.
</Note>
## Enabling the request-gated set
```bash
sglang generate --model-path MODEL_PATH --prompt "..." --quality high
sglang generate --model-path MODEL_PATH --prompt "..." --quality extra-high
```
The server default stays `lossless`; the OpenAI-compatible endpoints carry it per request. Images:
@@ -31,7 +47,7 @@ The server default stays `lossless`; the OpenAI-compatible endpoints carry it pe
```bash
curl -X POST http://${HOST}:${PORT}/v1/images/generations \
-H 'Content-Type: application/json' \
-d '{"model": "MODEL_PATH", "prompt": "...", "quality": "high"}'
-d '{"model": "MODEL_PATH", "prompt": "...", "quality": "extra-high"}'
```
Video, same field:
@@ -39,7 +55,7 @@ Video, same field:
```bash
curl -X POST http://${HOST}:${PORT}/v1/videos \
-H 'Content-Type: application/json' \
-d '{"model": "MODEL_PATH", "prompt": "...", "quality": "high"}'
-d '{"model": "MODEL_PATH", "prompt": "...", "quality": "extra-high"}'
```
<Warning>
@@ -50,28 +66,33 @@ The `quality` field in a **video response** body is unrelated. It is Sora-compat
<Warning>
Do not combine request-gated DiT fusions with `--enable-breakable-cuda-graph`.
BCG warmup captures the lossless module branches before a high-quality request
mounts its DiT fusions, so replay would bypass the requested kernels. SGLang
rejects this combination for models with eligible DiT quality sites. Models
whose high-quality path changes only VAE decode remain allowed because BCG
captures the DiT only.
BCG warmup captures the lossless module branches before an `extra-high` or
`high` request mounts its DiT fusions, so replay would bypass the requested
kernels. SGLang rejects this combination for models with eligible DiT quality
sites. Models whose request-gated path changes only VAE decode remain allowed
because BCG captures the DiT only.
</Warning>
These fusion families mount under `quality="high"`:
These fusion families mount under both `quality="extra-high"` and
`quality="high"`:
| Fusion | What it folds |
| --- | --- |
| Linear + tanh-GELU | Bias-add and GELU into the GEMM epilogue (cublasLt), removing the `[tokens, 4*dim]` intermediate round trip |
| Wan NVFP4 linear + GELU | Bias-add and GELU fused into Wan's NVFP4 FFN projection output |
| Qwen-Image added-QKV | Added Q/K/V projections fused into joint-buffer production |
| LayerNorm + modulate | `layer_norm(x, weight=(1 + scale), bias=shift)` in place of affine-free LN plus a separate modulate |
| LTX-2 RMSNorm + modulate | `rms_norm(x) * (1 + scale) + shift` in one launch |
| Gate RMSNorm (BF16-native) | `RMSNorm + tanh + mul + add` in one pass |
| HunyuanVideo strided QK RMSNorm | Per-head QK RMSNorm over the packed QKV layout |
| LingBot Video fused RMSNorm | Replaces the handwritten cast, square, mean, rsqrt, and multiply chain with existing Triton RMSNorm kernels |
| SANA-Video BF16-input linear attention | Keeps the first linear-attention GEMM's inputs in BF16 with FP32 accumulation/output; the second GEMM remains FP32 |
| FLUX-family VAE fast paths | Channels-last decode, GroupNorm(+SiLU), upsample, and attention replacements for FLUX.2 and AutoencoderKL-based FLUX.1, Z-Image, and SD3 pipelines |
| Wan VAE RMSNorm + SiLU | Replaces the channel-first RMSNorm/SiLU chain while keeping the decode in `channels_last_3d` |
## Kernel inventory
34 operators are registered in the kernel registry across 38 implementations (some operators carry several backends). Backends are named by provenance, not device: `JIT` compiles under nvcc *and* hipcc, `TRITON` runs on CUDA and ROCm, `CUTE_DSL` needs CUTLASS, `FLYDSL` is ROCm gfx950 only, `AOT` comes from the `sgl_kernel` wheel.
43 operators are registered in the kernel registry across 47 implementations (some operators carry several backends). Backends are named by provenance, not device: `JIT` compiles under nvcc *and* hipcc, `TRITON` runs on CUDA and ROCm, `CUTE_DSL` needs CUTLASS, `FLYDSL` is ROCm gfx950 only, `AOT` comes from the `sgl_kernel` wheel.
### Normalization
@@ -79,6 +100,7 @@ These fusion families mount under `quality="high"`:
| --- | --- | --- | --- |
| `rmsnorm_scale_shift` | Triton | bit-exact | RMSNorm + `* (1 + scale) + shift` (4 kernels) |
| `scale_residual_norm_scale_shift` | Triton / CuTe-DSL / FlyDSL | bit-exact (Triton) | the above plus the preceding `residual + gate * update` |
| `scale_residual_norm_scale_shift_nvfp4` | JIT CUDA | matches the selected NVFP4 producer contract | Qwen residual LayerNorm/modulation + FC1 NVFP4 quantization |
| `layernorm_modulate` | Triton | bit-exact | affine-free LayerNorm + adaLN modulate |
| `qk_head_layernorm` | Triton | bit-exact | per-head LayerNorm on q/k |
| `qk_rmsnorm_native` | Triton | bit-exact | Z-Image per-head QK RMSNorm |
@@ -107,8 +129,11 @@ These fusion families mount under `quality="high"`:
| Operator | Backend | Contract | Replaces |
| --- | --- | --- | --- |
| `fused_inplace_qknorm_rope` | JIT CUDA | one bf16 rounding step vs the split baseline; exact with `round_norm_before_rope=True` | separate QK-norm kernel + RoPE |
| `flux2_qkv_epilogue` | JIT CUDA | bit-exact against its selected BF16 reference chain | FLUX.2 QK RMSNorm + RoPE + joint text/image QKV packing |
| `qwen_qkv_epilogue` | JIT CUDA | bit-exact against its selected BF16 reference chain | Qwen-Image QK RMSNorm + RoPE + joint QKV writes on SM100+ |
| `rope_rotate_half` | Triton | bit-exact | `chunk` → `cat(-x2, x1)` → two muls + add → `cat(tail)`, about 7 kernels per projection |
| `interleaved_rope_fp64` | JIT CUDA | bit-exact | paired SANA-Video Q/K RoPE with fp64 tables, about 14 eager kernels |
| `helios_qk_rope` | JIT CUDA | bit-exact | paired in-place Helios Q/K RoPE with transposed frequency layout |
| `ltx2_qknorm_split_rope` | JIT CUDA | close (validated on B200) | LTX-2 QK-norm + split RoPE |
| `ltx25_decoder_rope` | JIT CUDA | bit-exact | paired LTX-2.5 decoder 3D RoPE from cached compact axis tables |
| `hunyuan_qkv_rope_pack` | Triton | bit-exact | QKV pack and RoPE in one pass |
@@ -137,10 +162,20 @@ Every kernel here only moves values (plus zero fill, plus at most one same-order
| `usp_merge_heads` | JIT CUDA | USP all-to-all output head merge (`permute` + `contiguous`) |
| `pack_qkv_destination_major` | Triton | Ulysses destination-major QKV pack |
| `varlen_pack_qkv`, `varlen_scatter_to_padded` | Triton | varlen gather/scatter around the masked attention path |
| `varlen_pack_segmented_qkv` | Triton | varlen gather from a virtual prefix/main Q/K/V sequence |
| `causal_conv3d_cat_pad` | JIT CUDA / Triton | causal Conv3d `cat` + `pad` |
| `cat_pad_channels_last_3d` | Triton | Wan causal VAE `cat + F.pad + contiguous` (three passes plus cache bookkeeping) in one pass |
| `dup_up3d_add` | Triton | `repeat_interleave + permute().contiguous() + add` |
### Quantized layout producers
These kernels preserve the quantized checkpoint path's selected reference operation. They are not a claim that FP8 or NVFP4 is equivalent to an unquantized BF16 checkpoint.
| Operator | Backend | Replaces |
| --- | --- | --- |
| `flux2_token_cat_fp8` | Triton | FLUX.2 single-block attention/MLP concatenation plus static FP8 quantization |
| `flux2_token_cat_nvfp4` | JIT CUDA | FLUX.2 single-block attention/MLP concatenation plus NVFP4 quantization |
## Coverage by model
Kernels are written against a specific eager chain in a specific model, so coverage is per-model rather than universal.
@@ -148,18 +183,18 @@ Kernels are written against a specific eager chain in a specific model, so cover
| Model | Fused paths |
| --- | --- |
| FLUX.1 | LN+modulate, modulate, residual-gate add, linear+GELU |
| FLUX.2 | LN+modulate, packed SwiGLU, residual-gate add |
| Qwen-Image | linear+GELU, select-0/1 LN modulation |
| FLUX.2 | LN+modulate, packed SwiGLU, gated residual/norm, residual-gate add, QK RMSNorm+RoPE+joint QKV packing, FP8/NVFP4 token-cat producers |
| Qwen-Image | linear+GELU, select-0/1 LN modulation, added-QKV fusion, QK RMSNorm+RoPE+joint QKV writes, residual norm/modulate+NVFP4 producer |
| GLM-Image | LN+modulate, per-head qk LN, residual-gate add, linear+GELU |
| ERNIE-Image | RMSNorm+scale/shift, residual-gated variant, rotate-half RoPE, residual-gate add |
| Z-Image | BF16-native RMSNorm scale / tanh-residual, per-head QK RMSNorm |
| Ideogram 4 | gate RMSNorm, SwiGLU, rotate-half RoPE, modulate, residual-gate add |
| LTX-2 | QK-norm + split RoPE, ada-values split, RMSNorm+modulate, modulate, residual-gate add, linear+GELU |
| LTX-2.5 decoder | paired 3D RoPE with shared axis-table cache |
| HunyuanVideo | QKV+RoPE pack, strided QK RMSNorm, linear+GELU |
| LingBot Video MoE | Fused RMSNorm at `quality=high` |
| HunyuanVideo / Helios | QKV+RoPE pack, strided QK RMSNorm, linear+GELU; Helios also has paired in-place Q/K RoPE |
| LingBot Video MoE | Fused RMSNorm at `quality=extra-high` or `quality=high` |
| Sana | LN+modulate, GLUMB bias+SiLU / bias+GLU, residual-gate add |
| SANA-Video | Packed QKV/KV; paired fp64 interleaved RoPE; LN+modulate, GLUMB bias+SiLU / bias+GLU, and residual-gate add during BCG; BF16-input linear attention at `quality=high` |
| SANA-Video | Packed QKV/KV; paired fp64 interleaved RoPE; LN+modulate, GLUMB bias+SiLU / bias+GLU, and residual-gate add during BCG; BF16-input linear attention at `quality=extra-high` or `quality=high` |
| Sana-WM | bidirectional gated delta-net, fused QK inverse-RMS |
| Wan | temb table slices; VAE cat+pad and DupUp3D add, `channels_last_3d` RMSNorm+SiLU |
| Cosmos3 / Krea2 / MiniMax-H3 | QK-norm + RoPE (Krea2 also CuTe-DSL norm+scale/shift; MiniMax-H3 also indexed modulation) |
@@ -201,9 +236,9 @@ The package `README.md` carries a selection matrix for the cases where several k
## References
- [Performance Optimization](./performance-optimization)
- [Attention Backends](./attention_backends)
- [Quantization](./quantization)
- [Profiling](./profiling)
- [Performance Optimization](/docs/sglang-diffusion/performance-optimization)
- [Attention Backends](/docs/sglang-diffusion/attention_backends)
- [Quantization](/docs/sglang-diffusion/quantization)
- [Profiling](/docs/sglang-diffusion/profiling)
- [`sglang/kernels/ops/diffusion`](https://github.com/sgl-project/sglang/tree/main/python/sglang/kernels/ops/diffusion) — source and selection matrix
- [RFC #29630](https://github.com/sgl-project/sglang/issues/29630) — the unified `sglang.kernels` namespace
docs/docs/sglang-diffusion/performance-optimization.mdx
+46 -28
View File
@@ -12,11 +12,28 @@ The docs use "output-preserving" instead of promising bit-exact "lossless" becau
## Start Here
1. Pick a serving or generation mode from [Deployment and Performance Modes](./deployment_cookbook). `--performance-mode auto` is the default; use `speed` when the model fits in GPU memory and latency matters most, `memory` when GPU memory is the bottleneck, and `manual` when every performance flag should be explicit.
2. Choose the right attention backend from [Attention Backends](./attention_backends).
3. Use [Sequence Parallelism](./ring_sp_performance) only when the model and video shape benefit from sequence splitting.
4. Use [Inference Batching](./dynamic_batching) for concurrent compatible requests during serving.
5. Use [Profiling](./profiling) before changing several levers at once.
1. Pick a serving or generation mode from [Deployment and Performance Modes](/docs/sglang-diffusion/deployment_cookbook). `--performance-mode auto` is the default; use `speed` when the model fits in GPU memory and latency matters most, `memory` when GPU memory is the bottleneck, and `manual` when every performance flag should be explicit.
2. Choose the right attention backend from [Attention Backends](/docs/sglang-diffusion/attention_backends).
3. Use [Sequence Parallelism](/docs/sglang-diffusion/ring_sp_performance) only when the model and video shape benefit from sequence splitting.
4. Use [Inference Batching](/docs/sglang-diffusion/dynamic_batching) for concurrent compatible requests during serving.
5. Use [Profiling](/docs/sglang-diffusion/profiling) before changing several levers at once.
## Choose a request quality tier
`--quality` is cumulative: a broader tier never drops an optimization from a
stricter tier.
| Tier | Optimization boundary |
| --- | --- |
| `lossless` (default) | The selected deployment's reference execution plus all unconditional bit-exact replacements |
| `extra-high` | Everything in `lossless`, plus only request-gated DiT/VAE kernel fusions; the tier does not itself enable sparse, caching, or other approximate paths |
| `high` | Everything in `extra-high`, plus model-owned high-only paths such as an audited Cache-DiT policy or lower-precision VAE decode |
Use `extra-high` when you want to isolate fusion wins from approximate
acceleration. A tier may be a no-op when the active model has no eligible path.
Separately configured quantization, attention, or caching options still apply.
See [Fused Kernels](/docs/sglang-diffusion/fused_kernels) for the current
request-gated families and their numerical contracts.
## Output-Preserving / Lossless-Style Levers
@@ -39,42 +56,42 @@ These settings should preserve model behavior while changing residency, parallel
<tr>
<td style={{padding: "9px 12px", fontWeight: 500}}><code>--performance-mode</code></td>
<td style={{padding: "9px 12px"}}>You want a safe preset for speed or memory without overriding explicit flags.</td>
<td style={{padding: "9px 12px"}}><a href="./deployment_cookbook">Deployment and Performance Modes</a></td>
<td style={{padding: "9px 12px"}}><a href="/docs/sglang-diffusion/deployment_cookbook">Deployment and Performance Modes</a></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500}}>Breakable CUDA graph</td>
<td style={{padding: "9px 12px"}}>A supported pipeline serves a fixed set of shapes and eager execution is launch-bound.</td>
<td style={{padding: "9px 12px"}}><a href="./api/cli">CLI reference</a></td>
<td style={{padding: "9px 12px"}}><a href="/docs/sglang-diffusion/api/cli">CLI reference</a></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500}}>Offload, FSDP, CFG parallelism</td>
<td style={{padding: "9px 12px"}}>GPU memory, multi-GPU residency, or CFG branch splitting is the main bottleneck.</td>
<td style={{padding: "9px 12px"}}><a href="./deployment_cookbook">Deployment and Performance Modes</a></td>
<td style={{padding: "9px 12px"}}><a href="/docs/sglang-diffusion/deployment_cookbook">Deployment and Performance Modes</a></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500}}>Sequence parallelism</td>
<td style={{padding: "9px 12px"}}>Long image/video sequences need sequence-level parallelism.</td>
<td style={{padding: "9px 12px"}}><a href="./ring_sp_performance">Sequence Parallelism</a></td>
<td style={{padding: "9px 12px"}}><a href="/docs/sglang-diffusion/ring_sp_performance">Sequence Parallelism</a></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500}}><code>--encoder-parallel</code></td>
<td style={{padding: "9px 12px"}}>Text/image encoding is a visible share of the request and the DiT replica sits idle during it.</td>
<td style={{padding: "9px 12px"}}><a href="./encoder_parallel">Encoder Parallelism</a></td>
<td style={{padding: "9px 12px"}}><a href="/docs/sglang-diffusion/encoder_parallel">Encoder Parallelism</a></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500}}>Attention backend</td>
<td style={{padding: "9px 12px"}}>Kernel choice dominates DiT latency or memory.</td>
<td style={{padding: "9px 12px"}}><a href="./attention_backends">Attention Backends</a></td>
<td style={{padding: "9px 12px"}}><a href="/docs/sglang-diffusion/attention_backends">Attention Backends</a></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500}}>Fused kernels</td>
<td style={{padding: "9px 12px"}}>You want to know which elementwise chains are already fused, or to opt into the request-gated set.</td>
<td style={{padding: "9px 12px"}}><a href="./fused_kernels">Fused Kernels</a></td>
<td style={{padding: "9px 12px"}}><a href="/docs/sglang-diffusion/fused_kernels">Fused Kernels</a></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500}}>Dynamic batching</td>
<td style={{padding: "9px 12px"}}>Serving many compatible requests concurrently.</td>
<td style={{padding: "9px 12px"}}><a href="./dynamic_batching">Inference Batching</a></td>
<td style={{padding: "9px 12px"}}><a href="/docs/sglang-diffusion/dynamic_batching">Inference Batching</a></td>
</tr>
</tbody>
</table>
@@ -100,22 +117,22 @@ These techniques can change the denoising path, numerical representation, or gen
<tr>
<td style={{padding: "9px 12px", fontWeight: 500}}>Cache-DiT</td>
<td style={{padding: "9px 12px"}}>Skips selected DiT block or step computation based on cache decisions.</td>
<td style={{padding: "9px 12px"}}><a href="./cache_dit">Cache-DiT</a></td>
<td style={{padding: "9px 12px"}}><a href="/docs/sglang-diffusion/cache_dit">Cache-DiT</a></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500}}>TeaCache</td>
<td style={{padding: "9px 12px"}}>Reuses residuals when consecutive denoising steps are similar enough.</td>
<td style={{padding: "9px 12px"}}><a href="./teacache">TeaCache</a></td>
<td style={{padding: "9px 12px"}}><a href="/docs/sglang-diffusion/teacache">TeaCache</a></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500}}>Progressive resolution</td>
<td style={{padding: "9px 12px"}}>Runs early denoising at lower latent resolution for supported pipelines.</td>
<td style={{padding: "9px 12px"}}><a href="./progressive_resolution">Progressive Resolution Generation</a></td>
<td style={{padding: "9px 12px"}}><a href="/docs/sglang-diffusion/progressive_resolution">Progressive Resolution Generation</a></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500}}>Quantization</td>
<td style={{padding: "9px 12px"}}>Uses lower-precision transformer weights or activations.</td>
<td style={{padding: "9px 12px"}}><a href="./quantization">Quantization</a></td>
<td style={{padding: "9px 12px"}}><a href="/docs/sglang-diffusion/quantization">Quantization</a></td>
</tr>
</tbody>
</table>
@@ -124,20 +141,21 @@ These techniques can change the denoising path, numerical representation, or gen
1. Establish a baseline with the target model, resolution, frame count, step count, and GPU type.
2. Select `--performance-mode` and explicit residency or parallelism flags.
3. Compare breakable CUDA graph against eager execution for supported fixed-shape pipelines. Pass every served resolution to `--warmup-resolutions` and confirm capture in the server log.
4. Tune attention backend and batching for the deployment pattern.
5. Profile if the bottleneck is unclear.
6. Add caching, progressive resolution, or quantization only after comparing output quality against your acceptance target.
3. Compare `quality=lossless` with `quality=extra-high` to isolate the request-gated fusion set.
4. Compare breakable CUDA graph against eager execution for supported fixed-shape pipelines. Pass every served resolution to `--warmup-resolutions` and confirm capture in the server log. Models with request-gated DiT fusions cannot combine those fusions with a graph captured from the lossless branches.
5. Tune attention backend and batching for the deployment pattern.
6. Profile if the bottleneck is unclear.
7. Add `quality=high`, caching, progressive resolution, or quantization only after comparing output quality against your acceptance target.
## Diagnostics
[Profiling](./profiling) is not an optimization technique by itself. It belongs in the performance workflow because it tells you which stage, kernel, or denoising step is worth optimizing before you change multiple levers.
[Profiling](/docs/sglang-diffusion/profiling) is not an optimization technique by itself. It belongs in the performance workflow because it tells you which stage, kernel, or denoising step is worth optimizing before you change multiple levers.
## References
- [Deployment and Performance Modes](./deployment_cookbook)
- [Attention Backends](./attention_backends)
- [Fused Kernels](./fused_kernels)
- [Sequence Parallelism](./ring_sp_performance)
- [Caching Strategies](./caching-acceleration)
- [Profiling](./profiling)
- [Deployment and Performance Modes](/docs/sglang-diffusion/deployment_cookbook)
- [Attention Backends](/docs/sglang-diffusion/attention_backends)
- [Fused Kernels](/docs/sglang-diffusion/fused_kernels)
- [Sequence Parallelism](/docs/sglang-diffusion/ring_sp_performance)
- [Caching Strategies](/docs/sglang-diffusion/caching-acceleration)
- [Profiling](/docs/sglang-diffusion/profiling)
docs/src/snippets/configs/MiniMaxAI/minimax-h3.jsx
+7 -2
View File
@@ -486,7 +486,7 @@ return {
title: "Quality",
scope: "request",
docsHref: "/docs/sglang-diffusion/cache_dit",
description: "Reference execution or the audited Cache-DiT acceleration preset.",
description: "Cumulative reference, fusion-only, or audited Cache-DiT execution.",
quality: "Sampling policy",
learnMore: "#choose-the-quality-level",
default: "lossless",
@@ -497,6 +497,11 @@ return {
recommended: true,
description: "Reference-exact denoising without Cache-DiT approximation.",
},
{
id: "extra-high",
label: "Extra high",
description: "Includes fusion-only request paths but not Cache-DiT; MiniMax-H3 currently follows its lossless denoise path at this tier.",
},
{
id: "high",
label: "Audited high",
@@ -669,7 +674,7 @@ return {
|| (s.execution === "bcg" && ["b200", "h200"].includes(s.hw) && s.weights === "ref2va");
const serveVerified = topologyVerified && encoderVerified && attentionVerified
&& precisionVerified && executionVerified;
const requestVerified = topologyVerified && (s.quality === "lossless"
const requestVerified = topologyVerified && (["lossless", "extra-high"].includes(s.quality)
|| (s.quality === "high" && highAudited && s.execution === "eager"));
const topologyParts = [];