[SKILLS] Tiny upgrade diffusion skills (#24273)
This commit is contained in:
Xiaoyu Zhang
@@ -34,6 +34,13 @@ First use [../sglang-diffusion-benchmark-profile/SKILL.md](../sglang-diffusion-b
|
||||
- rule out existing mainline fast paths
|
||||
- prove the run stayed on the native SGLang diffusion backend, not a diffusers fallback
|
||||
|
||||
Before opening AKO, also read
|
||||
[../sglang-diffusion-benchmark-profile/existing-fast-paths.md](../sglang-diffusion-benchmark-profile/existing-fast-paths.md).
|
||||
It records current mainline fusions plus the open PR watchlist for diffusion
|
||||
kernel, VAE, attention, cache, and scheduling work. If an open PR already covers
|
||||
the same shape family, use it as prior art or decide whether to rebase/extend it
|
||||
instead of starting a duplicate kernel.
|
||||
|
||||
If a future specialized optimization skill matches the kernel family better than AKO4ALL, hand off there instead. The diagnosis contract stays the same.
|
||||
|
||||
## Mandatory AKO4ALL Preflight
|
||||
@@ -58,6 +65,8 @@ By default it uses the existing `origin` URL, or `AKO4ALL_URL` if you need to ov
|
||||
- Identify the exact kernel entry point and runtime call sites in `sglang`.
|
||||
- Record the target shapes, dtypes, model families, and whether the kernel is on a hot path.
|
||||
- Reuse existing unit tests and benchmark entry points when they already exist.
|
||||
- Record whether the hotspot overlaps an open PR from `existing-fast-paths.md`;
|
||||
if it does, note the PR number in the AKO context and final PR artifacts.
|
||||
|
||||
### 2. Bootstrap the AKO Harness
|
||||
|
||||
|
||||
+3
-2
@@ -42,8 +42,8 @@ If any benchmark, perf-dump, or `torch.profiler` command prints one of those sig
|
||||
|
||||
## Main Reference
|
||||
|
||||
- [benchmark-and-profile.md](benchmark-and-profile.md) — canonical denoise benchmark, perf dump, and `torch.profiler` workflow; uses the checked-in nightly-aligned presets, plus `LTX-2`, `LTX-2.3` one-stage, and `LTX-2.3` two-stage benchmark recipes
|
||||
- [existing-fast-paths.md](existing-fast-paths.md) — map bottlenecks to existing fused kernels, packed QKV paths, fused `QK norm + RoPE`, and distributed overlap patterns before proposing new code
|
||||
- [benchmark-and-profile.md](benchmark-and-profile.md) — canonical denoise benchmark, perf dump, and `torch.profiler` workflow; uses checked-in nightly-aligned presets plus skill-only stress recipes such as `LTX-2.3` one-stage/two-stage, HunyuanVideo, MOVA, and Helios
|
||||
- [existing-fast-paths.md](existing-fast-paths.md) — map bottlenecks to existing fused kernels, packed QKV paths, fused `QK norm + RoPE`, distributed overlap patterns, and open optimization PRs before proposing new code
|
||||
- [scripts/diffusion_skill_env.py](scripts/diffusion_skill_env.py) — preflight helper: repo root discovery via `sglang.__file__`, write-access probe, benchmark/profile output directories, idle GPU selection
|
||||
- [scripts/bench_diffusion_denoise.py](scripts/bench_diffusion_denoise.py) — end-to-end denoise benchmark preset runner via `sglang generate`; pins `--backend=sglang`, supports `--no-torch-compile`, and saves perf dumps by label for `compare_perf.py`
|
||||
|
||||
@@ -52,6 +52,7 @@ If any benchmark, perf-dump, or `torch.profiler` command prints one of those sig
|
||||
Before calling a diffusion hotspot "new", first classify it with `existing-fast-paths.md`.
|
||||
|
||||
Always rule out these existing families first:
|
||||
- HunyuanVideo VAE GroupNorm+SiLU
|
||||
- Z-Image residual-form modulation
|
||||
- fused diffusion `QK norm + RoPE`
|
||||
- NVFP4 / Nunchaku packed QKV
|
||||
|
||||
+51
-17
@@ -97,9 +97,11 @@ Treat `"$BENCH_PY"` as the source of truth for preset order.
|
||||
|
||||
Nightly diffusion comparison is server/API based (`sglang serve` plus requests).
|
||||
This skill stays on `sglang generate` for local benchmarking and profiling, but
|
||||
the first 9 presets in `bench_diffusion_denoise.py` are aligned to nightly on
|
||||
model, prompt, negative prompt, reference image, size, frames, fps, seed, GPU
|
||||
count, and any explicitly overridden sampling or parallelism flags.
|
||||
the nightly-aligned presets in `bench_diffusion_denoise.py` mirror
|
||||
`scripts/ci/utils/diffusion/comparison_configs.json` on model, task, prompt,
|
||||
reference image, size, frames, seed, GPU count, serve args, and the request
|
||||
defaults used by `run_comparison.py` when a case omits steps or guidance.
|
||||
When in doubt, re-check that JSON before trusting this reference.
|
||||
|
||||
List the current preset order:
|
||||
|
||||
@@ -135,6 +137,15 @@ PYTHONPATH=python python3 "$BENCH_PY" \
|
||||
--output-dir "${BENCH_DIR}"
|
||||
```
|
||||
|
||||
Run the nightly-aligned `LTX-2.3` TI2V two-stage preset:
|
||||
|
||||
```bash
|
||||
PYTHONPATH=python python3 "$BENCH_PY" \
|
||||
--model ltx23-ti2v-two-stage \
|
||||
--label baseline \
|
||||
--output-dir "${BENCH_DIR}"
|
||||
```
|
||||
|
||||
Run the `LTX-2.3` two-stage skill preset:
|
||||
|
||||
```bash
|
||||
@@ -164,10 +175,11 @@ Nightly-aligned presets come first; skill-only presets stay available after them
|
||||
| `zimage` | `Tongyi-MAI/Z-Image-Turbo` | Yes: `zimage_turbo_t2i_1024` | Aligned to nightly prompt and guidance 4.0 |
|
||||
| `wan-t2v` | `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | Yes: `wan22_t2v_a14b_720p` | Aligned to nightly CFG-parallel 4-GPU launch |
|
||||
| `wan-ti2v` | `Wan-AI/Wan2.2-TI2V-5B-Diffusers` | Yes: `wan22_ti2v_5b_720p` | Uses the nightly cat image and motion prompt |
|
||||
| `ltx2` | `Lightricks/LTX-2` | Yes: `ltx2_twostage_t2v` | Uses `LTX2TwoStagePipeline`; nightly-aligned prompt, negative prompt, 1536x1024, 121 frames, fps 24, seed 1234 |
|
||||
| `ltx2` | `Lightricks/LTX-2` | Yes: `ltx2_twostage_t2v` | Uses `LTX2TwoStagePipeline`; 2 GPUs, CFG parallel, 768x512, 121 frames, seed 42 |
|
||||
| `ltx23-ti2v-two-stage` | `Lightricks/LTX-2.3` | Yes: `ltx2.3_twostage_ti2v_2gpus` | Uses the nightly cat image, motion prompt, `LTX2TwoStagePipeline`, 2 GPUs, 768x512, 121 frames, seed 42 |
|
||||
| `wan-i2v` | `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | Yes: `wan22_i2v_a14b_720p` | Aligned to nightly CFG-parallel 4-GPU launch |
|
||||
| `ltx23-one-stage` | `Lightricks/LTX-2.3` | No | Skill-only extra preset for the native `LTX-2.3` one-stage baseline; 2 GPUs, 768x512, 121 frames, fps 24, 30 steps, guidance 3.0, seed 1234 |
|
||||
| `ltx23-two-stage` | `Lightricks/LTX-2.3` | No | Skill-only extra preset for the native `LTX-2.3` two-stage path; uses `LTX2TwoStagePipeline`, 2 GPUs, 1536x1024, 121 frames, fps 24, 30 steps, guidance 3.0, seed 1234 |
|
||||
| `ltx23-two-stage` | `Lightricks/LTX-2.3` | No | Skill-only high-resolution stress preset for the native `LTX-2.3` two-stage path; uses `LTX2TwoStagePipeline`, 2 GPUs, 1536x1024, 121 frames, fps 24, 30 steps, guidance 3.0, seed 1234 |
|
||||
| `hunyuanvideo` | `hunyuanvideo-community/HunyuanVideo` | No | Skill-only extra preset |
|
||||
| `mova-720p` | `OpenMOSS-Team/MOVA-720p` | No | Skill-only extra preset |
|
||||
| `helios` | `BestWishYsh/Helios-Base` | No | Skill-only extra preset |
|
||||
@@ -184,17 +196,34 @@ and **best latency tuning**:
|
||||
sglang generate \
|
||||
--model-path=Lightricks/LTX-2 \
|
||||
--pipeline-class-name=LTX2TwoStagePipeline \
|
||||
--prompt="A beautiful sunset over the ocean" \
|
||||
--negative-prompt="shaky, glitchy, low quality, worst quality, deformed, distorted, disfigured, motion smear, motion artifacts, fused fingers, bad anatomy, weird hand, ugly, transition, static." \
|
||||
--width=1536 --height=1024 \
|
||||
--num-frames=121 --fps=24 \
|
||||
--seed=1234 --num-gpus=1 \
|
||||
--prompt="A cat and a dog baking a cake together in a kitchen." \
|
||||
--width=768 --height=512 \
|
||||
--num-frames=121 \
|
||||
--num-inference-steps=50 --guidance-scale=4.0 \
|
||||
--seed=42 --num-gpus=2 --enable-cfg-parallel \
|
||||
--save-output --enable-torch-compile --warmup
|
||||
```
|
||||
|
||||
`LTX2TwoStagePipeline` is a native path. The spatial upsampler and distilled
|
||||
LoRA are auto-resolved from the same model snapshot unless you override them.
|
||||
|
||||
### Manual command example: LTX-2.3 TI2V Two-Stage
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
--model-path=Lightricks/LTX-2.3 \
|
||||
--pipeline-class-name=LTX2TwoStagePipeline \
|
||||
--prompt="The cat starts walking slowly towards the camera." \
|
||||
--image-path="${ASSET_DIR}/cat.png" \
|
||||
--width=768 --height=512 \
|
||||
--num-frames=121 \
|
||||
--num-inference-steps=50 --guidance-scale=4.0 \
|
||||
--seed=42 --num-gpus=2 \
|
||||
--save-output --enable-torch-compile --warmup
|
||||
```
|
||||
|
||||
This matches the nightly comparison case `ltx2.3_twostage_ti2v_2gpus`.
|
||||
|
||||
### Manual command example: LTX-2.3 One-Stage
|
||||
|
||||
```bash
|
||||
@@ -212,7 +241,7 @@ sglang generate \
|
||||
Use this when you want the native `LTX2Pipeline` baseline for `LTX-2.3` at the
|
||||
validated one-stage resolution.
|
||||
|
||||
### Manual command example: LTX-2.3 Two-Stage
|
||||
### Manual command example: LTX-2.3 Two-Stage High-Resolution Stress
|
||||
|
||||
```bash
|
||||
sglang generate \
|
||||
@@ -227,8 +256,8 @@ sglang generate \
|
||||
--save-output --enable-torch-compile --warmup
|
||||
```
|
||||
|
||||
This matches the `ltx23-two-stage` skill preset and is a good benchmark target
|
||||
for the native `LTX-2.3` two-stage path.
|
||||
This matches the skill-only `ltx23-two-stage` preset. Use it as a
|
||||
high-resolution stress target, not as a nightly comparison case.
|
||||
|
||||
### Manual command example: Wan2.2-I2V-A14B 720P
|
||||
|
||||
@@ -295,7 +324,7 @@ By default SGLang profiles the denoising stage. The default sampling window is
|
||||
5 profiled timesteps after warmup.
|
||||
|
||||
```bash
|
||||
SGLANG_TORCH_PROFILER_DIR="${PROFILE_DIR}/torch" \
|
||||
SGLANG_DIFFUSION_TORCH_PROFILER_DIR="${PROFILE_DIR}/torch" \
|
||||
sglang generate \
|
||||
--model-path=black-forest-labs/FLUX.1-dev \
|
||||
--prompt="A futuristic cyberpunk city at night" \
|
||||
@@ -308,8 +337,9 @@ Use `--profile-all-stages` only when you really need text encoder, VAE, or
|
||||
other non-denoise stages too.
|
||||
|
||||
The generated trace path is printed in the console and also lands under
|
||||
`./logs/` or `SGLANG_TORCH_PROFILER_DIR`. Open it in Perfetto if you want a
|
||||
timeline view:
|
||||
`SGLANG_DIFFUSION_TORCH_PROFILER_DIR`. The diffusion profiler falls back to
|
||||
`SGLANG_TORCH_PROFILER_DIR` and then `./logs` when the diffusion-specific env
|
||||
var is unset. Open the trace in Perfetto if you want a timeline view:
|
||||
- https://ui.perfetto.dev/
|
||||
|
||||
### 3. Rank the hot CUDA kernels
|
||||
@@ -323,7 +353,11 @@ import gzip
|
||||
import json
|
||||
import os
|
||||
|
||||
log_dir = os.environ.get("SGLANG_TORCH_PROFILER_DIR", "./logs")
|
||||
log_dir = (
|
||||
os.environ.get("SGLANG_DIFFUSION_TORCH_PROFILER_DIR")
|
||||
or os.environ.get("SGLANG_TORCH_PROFILER_DIR")
|
||||
or "./logs"
|
||||
)
|
||||
trace_path = sorted(
|
||||
glob.glob(f"{log_dir}/*.trace.json.gz"),
|
||||
key=os.path.getmtime,
|
||||
|
||||
+45
@@ -9,10 +9,14 @@ configuration first before handing the problem to a specialized kernel-optimizat
|
||||
- `python/sglang/multimodal_gen/runtime/layers/elementwise.py`
|
||||
- `python/sglang/multimodal_gen/runtime/layers/rotary_embedding/utils.py`
|
||||
- `python/sglang/jit_kernel/diffusion/triton/scale_shift.py`
|
||||
- `python/sglang/jit_kernel/diffusion/group_norm_silu.py`
|
||||
- `python/sglang/jit_kernel/diffusion/triton/group_norm_silu.py`
|
||||
- `python/sglang/jit_kernel/diffusion/triton/norm.py`
|
||||
- `python/sglang/jit_kernel/diffusion/triton/rmsnorm_onepass.py`
|
||||
- `python/sglang/jit_kernel/diffusion/triton/rotary.py`
|
||||
- `python/sglang/jit_kernel/diffusion/cutedsl/scale_residual_norm_scale_shift.py`
|
||||
- `python/sglang/jit_kernel/tests/diffusion/test_group_norm_silu.py`
|
||||
- `python/sglang/jit_kernel/benchmark/diffusion/bench_group_norm_silu.py`
|
||||
- `python/sglang/jit_kernel/norm.py`
|
||||
- `python/sglang/multimodal_gen/runtime/platforms/cuda.py`
|
||||
- `python/sglang/multimodal_gen/runtime/layers/attention/selector.py`
|
||||
@@ -68,6 +72,15 @@ configuration first before handing the problem to a specialized kernel-optimizat
|
||||
- NPU fallback: `npu_fallback.apply_rotary_embedding_native`.
|
||||
- Validation: `python/sglang/jit_kernel/tests/test_rope.py`.
|
||||
|
||||
7. HunyuanVideo VAE GroupNorm + SiLU fusion
|
||||
- Kernel: `triton_group_norm_silu`
|
||||
- Locations: `diffusion/group_norm_silu.py`, `triton/group_norm_silu.py`, `runtime/models/vaes/hunyuanvae.py`
|
||||
- Use case: `activation(group_norm(x))` when the activation is non-inplace `nn.SiLU` and the GroupNorm is affine.
|
||||
- Enablement: mainline uses `apply_group_norm_silu(...)` in HunyuanVideo VAE paths by default; there is no env toggle. The wrapper dispatches to Triton only when guards pass.
|
||||
- Constraints: CUDA inference path only; no grad, `x.requires_grad == False`, `nn.GroupNorm`, `nn.SiLU(inplace=False)`, affine norm with weight and bias. Unsupported cases fall back to native `activation(norm(x))`.
|
||||
- Validation: `python/sglang/jit_kernel/tests/diffusion/test_group_norm_silu.py`.
|
||||
- Microbench: `python/sglang/jit_kernel/benchmark/diffusion/bench_group_norm_silu.py`.
|
||||
|
||||
**Faster CUDA Kernel Usage Points**
|
||||
|
||||
1. sgl-kernel RMSNorm and fused add RMSNorm
|
||||
@@ -112,6 +125,7 @@ configuration first before handing the problem to a specialized kernel-optimizat
|
||||
- Supported head dims: `64, 128, 256`.
|
||||
- Behavior: `apply_qk_norm_rope` prefers the fused JIT kernel when all guards pass; otherwise it falls back to `apply_qk_norm(...)` plus `apply_flashinfer_rope_qk_inplace(...)`.
|
||||
- Validation: `python/sglang/jit_kernel/tests/diffusion/test_qknorm_rope.py`.
|
||||
- Watchlist: PR #24025 adds LTX2-specific QK norm fusion work. Until merged, treat LTX2 traces that miss the generic fused path as an enablement/shape-guard issue first, not proof that the PR path exists locally.
|
||||
|
||||
**Nunchaku Fused GELU MLP**
|
||||
|
||||
@@ -132,6 +146,7 @@ configuration first before handing the problem to a specialized kernel-optimizat
|
||||
- AdaLN modulation: `LayerNormScaleShift`, `RMSNormScaleShift`, `ScaleResidual*` in `layernorm.py`.
|
||||
- Qwen-Image gating: `fuse_scale_shift_gate_select01_kernel` in `qwen_image.py`.
|
||||
- Z-Image residual-form modulation: `fused_norm_tanh_mul_add` and `fused_norm_tanh_mul_add_norm_scale` in `zimage.py`.
|
||||
- HunyuanVideo VAE GroupNorm+SiLU: `apply_group_norm_silu` in `hunyuanvae.py`; default-eligible when wrapper guards pass.
|
||||
- QK norm: `apply_qk_norm` used in `flux.py`, `flux_2.py`, `qwen_image.py`, `zimage.py`, `wanvideo.py`, `ltx_2.py`, `hunyuanvideo.py`.
|
||||
- QK norm + RoPE: `apply_qk_norm_rope` in `layernorm.py`; use this path when the model wants fused attention prep instead of separate QK norm and RoPE calls.
|
||||
- Nunchaku fused GELU MLP: `_fused_gelu_mlp` in `flux.py` for quantized FLUX-family checkpoints.
|
||||
@@ -146,6 +161,36 @@ configuration first before handing the problem to a specialized kernel-optimizat
|
||||
- Dual-stream diffusion models: `use_dual_stream = True` in models such as `hunyuan3d.py` is an existing overlap family.
|
||||
- Workflow rule: if a hotspot is communication-heavy, rule out these in-repo overlap families before proposing a brand new overlap design.
|
||||
|
||||
**Open PR Watchlist**
|
||||
|
||||
As of 2026-05-02, these SGLang PRs were still open. Use them as upstream
|
||||
direction and prior art, not as current-main behavior. Re-check the PR state
|
||||
before relying on any file path or flag.
|
||||
|
||||
- Norm, modulation, and packed projection fusions:
|
||||
- #24025 LTX2 QK norm fusion.
|
||||
- #24059 Helios fused norm modulation.
|
||||
- #24117 Z-Image packed QKV.
|
||||
- #19488 Wan cross-block elementwise fusion.
|
||||
- #19249 Z-Image `scale residual norm scale shift` plus `add gate norm` fusion.
|
||||
- #18897 dual norm fusion for FLUX-family paths (draft).
|
||||
- #20429 Qwen-Image layernorm and `fuse_scale_shift_gate_select01` work.
|
||||
- #20530 MOVA fused RMSNorm + interleaved RoPE.
|
||||
- VAE and decode-side acceleration:
|
||||
- #22531 LTX2 parallel VAE support and #20927 batched tiled VAE decode (draft).
|
||||
- Attention, communication, and runtime scheduling:
|
||||
- #22805 FLUX.2 packed QKV for all-to-all.
|
||||
- #21742 hybrid attention schedule.
|
||||
- #24053 USP attention with replicated prefixes.
|
||||
- #18764 dynamic batching v0.
|
||||
- #24200 disaggregated diffusion v2.
|
||||
- Cache and CUDA graph:
|
||||
- #21613 TeaCache refactor.
|
||||
- #24227 WanVideo TeaCache skipping fix.
|
||||
- #20447 TeaCache support for GLM-Image, Qwen-Image, and related models.
|
||||
- #19516 Qwen-Image CUDA Graph.
|
||||
- #21912 Z-Image Turbo FP8 full quantization and CUDA Graph.
|
||||
|
||||
**Constraints and Fallbacks**
|
||||
- `scale_shift` Triton requires CUDA + contiguous `x`. NPU swaps to native.
|
||||
- CuTe DSL fused norms require `D % 256 == 0` and `D <= 8192`.
|
||||
|
||||
+54
-29
@@ -12,7 +12,7 @@ Usage:
|
||||
# Tag the run for later compare_perf.py usage
|
||||
python3 python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py --model flux --label tuned
|
||||
|
||||
# All 14 preset models
|
||||
# All 15 preset models
|
||||
python3 python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py --all
|
||||
|
||||
# Show preset order, model path, and nightly mapping
|
||||
@@ -58,10 +58,13 @@ DIFFUSERS_FALLBACK_SIGNALS = (
|
||||
"using diffusers backend",
|
||||
"loaded diffusers pipeline",
|
||||
)
|
||||
CATALOG_TABLE_WIDTH = 105
|
||||
RESULTS_TABLE_WIDTH = 105
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Model configs — kept in exact sync with benchmark-and-profile.md
|
||||
# Nightly-aligned presets come first, followed by skill-only extras.
|
||||
# Nightly-aligned presets mirror scripts/ci/utils/diffusion/comparison_configs.json
|
||||
# first, followed by skill-only extras.
|
||||
# Each entry produces the same `sglang generate` command as shown in that doc.
|
||||
# ---------------------------------------------------------------------------
|
||||
MODELS = {
|
||||
@@ -166,19 +169,36 @@ MODELS = {
|
||||
"ltx2": {
|
||||
"nightly_case_id": "ltx2_twostage_t2v",
|
||||
"path": "Lightricks/LTX-2",
|
||||
"prompt": "A beautiful sunset over the ocean",
|
||||
"negative_prompt": "shaky, glitchy, low quality, worst quality, deformed, distorted, disfigured, motion smear, motion artifacts, fused fingers, bad anatomy, weird hand, ugly, transition, static.",
|
||||
"seed": 1234,
|
||||
"prompt": "A cat and a dog baking a cake together in a kitchen.",
|
||||
"extra_args": [
|
||||
"--pipeline-class-name=LTX2TwoStagePipeline",
|
||||
"--width=1536",
|
||||
"--height=1024",
|
||||
"--width=768",
|
||||
"--height=512",
|
||||
"--num-frames=121",
|
||||
"--fps=24",
|
||||
"--num-gpus=1",
|
||||
"--num-inference-steps=50",
|
||||
"--guidance-scale=4.0",
|
||||
"--num-gpus=2",
|
||||
"--enable-cfg-parallel",
|
||||
],
|
||||
},
|
||||
# 9. Nightly: wan22_i2v_a14b_720p
|
||||
# 9. Nightly: ltx2.3_twostage_ti2v_2gpus
|
||||
# Requires: <repo>/inputs/diffusion_benchmark/figs/cat.png
|
||||
"ltx23-ti2v-two-stage": {
|
||||
"nightly_case_id": "ltx2.3_twostage_ti2v_2gpus",
|
||||
"path": "Lightricks/LTX-2.3",
|
||||
"prompt": "The cat starts walking slowly towards the camera.",
|
||||
"image_path": str(ASSET_DIR / "cat.png"),
|
||||
"extra_args": [
|
||||
"--pipeline-class-name=LTX2TwoStagePipeline",
|
||||
"--width=768",
|
||||
"--height=512",
|
||||
"--num-frames=121",
|
||||
"--num-inference-steps=50",
|
||||
"--guidance-scale=4.0",
|
||||
"--num-gpus=2",
|
||||
],
|
||||
},
|
||||
# 10. Nightly: wan22_i2v_a14b_720p
|
||||
# Requires: <repo>/inputs/diffusion_benchmark/figs/cat.png
|
||||
"wan-i2v": {
|
||||
"nightly_case_id": "wan22_i2v_a14b_720p",
|
||||
@@ -197,7 +217,7 @@ MODELS = {
|
||||
"--pin-cpu-memory",
|
||||
],
|
||||
},
|
||||
# 10. Skill-only extra preset
|
||||
# 11. Skill-only extra preset
|
||||
"ltx23-one-stage": {
|
||||
"path": "Lightricks/LTX-2.3",
|
||||
"prompt": "A beautiful sunset over the ocean",
|
||||
@@ -213,7 +233,7 @@ MODELS = {
|
||||
"--num-gpus=2",
|
||||
],
|
||||
},
|
||||
# 11. Skill-only extra preset
|
||||
# 12. Skill-only extra preset
|
||||
"ltx23-two-stage": {
|
||||
"path": "Lightricks/LTX-2.3",
|
||||
"prompt": "A beautiful sunset over the ocean",
|
||||
@@ -230,7 +250,7 @@ MODELS = {
|
||||
"--num-gpus=2",
|
||||
],
|
||||
},
|
||||
# 12. Skill-only extra preset
|
||||
# 13. Skill-only extra preset
|
||||
"hunyuanvideo": {
|
||||
"path": "hunyuanvideo-community/HunyuanVideo",
|
||||
"prompt": "A cat and a dog baking a cake together in a kitchen. The cat is carefully measuring flour, while the dog is stirring the batter with a wooden spoon. The kitchen is cozy, with sunlight streaming through the window.",
|
||||
@@ -243,7 +263,7 @@ MODELS = {
|
||||
"--num-inference-steps=30",
|
||||
],
|
||||
},
|
||||
# 13. Skill-only extra preset
|
||||
# 14. Skill-only extra preset
|
||||
# Requires: <repo>/inputs/diffusion_benchmark/figs/mova_single_person.jpg
|
||||
"mova-720p": {
|
||||
"path": "OpenMOSS-Team/MOVA-720p",
|
||||
@@ -259,7 +279,7 @@ MODELS = {
|
||||
"--num-inference-steps=2",
|
||||
],
|
||||
},
|
||||
# 14. Skill-only extra preset
|
||||
# 15. Skill-only extra preset
|
||||
"helios": {
|
||||
"path": "BestWishYsh/Helios-Base",
|
||||
"prompt": "A curious raccoon",
|
||||
@@ -285,7 +305,12 @@ def required_gpus_for_model(model_key: str) -> int:
|
||||
return 4
|
||||
if model_key == "mova-720p":
|
||||
return 4
|
||||
if model_key in {"ltx23-one-stage", "ltx23-two-stage"}:
|
||||
if model_key in {
|
||||
"ltx2",
|
||||
"ltx23-ti2v-two-stage",
|
||||
"ltx23-one-stage",
|
||||
"ltx23-two-stage",
|
||||
}:
|
||||
return 2
|
||||
return 1
|
||||
|
||||
@@ -297,16 +322,16 @@ def model_nightly_case_id(model_key: str) -> str:
|
||||
def print_model_catalog():
|
||||
"""Print preset order, model path, and whether each preset maps to nightly."""
|
||||
print()
|
||||
print("=" * 95)
|
||||
print("=" * CATALOG_TABLE_WIDTH)
|
||||
print("MODEL PRESETS — Nightly-aligned first, skill-only extras after")
|
||||
print("=" * 95)
|
||||
print(f"{'Preset':<17} {'Nightly':<28} {'Model Path':<46} {'GPUs':>4}")
|
||||
print("-" * 95)
|
||||
print("=" * CATALOG_TABLE_WIDTH)
|
||||
print(f"{'Preset':<24} {'Nightly':<28} {'Model Path':<46} {'GPUs':>4}")
|
||||
print("-" * CATALOG_TABLE_WIDTH)
|
||||
for model_key, cfg in MODELS.items():
|
||||
print(
|
||||
f"{model_key:<17} {model_nightly_case_id(model_key):<28} {cfg['path']:<46} {required_gpus_for_model(model_key):>4}"
|
||||
f"{model_key:<24} {model_nightly_case_id(model_key):<28} {cfg['path']:<46} {required_gpus_for_model(model_key):>4}"
|
||||
)
|
||||
print("-" * 112)
|
||||
print("-" * CATALOG_TABLE_WIDTH)
|
||||
print(
|
||||
"Nightly column shows the comparison_configs.json case id; '-' means skill-only."
|
||||
)
|
||||
@@ -498,15 +523,15 @@ def run_benchmark_once(
|
||||
def print_results_table(results: list[dict]):
|
||||
"""Print a compact table for one or more benchmark runs."""
|
||||
print()
|
||||
print("=" * 80)
|
||||
print("=" * RESULTS_TABLE_WIDTH)
|
||||
print("BENCHMARK RESULTS — Denoise Latency (primary metric ★)")
|
||||
print("(Models and params match benchmark-and-profile.md)")
|
||||
print("=" * 80)
|
||||
print("=" * RESULTS_TABLE_WIDTH)
|
||||
|
||||
print(
|
||||
f"{'Model':<14} {'Nightly':<24} {'Label':<12} {'Denoise(s)':>12} {'E2E(s)':>10} {'Peak Mem(GB)':>14}"
|
||||
f"{'Model':<24} {'Nightly':<28} {'Label':<12} {'Denoise(s)':>12} {'E2E(s)':>10} {'Peak Mem(GB)':>14}"
|
||||
)
|
||||
print("-" * 92)
|
||||
print("-" * RESULTS_TABLE_WIDTH)
|
||||
|
||||
for result in results:
|
||||
denoise_s = result.get("denoise_latency_s")
|
||||
@@ -516,10 +541,10 @@ def print_results_table(results: list[dict]):
|
||||
e2e_text = f"{e2e_s:.2f}" if isinstance(e2e_s, float) else "n/a"
|
||||
mem_text = f"{peak_mem:.1f}" if isinstance(peak_mem, float) else "n/a"
|
||||
print(
|
||||
f"{result['model']:<14} {model_nightly_case_id(result['model']):<24} {result['label']:<12} {denoise_text:>12} {e2e_text:>10} {mem_text:>14}"
|
||||
f"{result['model']:<24} {model_nightly_case_id(result['model']):<28} {result['label']:<12} {denoise_text:>12} {e2e_text:>10} {mem_text:>14}"
|
||||
)
|
||||
|
||||
print("-" * 92)
|
||||
print("-" * RESULTS_TABLE_WIDTH)
|
||||
print()
|
||||
print(
|
||||
"★ Denoise latency = sum of stages ending with DenoisingStage plus any RefinementStage."
|
||||
@@ -538,7 +563,7 @@ def main():
|
||||
choices=list(MODELS.keys()),
|
||||
help="Model to benchmark (default: flux)",
|
||||
)
|
||||
parser.add_argument("--all", action="store_true", help="Benchmark all 14 models")
|
||||
parser.add_argument("--all", action="store_true", help="Benchmark all 15 models")
|
||||
parser.add_argument(
|
||||
"--list-models",
|
||||
action="store_true",
|
||||
|
||||
@@ -71,6 +71,27 @@ Before writing CLI examples, re-read the active branch's `docs/diffusion/quantiz
|
||||
|
||||
B200 CI coverage can include loose BF16-vs-quantized quality checks. Inspect the active branch's `run_suite.py` before assuming they are part of the suite; mainline and feature branches may differ. Those checks are intended to catch blank, corrupted, or obviously divergent images, not exact image parity.
|
||||
|
||||
Mainline documentation now uses `lmsys/*` for the five converted ModelOpt
|
||||
checkpoint repos; the FLUX.2 NVFP4 raw export remains
|
||||
`black-forest-labs/FLUX.2-dev-NVFP4`. Do not use older `BBuf/*` examples unless
|
||||
you are explicitly testing a historical branch.
|
||||
|
||||
## Open PR Watchlist
|
||||
|
||||
As of 2026-05-02, these related SGLang PRs were open. Treat them as future
|
||||
support or migration work until they merge and the docs/CI matrix is updated.
|
||||
|
||||
- #23155 adds Qwen Image ModelOpt FP8 support.
|
||||
- #23199 adds HunyuanVideo ModelOpt FP8 support.
|
||||
- #23373 adds a runtime quantization flag; keep PTQ/export workflows separate from runtime quant examples until the CLI behavior is merged.
|
||||
- #24024 adds transformer FP8-cast compatibility mode.
|
||||
- #24186 re-enables B200 multimodal CI with NVFP4 fixes for FLUX.2 and Wan2.2.
|
||||
|
||||
Do not expand the validated matrix beyond FLUX.1, FLUX.2, and Wan2.2 solely
|
||||
because one of these PRs exists. Add a row only after the exact checkpoint,
|
||||
loader path, accuracy check, and benchmark scope are validated on the active
|
||||
branch.
|
||||
|
||||
## Documentation Maintenance
|
||||
|
||||
- Keep the validated ModelOpt support matrix in `docs/diffusion/quantization.md`.
|
||||
|
||||
@@ -54,7 +54,7 @@ These options **trade output quality** for speed or VRAM savings. Results will d
|
||||
| Option | CLI Flag / Env Var | What It Does | Speedup | Quality Impact / Limitations |
|
||||
|---|---|---|---|---|
|
||||
| **Approximate Attention** | `--attention-backend sage_attn` / `sage_attn_3` / `sliding_tile_attn` / `video_sparse_attn` / `sparse_video_gen_2_attn` / `vmoba_attn` / `sla_attn` / `sage_sla_attn` | Replaces exact attention with approximate or sparse variants. `sage_attn`: INT8/FP8 quantized Q·K; `sliding_tile_attn`: spatial-temporal tile skipping; others: model-specific sparse patterns. | ~1.5–2x on attention (varies by backend) | Quality degradation varies by backend and model. `sage_attn` is the most general; sparse backends (`sliding_tile_attn`, `video_sparse_attn`, etc.) are video-model-specific and may require config files (e.g. `--mask-strategy-file-path` for STA). Requires corresponding packages installed. |
|
||||
| **Cache-DiT** | `SGLANG_CACHE_DIT_ENABLED=true` + `--cache-dit-config <path>` | Caches intermediate residuals across denoising steps and skips redundant computations via a Selective Computation Mask (SCM). | ~1.5–2x on supported models | Quality depends on SCM config. Incompatible with `--dit-layerwise-offload`. Requires correct per-model config YAML. |
|
||||
| **Cache-DiT** | Native: `SGLANG_CACHE_DIT_ENABLED=true` plus `SGLANG_CACHE_DIT_*` env vars. Diffusers backend: `--backend diffusers --cache-dit-config <yaml-or-json>` | Caches intermediate residuals across denoising steps and skips redundant computations via DBCache, TaylorSeer, and optional SCM. | ~1.5-2x on supported models | Quality depends on cache policy. Incompatible with `--dit-layerwise-offload`. Do not pass `--cache-dit-config` for native SGLang tuning unless you are intentionally using the diffusers backend flow. |
|
||||
| **Quantized Models (Nunchaku / SVDQuant)** | `--enable-svdquant --transformer-weights-path <path>` + optional `--quantization-precision int4\|nvfp4`, `--quantization-rank 32` | W4A4-style quantization via [Nunchaku](https://nunchaku.tech). Reduces DiT weight memory by ~4x. Precision/rank can be auto-inferred from weight filename or set explicitly. | ~1.5–2x compute speedup | Lossy quantization; quality depends on rank and precision. Requires pre-quantized weights. Ampere (SM8x) or SM12x only (no Hopper SM90). Higher rank = better quality but more memory. |
|
||||
| **Pre-quantized Transformer Override** | `--transformer-path <dir-or-repo>` / `--transformer-weights-path <path>` | Load a quantized transformer component or raw transformer weights. For converted ModelOpt FP8/NVFP4 directories, prefer `--transformer-path`; use `--transformer-weights-path` for weight-only artifacts the model loader expects. | ~1.3–1.5x compute (dtype dependent) | Requires a validated quantized transformer override, such as one produced by the ModelOpt helper tools. Quality is usually slightly worse than BF16 and depends on the format, fallback layers, and calibration scope. |
|
||||
| **Component Precision Override** | `--dit-precision fp16`, `--vae-precision fp16\|bf16` | On-the-fly dtype conversion for individual components. E.g. convert a BF16 model to FP16 at load time, or run VAE in BF16 instead of FP32. | Reduces memory; FP16 can be faster on some GPUs | May affect numerical stability. VAE is FP32 by default for accuracy; lowering it is lossy. DiT defaults to BF16. |
|
||||
@@ -82,21 +82,35 @@ For Wan2.2 specifically:
|
||||
- for pure latency tuning, benchmark pure Ulysses too, for example `--ulysses-degree=4 --ring-degree=1` on 4 GPUs
|
||||
- on 8 GPUs, compare pure `--ulysses-degree=8` against `--enable-cfg-parallel --ulysses-degree=4`
|
||||
|
||||
### Nightly-aligned model, single GPU: LTX-2 two-stage
|
||||
### Nightly-aligned model, 2 GPUs: LTX-2 two-stage
|
||||
|
||||
```bash
|
||||
sglang generate --model-path Lightricks/LTX-2 \
|
||||
--pipeline-class-name LTX2TwoStagePipeline \
|
||||
--prompt "A beautiful sunset over the ocean" \
|
||||
--negative-prompt "shaky, glitchy, low quality, worst quality, deformed, distorted, disfigured, motion smear, motion artifacts, fused fingers, bad anatomy, weird hand, ugly, transition, static." \
|
||||
--width 1536 --height 1024 \
|
||||
--num-frames 121 --fps 24 \
|
||||
--seed 1234 --num-gpus 1 \
|
||||
--prompt "A cat and a dog baking a cake together in a kitchen." \
|
||||
--width 768 --height 512 \
|
||||
--num-frames 121 --num-inference-steps 50 --guidance-scale 4.0 \
|
||||
--seed 42 --num-gpus 2 --enable-cfg-parallel \
|
||||
--enable-torch-compile --warmup --save-output
|
||||
```
|
||||
|
||||
Note: this generate recipe is aligned with the nightly comparison case `ltx2_twostage_t2v`. `LTX2TwoStagePipeline` is a native path and auto-resolves the spatial upsampler plus distilled LoRA from the same model snapshot unless you override them.
|
||||
|
||||
### Nightly-aligned model, 2 GPUs: LTX-2.3 TI2V two-stage
|
||||
|
||||
```bash
|
||||
sglang generate --model-path Lightricks/LTX-2.3 \
|
||||
--pipeline-class-name LTX2TwoStagePipeline \
|
||||
--prompt "The cat starts walking slowly towards the camera." \
|
||||
--image-path "${ASSET_DIR}/cat.png" \
|
||||
--width 768 --height 512 \
|
||||
--num-frames 121 --num-inference-steps 50 --guidance-scale 4.0 \
|
||||
--seed 42 --num-gpus 2 \
|
||||
--enable-torch-compile --warmup --save-output
|
||||
```
|
||||
|
||||
Note: this matches the nightly comparison case `ltx2.3_twostage_ti2v_2gpus`. Download `${ASSET_DIR}/cat.png` with the benchmark/profile skill before running it.
|
||||
|
||||
### Native baseline, 2 GPUs: LTX-2.3 one-stage
|
||||
|
||||
```bash
|
||||
@@ -112,7 +126,7 @@ sglang generate --model-path Lightricks/LTX-2.3 \
|
||||
|
||||
Note: use this as the native `LTX2Pipeline` baseline for `LTX-2.3`. It keeps the validated one-stage resolution and explicit `LTX-2.3` sampling defaults, and matches the `ltx23-one-stage` benchmark preset in `sglang-diffusion-benchmark-profile`.
|
||||
|
||||
### Benchmark target, 2 GPUs: LTX-2.3 two-stage
|
||||
### Skill-only stress target, 2 GPUs: LTX-2.3 two-stage high resolution
|
||||
|
||||
```bash
|
||||
sglang generate --model-path Lightricks/LTX-2.3 \
|
||||
@@ -126,7 +140,7 @@ sglang generate --model-path Lightricks/LTX-2.3 \
|
||||
--enable-torch-compile --warmup --save-output
|
||||
```
|
||||
|
||||
Note: this is the recommended benchmark command for the `LTX-2.3` two-stage path. It uses the native `LTX2TwoStagePipeline` and matches the `ltx23-two-stage` benchmark preset in `sglang-diffusion-benchmark-profile`.
|
||||
Note: this is a high-resolution stress target for the native `LTX-2.3` two-stage path. It matches the skill-only `ltx23-two-stage` benchmark preset, not a nightly comparison case.
|
||||
|
||||
### Maximum speed, image model, single GPU, lossless
|
||||
|
||||
@@ -149,26 +163,64 @@ sglang generate --model-path <MODEL> \
|
||||
--prompt "..." --save-output
|
||||
```
|
||||
|
||||
### Maximum speed, lossy (SageAttention + Cache-DiT)
|
||||
### Maximum speed, lossy native path (SageAttention + Cache-DiT)
|
||||
|
||||
```bash
|
||||
SGLANG_CACHE_DIT_ENABLED=true sglang generate --model-path <MODEL> \
|
||||
--attention-backend sage_attn \
|
||||
--cache-dit-config <config.yaml> \
|
||||
--dit-layerwise-offload false \
|
||||
--enable-torch-compile --warmup \
|
||||
--prompt "..." --save-output
|
||||
```
|
||||
|
||||
Add native Cache-DiT knobs such as `SGLANG_CACHE_DIT_SCM_PRESET=medium`,
|
||||
`SGLANG_CACHE_DIT_RDT=0.24`, or `SGLANG_CACHE_DIT_TAYLORSEER=true` only after
|
||||
you have a BF16 baseline output to compare against.
|
||||
|
||||
For a diffusers-backend Cache-DiT YAML/JSON config baseline, make the fallback
|
||||
explicit:
|
||||
|
||||
```bash
|
||||
sglang generate --backend diffusers --model-path <MODEL> \
|
||||
--cache-dit-config <config.yaml> \
|
||||
--dit-layerwise-offload false \
|
||||
--prompt "..." --save-output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Model-Specific Starting Points
|
||||
|
||||
Use these as first commands to benchmark, not as universal winners.
|
||||
|
||||
| Model family | First performance shape | Starting flags | Notes |
|
||||
|---|---|---|---|
|
||||
| FLUX.1 / FLUX.2 image | 1024x1024, 50 steps, 1 GPU | `--enable-torch-compile --warmup --dit-layerwise-offload false` | `black-forest-labs/FLUX.*` repos are gated; for FP8/NVFP4 use validated `--transformer-path` or `--transformer-weights-path` flows from the quant skill. |
|
||||
| Qwen-Image / Qwen-Image-Edit | 1024x1024, 50 steps, 1 GPU | `--enable-torch-compile --warmup`; optionally native `SGLANG_CACHE_DIT_ENABLED=true` | Cache-DiT is lossy. For edit tasks, keep reference image, seed, and output size fixed. |
|
||||
| Z-Image-Turbo | 1024x1024, 9 steps, guidance 4.0 | `--enable-torch-compile --warmup` | Mainline has Z-Image tanh/gate norm fusions; PR #21912 tracks FP8 plus CUDA Graph work. |
|
||||
| Wan2.2 A14B T2V/I2V | 720p, 81 frames | Nightly: `--num-gpus 4 --enable-cfg-parallel --ulysses-degree 2 --text-encoder-cpu-offload --pin-cpu-memory` | For lowest latency, also benchmark pure Ulysses on the same GPUs. |
|
||||
| Wan2.2 TI2V 5B | 720p, 81 frames, 1 GPU | `--enable-torch-compile --warmup` | Keep the input image and motion prompt fixed when comparing sparse attention or Cache-DiT. |
|
||||
| LTX-2 / LTX-2.3 | 768x512, 121 frames, 2 GPUs | `--pipeline-class-name LTX2TwoStagePipeline --enable-torch-compile --warmup`; LTX-2 nightly also uses `--enable-cfg-parallel` | Use the benchmark/profile skill presets for exact nightly alignment. PRs #22441, #24025, and #23736 track additional LTX2 perf/parallel work. |
|
||||
| HunyuanVideo | 848x480 or 720p class video | `--text-encoder-cpu-offload --pin-cpu-memory --enable-torch-compile --warmup` | Check VAE decode separately. GroupNorm+SiLU is default-eligible in mainline when wrapper guards pass; use `bench_group_norm_silu.py` when VAE residual blocks are hot. |
|
||||
| JoyAI-Image-Edit | 1024-class TI2I, 40 steps, guidance 4.0 | Start from the CI/default TI2I path for `jdopensource/JoyAI-Image-Edit-Diffusers`; add `--warmup` and benchmark `--enable-torch-compile` separately | Newly supported image-edit path. Keep the input image, prompt, seed, and output size fixed; sequence shard is auto-enabled for Joy pipelines. |
|
||||
| MOVA / Helios | Use the benchmark/profile presets first | `--enable-torch-compile --warmup`; pin offload flags explicitly | PR #20530 tracks MOVA fused RMSNorm+RoPE; PR #24059 tracks Helios fused norm modulation. |
|
||||
|
||||
## Open PR Watchlist
|
||||
|
||||
As of 2026-05-02, these performance PRs were open. Treat them as direction and
|
||||
prior art until merged:
|
||||
|
||||
- Fusion/kernel: #24025 LTX2 QK norm, #24059 Helios norm modulation, #24117 Z-Image packed QKV, #19488 Wan elementwise cross-block fusion, #19249 Z-Image gate/norm fusion, #20429 Qwen-Image layernorm/modulation, #20530 MOVA RMSNorm+RoPE.
|
||||
- VAE/decode: #22531 LTX2 parallel VAE, #20927 batched tiled VAE decode.
|
||||
- Runtime/parallel/cache: #22805 FLUX.2 packed QKV for A2A, #21742 hybrid attention schedule, #24053 USP replicated-prefix fix, #21613 TeaCache refactor, #24227 WanVideo TeaCache fix, #18764 dynamic batching, #24200 disaggregated diffusion.
|
||||
|
||||
## Tips
|
||||
|
||||
- **Benchmarking**: always use `--warmup` and look for the line ending with `(with warmup excluded)` for accurate timing.
|
||||
- **Perf dump**: use `--perf-dump-path result.json` to save structured metrics, then compare with `python python/sglang/multimodal_gen/benchmarks/compare_perf.py baseline.json result.json`.
|
||||
- **Offload tuning**: after the first request, the runtime logs peak GPU memory and which components could stay resident. Use this to decide which `--*-cpu-offload` flags to disable.
|
||||
- **Backend selection**: `--backend sglang` (default, auto-detected) enables all native optimizations (fused kernels, SP, etc.). `--backend diffusers` falls back to vanilla Diffusers pipelines but supports `--cache-dit-config` and diffusers attention backends.
|
||||
- **Backend selection**: `--backend sglang` (default, auto-detected) enables native optimizations (fused kernels, SP, native Cache-DiT env knobs, etc.). `--backend diffusers` falls back to Diffusers pipelines and is the path that accepts `--cache-dit-config` plus diffusers attention backend names.
|
||||
- **Wan2.2-I2V sizing**: explicit `--width/--height` on `Wan2.2-I2V-A14B` control the target area while preserving the condition-image aspect ratio.
|
||||
- **Mainline diffusion fast paths**: before proposing a new kernel or overlap scheme, check `sglang-diffusion-benchmark-profile/existing-fast-paths.md`. It covers Z-Image residual-form modulation, fused diffusion `QK norm + RoPE`, and existing multi-GPU overlap families such as Ulysses / USP and turbo-layer async all-to-all.
|
||||
- **Mainline diffusion fast paths**: before proposing a new kernel or overlap scheme, check `sglang-diffusion-benchmark-profile/existing-fast-paths.md`. It covers GroupNorm+SiLU, Z-Image residual-form modulation, fused diffusion `QK norm + RoPE`, packed QKV/NVFP4 expectations, and existing multi-GPU overlap families such as Ulysses / USP and turbo-layer async all-to-all.
|
||||
- **NVFP4 trace interpretation**: on FLUX.2 NVFP4 and Nunchaku-style checkpoints, packed QKV is expected. SGLang intentionally uses fused projection modules such as `to_qkv` / `to_added_qkv` instead of separate `to_q` / `to_k` / `to_v`, so a split-QKV trace usually means the quantized path did not engage rather than a brand new fusion opportunity.
|
||||
- **Hotspot workflow split**: use `sglang-diffusion-benchmark-profile` to prove and classify a slowdown with perf dumps plus `torch.profiler`; hand concrete kernel work to `sglang-diffusion-ako4all-kernel` or another specialized optimization skill instead of expanding the benchmark skill.
|
||||
|
||||
Reference in New Issue
Block a user