diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-ako4all-kernel/SKILL.md b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-ako4all-kernel/SKILL.md index aaecd641f..423ea408e 100644 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-ako4all-kernel/SKILL.md +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-ako4all-kernel/SKILL.md @@ -34,6 +34,13 @@ First use [../sglang-diffusion-benchmark-profile/SKILL.md](../sglang-diffusion-b - rule out existing mainline fast paths - prove the run stayed on the native SGLang diffusion backend, not a diffusers fallback +Before opening AKO, also read +[../sglang-diffusion-benchmark-profile/existing-fast-paths.md](../sglang-diffusion-benchmark-profile/existing-fast-paths.md). +It records current mainline fusions plus the open PR watchlist for diffusion +kernel, VAE, attention, cache, and scheduling work. If an open PR already covers +the same shape family, use it as prior art or decide whether to rebase/extend it +instead of starting a duplicate kernel. + If a future specialized optimization skill matches the kernel family better than AKO4ALL, hand off there instead. The diagnosis contract stays the same. ## Mandatory AKO4ALL Preflight @@ -58,6 +65,8 @@ By default it uses the existing `origin` URL, or `AKO4ALL_URL` if you need to ov - Identify the exact kernel entry point and runtime call sites in `sglang`. - Record the target shapes, dtypes, model families, and whether the kernel is on a hot path. - Reuse existing unit tests and benchmark entry points when they already exist. +- Record whether the hotspot overlaps an open PR from `existing-fast-paths.md`; + if it does, note the PR number in the AKO context and final PR artifacts. ### 2. Bootstrap the AKO Harness diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/SKILL.md b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/SKILL.md index 28fd91a1e..e83b1a2fd 100644 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/SKILL.md +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/SKILL.md @@ -42,8 +42,8 @@ If any benchmark, perf-dump, or `torch.profiler` command prints one of those sig ## Main Reference -- [benchmark-and-profile.md](benchmark-and-profile.md) — canonical denoise benchmark, perf dump, and `torch.profiler` workflow; uses the checked-in nightly-aligned presets, plus `LTX-2`, `LTX-2.3` one-stage, and `LTX-2.3` two-stage benchmark recipes -- [existing-fast-paths.md](existing-fast-paths.md) — map bottlenecks to existing fused kernels, packed QKV paths, fused `QK norm + RoPE`, and distributed overlap patterns before proposing new code +- [benchmark-and-profile.md](benchmark-and-profile.md) — canonical denoise benchmark, perf dump, and `torch.profiler` workflow; uses checked-in nightly-aligned presets plus skill-only stress recipes such as `LTX-2.3` one-stage/two-stage, HunyuanVideo, MOVA, and Helios +- [existing-fast-paths.md](existing-fast-paths.md) — map bottlenecks to existing fused kernels, packed QKV paths, fused `QK norm + RoPE`, distributed overlap patterns, and open optimization PRs before proposing new code - [scripts/diffusion_skill_env.py](scripts/diffusion_skill_env.py) — preflight helper: repo root discovery via `sglang.__file__`, write-access probe, benchmark/profile output directories, idle GPU selection - [scripts/bench_diffusion_denoise.py](scripts/bench_diffusion_denoise.py) — end-to-end denoise benchmark preset runner via `sglang generate`; pins `--backend=sglang`, supports `--no-torch-compile`, and saves perf dumps by label for `compare_perf.py` @@ -52,6 +52,7 @@ If any benchmark, perf-dump, or `torch.profiler` command prints one of those sig Before calling a diffusion hotspot "new", first classify it with `existing-fast-paths.md`. Always rule out these existing families first: +- HunyuanVideo VAE GroupNorm+SiLU - Z-Image residual-form modulation - fused diffusion `QK norm + RoPE` - NVFP4 / Nunchaku packed QKV diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/benchmark-and-profile.md b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/benchmark-and-profile.md index 9d0da0810..9544901f0 100644 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/benchmark-and-profile.md +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/benchmark-and-profile.md @@ -97,9 +97,11 @@ Treat `"$BENCH_PY"` as the source of truth for preset order. Nightly diffusion comparison is server/API based (`sglang serve` plus requests). This skill stays on `sglang generate` for local benchmarking and profiling, but -the first 9 presets in `bench_diffusion_denoise.py` are aligned to nightly on -model, prompt, negative prompt, reference image, size, frames, fps, seed, GPU -count, and any explicitly overridden sampling or parallelism flags. +the nightly-aligned presets in `bench_diffusion_denoise.py` mirror +`scripts/ci/utils/diffusion/comparison_configs.json` on model, task, prompt, +reference image, size, frames, seed, GPU count, serve args, and the request +defaults used by `run_comparison.py` when a case omits steps or guidance. +When in doubt, re-check that JSON before trusting this reference. List the current preset order: @@ -135,6 +137,15 @@ PYTHONPATH=python python3 "$BENCH_PY" \ --output-dir "${BENCH_DIR}" ``` +Run the nightly-aligned `LTX-2.3` TI2V two-stage preset: + +```bash +PYTHONPATH=python python3 "$BENCH_PY" \ + --model ltx23-ti2v-two-stage \ + --label baseline \ + --output-dir "${BENCH_DIR}" +``` + Run the `LTX-2.3` two-stage skill preset: ```bash @@ -164,10 +175,11 @@ Nightly-aligned presets come first; skill-only presets stay available after them | `zimage` | `Tongyi-MAI/Z-Image-Turbo` | Yes: `zimage_turbo_t2i_1024` | Aligned to nightly prompt and guidance 4.0 | | `wan-t2v` | `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | Yes: `wan22_t2v_a14b_720p` | Aligned to nightly CFG-parallel 4-GPU launch | | `wan-ti2v` | `Wan-AI/Wan2.2-TI2V-5B-Diffusers` | Yes: `wan22_ti2v_5b_720p` | Uses the nightly cat image and motion prompt | -| `ltx2` | `Lightricks/LTX-2` | Yes: `ltx2_twostage_t2v` | Uses `LTX2TwoStagePipeline`; nightly-aligned prompt, negative prompt, 1536x1024, 121 frames, fps 24, seed 1234 | +| `ltx2` | `Lightricks/LTX-2` | Yes: `ltx2_twostage_t2v` | Uses `LTX2TwoStagePipeline`; 2 GPUs, CFG parallel, 768x512, 121 frames, seed 42 | +| `ltx23-ti2v-two-stage` | `Lightricks/LTX-2.3` | Yes: `ltx2.3_twostage_ti2v_2gpus` | Uses the nightly cat image, motion prompt, `LTX2TwoStagePipeline`, 2 GPUs, 768x512, 121 frames, seed 42 | | `wan-i2v` | `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | Yes: `wan22_i2v_a14b_720p` | Aligned to nightly CFG-parallel 4-GPU launch | | `ltx23-one-stage` | `Lightricks/LTX-2.3` | No | Skill-only extra preset for the native `LTX-2.3` one-stage baseline; 2 GPUs, 768x512, 121 frames, fps 24, 30 steps, guidance 3.0, seed 1234 | -| `ltx23-two-stage` | `Lightricks/LTX-2.3` | No | Skill-only extra preset for the native `LTX-2.3` two-stage path; uses `LTX2TwoStagePipeline`, 2 GPUs, 1536x1024, 121 frames, fps 24, 30 steps, guidance 3.0, seed 1234 | +| `ltx23-two-stage` | `Lightricks/LTX-2.3` | No | Skill-only high-resolution stress preset for the native `LTX-2.3` two-stage path; uses `LTX2TwoStagePipeline`, 2 GPUs, 1536x1024, 121 frames, fps 24, 30 steps, guidance 3.0, seed 1234 | | `hunyuanvideo` | `hunyuanvideo-community/HunyuanVideo` | No | Skill-only extra preset | | `mova-720p` | `OpenMOSS-Team/MOVA-720p` | No | Skill-only extra preset | | `helios` | `BestWishYsh/Helios-Base` | No | Skill-only extra preset | @@ -184,17 +196,34 @@ and **best latency tuning**: sglang generate \ --model-path=Lightricks/LTX-2 \ --pipeline-class-name=LTX2TwoStagePipeline \ - --prompt="A beautiful sunset over the ocean" \ - --negative-prompt="shaky, glitchy, low quality, worst quality, deformed, distorted, disfigured, motion smear, motion artifacts, fused fingers, bad anatomy, weird hand, ugly, transition, static." \ - --width=1536 --height=1024 \ - --num-frames=121 --fps=24 \ - --seed=1234 --num-gpus=1 \ + --prompt="A cat and a dog baking a cake together in a kitchen." \ + --width=768 --height=512 \ + --num-frames=121 \ + --num-inference-steps=50 --guidance-scale=4.0 \ + --seed=42 --num-gpus=2 --enable-cfg-parallel \ --save-output --enable-torch-compile --warmup ``` `LTX2TwoStagePipeline` is a native path. The spatial upsampler and distilled LoRA are auto-resolved from the same model snapshot unless you override them. +### Manual command example: LTX-2.3 TI2V Two-Stage + +```bash +sglang generate \ + --model-path=Lightricks/LTX-2.3 \ + --pipeline-class-name=LTX2TwoStagePipeline \ + --prompt="The cat starts walking slowly towards the camera." \ + --image-path="${ASSET_DIR}/cat.png" \ + --width=768 --height=512 \ + --num-frames=121 \ + --num-inference-steps=50 --guidance-scale=4.0 \ + --seed=42 --num-gpus=2 \ + --save-output --enable-torch-compile --warmup +``` + +This matches the nightly comparison case `ltx2.3_twostage_ti2v_2gpus`. + ### Manual command example: LTX-2.3 One-Stage ```bash @@ -212,7 +241,7 @@ sglang generate \ Use this when you want the native `LTX2Pipeline` baseline for `LTX-2.3` at the validated one-stage resolution. -### Manual command example: LTX-2.3 Two-Stage +### Manual command example: LTX-2.3 Two-Stage High-Resolution Stress ```bash sglang generate \ @@ -227,8 +256,8 @@ sglang generate \ --save-output --enable-torch-compile --warmup ``` -This matches the `ltx23-two-stage` skill preset and is a good benchmark target -for the native `LTX-2.3` two-stage path. +This matches the skill-only `ltx23-two-stage` preset. Use it as a +high-resolution stress target, not as a nightly comparison case. ### Manual command example: Wan2.2-I2V-A14B 720P @@ -295,7 +324,7 @@ By default SGLang profiles the denoising stage. The default sampling window is 5 profiled timesteps after warmup. ```bash -SGLANG_TORCH_PROFILER_DIR="${PROFILE_DIR}/torch" \ +SGLANG_DIFFUSION_TORCH_PROFILER_DIR="${PROFILE_DIR}/torch" \ sglang generate \ --model-path=black-forest-labs/FLUX.1-dev \ --prompt="A futuristic cyberpunk city at night" \ @@ -308,8 +337,9 @@ Use `--profile-all-stages` only when you really need text encoder, VAE, or other non-denoise stages too. The generated trace path is printed in the console and also lands under -`./logs/` or `SGLANG_TORCH_PROFILER_DIR`. Open it in Perfetto if you want a -timeline view: +`SGLANG_DIFFUSION_TORCH_PROFILER_DIR`. The diffusion profiler falls back to +`SGLANG_TORCH_PROFILER_DIR` and then `./logs` when the diffusion-specific env +var is unset. Open the trace in Perfetto if you want a timeline view: - https://ui.perfetto.dev/ ### 3. Rank the hot CUDA kernels @@ -323,7 +353,11 @@ import gzip import json import os -log_dir = os.environ.get("SGLANG_TORCH_PROFILER_DIR", "./logs") +log_dir = ( + os.environ.get("SGLANG_DIFFUSION_TORCH_PROFILER_DIR") + or os.environ.get("SGLANG_TORCH_PROFILER_DIR") + or "./logs" +) trace_path = sorted( glob.glob(f"{log_dir}/*.trace.json.gz"), key=os.path.getmtime, diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/existing-fast-paths.md b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/existing-fast-paths.md index 2bbb0c88c..24399eee3 100644 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/existing-fast-paths.md +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/existing-fast-paths.md @@ -9,10 +9,14 @@ configuration first before handing the problem to a specialized kernel-optimizat - `python/sglang/multimodal_gen/runtime/layers/elementwise.py` - `python/sglang/multimodal_gen/runtime/layers/rotary_embedding/utils.py` - `python/sglang/jit_kernel/diffusion/triton/scale_shift.py` +- `python/sglang/jit_kernel/diffusion/group_norm_silu.py` +- `python/sglang/jit_kernel/diffusion/triton/group_norm_silu.py` - `python/sglang/jit_kernel/diffusion/triton/norm.py` - `python/sglang/jit_kernel/diffusion/triton/rmsnorm_onepass.py` - `python/sglang/jit_kernel/diffusion/triton/rotary.py` - `python/sglang/jit_kernel/diffusion/cutedsl/scale_residual_norm_scale_shift.py` +- `python/sglang/jit_kernel/tests/diffusion/test_group_norm_silu.py` +- `python/sglang/jit_kernel/benchmark/diffusion/bench_group_norm_silu.py` - `python/sglang/jit_kernel/norm.py` - `python/sglang/multimodal_gen/runtime/platforms/cuda.py` - `python/sglang/multimodal_gen/runtime/layers/attention/selector.py` @@ -68,6 +72,15 @@ configuration first before handing the problem to a specialized kernel-optimizat - NPU fallback: `npu_fallback.apply_rotary_embedding_native`. - Validation: `python/sglang/jit_kernel/tests/test_rope.py`. +7. HunyuanVideo VAE GroupNorm + SiLU fusion +- Kernel: `triton_group_norm_silu` +- Locations: `diffusion/group_norm_silu.py`, `triton/group_norm_silu.py`, `runtime/models/vaes/hunyuanvae.py` +- Use case: `activation(group_norm(x))` when the activation is non-inplace `nn.SiLU` and the GroupNorm is affine. +- Enablement: mainline uses `apply_group_norm_silu(...)` in HunyuanVideo VAE paths by default; there is no env toggle. The wrapper dispatches to Triton only when guards pass. +- Constraints: CUDA inference path only; no grad, `x.requires_grad == False`, `nn.GroupNorm`, `nn.SiLU(inplace=False)`, affine norm with weight and bias. Unsupported cases fall back to native `activation(norm(x))`. +- Validation: `python/sglang/jit_kernel/tests/diffusion/test_group_norm_silu.py`. +- Microbench: `python/sglang/jit_kernel/benchmark/diffusion/bench_group_norm_silu.py`. + **Faster CUDA Kernel Usage Points** 1. sgl-kernel RMSNorm and fused add RMSNorm @@ -112,6 +125,7 @@ configuration first before handing the problem to a specialized kernel-optimizat - Supported head dims: `64, 128, 256`. - Behavior: `apply_qk_norm_rope` prefers the fused JIT kernel when all guards pass; otherwise it falls back to `apply_qk_norm(...)` plus `apply_flashinfer_rope_qk_inplace(...)`. - Validation: `python/sglang/jit_kernel/tests/diffusion/test_qknorm_rope.py`. +- Watchlist: PR #24025 adds LTX2-specific QK norm fusion work. Until merged, treat LTX2 traces that miss the generic fused path as an enablement/shape-guard issue first, not proof that the PR path exists locally. **Nunchaku Fused GELU MLP** @@ -132,6 +146,7 @@ configuration first before handing the problem to a specialized kernel-optimizat - AdaLN modulation: `LayerNormScaleShift`, `RMSNormScaleShift`, `ScaleResidual*` in `layernorm.py`. - Qwen-Image gating: `fuse_scale_shift_gate_select01_kernel` in `qwen_image.py`. - Z-Image residual-form modulation: `fused_norm_tanh_mul_add` and `fused_norm_tanh_mul_add_norm_scale` in `zimage.py`. +- HunyuanVideo VAE GroupNorm+SiLU: `apply_group_norm_silu` in `hunyuanvae.py`; default-eligible when wrapper guards pass. - QK norm: `apply_qk_norm` used in `flux.py`, `flux_2.py`, `qwen_image.py`, `zimage.py`, `wanvideo.py`, `ltx_2.py`, `hunyuanvideo.py`. - QK norm + RoPE: `apply_qk_norm_rope` in `layernorm.py`; use this path when the model wants fused attention prep instead of separate QK norm and RoPE calls. - Nunchaku fused GELU MLP: `_fused_gelu_mlp` in `flux.py` for quantized FLUX-family checkpoints. @@ -146,6 +161,36 @@ configuration first before handing the problem to a specialized kernel-optimizat - Dual-stream diffusion models: `use_dual_stream = True` in models such as `hunyuan3d.py` is an existing overlap family. - Workflow rule: if a hotspot is communication-heavy, rule out these in-repo overlap families before proposing a brand new overlap design. +**Open PR Watchlist** + +As of 2026-05-02, these SGLang PRs were still open. Use them as upstream +direction and prior art, not as current-main behavior. Re-check the PR state +before relying on any file path or flag. + +- Norm, modulation, and packed projection fusions: + - #24025 LTX2 QK norm fusion. + - #24059 Helios fused norm modulation. + - #24117 Z-Image packed QKV. + - #19488 Wan cross-block elementwise fusion. + - #19249 Z-Image `scale residual norm scale shift` plus `add gate norm` fusion. + - #18897 dual norm fusion for FLUX-family paths (draft). + - #20429 Qwen-Image layernorm and `fuse_scale_shift_gate_select01` work. + - #20530 MOVA fused RMSNorm + interleaved RoPE. +- VAE and decode-side acceleration: + - #22531 LTX2 parallel VAE support and #20927 batched tiled VAE decode (draft). +- Attention, communication, and runtime scheduling: + - #22805 FLUX.2 packed QKV for all-to-all. + - #21742 hybrid attention schedule. + - #24053 USP attention with replicated prefixes. + - #18764 dynamic batching v0. + - #24200 disaggregated diffusion v2. +- Cache and CUDA graph: + - #21613 TeaCache refactor. + - #24227 WanVideo TeaCache skipping fix. + - #20447 TeaCache support for GLM-Image, Qwen-Image, and related models. + - #19516 Qwen-Image CUDA Graph. + - #21912 Z-Image Turbo FP8 full quantization and CUDA Graph. + **Constraints and Fallbacks** - `scale_shift` Triton requires CUDA + contiguous `x`. NPU swaps to native. - CuTe DSL fused norms require `D % 256 == 0` and `D <= 8192`. diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py index 3ffce18e0..e600187d6 100755 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py @@ -12,7 +12,7 @@ Usage: # Tag the run for later compare_perf.py usage python3 python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py --model flux --label tuned - # All 14 preset models + # All 15 preset models python3 python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py --all # Show preset order, model path, and nightly mapping @@ -58,10 +58,13 @@ DIFFUSERS_FALLBACK_SIGNALS = ( "using diffusers backend", "loaded diffusers pipeline", ) +CATALOG_TABLE_WIDTH = 105 +RESULTS_TABLE_WIDTH = 105 # --------------------------------------------------------------------------- # Model configs — kept in exact sync with benchmark-and-profile.md -# Nightly-aligned presets come first, followed by skill-only extras. +# Nightly-aligned presets mirror scripts/ci/utils/diffusion/comparison_configs.json +# first, followed by skill-only extras. # Each entry produces the same `sglang generate` command as shown in that doc. # --------------------------------------------------------------------------- MODELS = { @@ -166,19 +169,36 @@ MODELS = { "ltx2": { "nightly_case_id": "ltx2_twostage_t2v", "path": "Lightricks/LTX-2", - "prompt": "A beautiful sunset over the ocean", - "negative_prompt": "shaky, glitchy, low quality, worst quality, deformed, distorted, disfigured, motion smear, motion artifacts, fused fingers, bad anatomy, weird hand, ugly, transition, static.", - "seed": 1234, + "prompt": "A cat and a dog baking a cake together in a kitchen.", "extra_args": [ "--pipeline-class-name=LTX2TwoStagePipeline", - "--width=1536", - "--height=1024", + "--width=768", + "--height=512", "--num-frames=121", - "--fps=24", - "--num-gpus=1", + "--num-inference-steps=50", + "--guidance-scale=4.0", + "--num-gpus=2", + "--enable-cfg-parallel", ], }, - # 9. Nightly: wan22_i2v_a14b_720p + # 9. Nightly: ltx2.3_twostage_ti2v_2gpus + # Requires: /inputs/diffusion_benchmark/figs/cat.png + "ltx23-ti2v-two-stage": { + "nightly_case_id": "ltx2.3_twostage_ti2v_2gpus", + "path": "Lightricks/LTX-2.3", + "prompt": "The cat starts walking slowly towards the camera.", + "image_path": str(ASSET_DIR / "cat.png"), + "extra_args": [ + "--pipeline-class-name=LTX2TwoStagePipeline", + "--width=768", + "--height=512", + "--num-frames=121", + "--num-inference-steps=50", + "--guidance-scale=4.0", + "--num-gpus=2", + ], + }, + # 10. Nightly: wan22_i2v_a14b_720p # Requires: /inputs/diffusion_benchmark/figs/cat.png "wan-i2v": { "nightly_case_id": "wan22_i2v_a14b_720p", @@ -197,7 +217,7 @@ MODELS = { "--pin-cpu-memory", ], }, - # 10. Skill-only extra preset + # 11. Skill-only extra preset "ltx23-one-stage": { "path": "Lightricks/LTX-2.3", "prompt": "A beautiful sunset over the ocean", @@ -213,7 +233,7 @@ MODELS = { "--num-gpus=2", ], }, - # 11. Skill-only extra preset + # 12. Skill-only extra preset "ltx23-two-stage": { "path": "Lightricks/LTX-2.3", "prompt": "A beautiful sunset over the ocean", @@ -230,7 +250,7 @@ MODELS = { "--num-gpus=2", ], }, - # 12. Skill-only extra preset + # 13. Skill-only extra preset "hunyuanvideo": { "path": "hunyuanvideo-community/HunyuanVideo", "prompt": "A cat and a dog baking a cake together in a kitchen. The cat is carefully measuring flour, while the dog is stirring the batter with a wooden spoon. The kitchen is cozy, with sunlight streaming through the window.", @@ -243,7 +263,7 @@ MODELS = { "--num-inference-steps=30", ], }, - # 13. Skill-only extra preset + # 14. Skill-only extra preset # Requires: /inputs/diffusion_benchmark/figs/mova_single_person.jpg "mova-720p": { "path": "OpenMOSS-Team/MOVA-720p", @@ -259,7 +279,7 @@ MODELS = { "--num-inference-steps=2", ], }, - # 14. Skill-only extra preset + # 15. Skill-only extra preset "helios": { "path": "BestWishYsh/Helios-Base", "prompt": "A curious raccoon", @@ -285,7 +305,12 @@ def required_gpus_for_model(model_key: str) -> int: return 4 if model_key == "mova-720p": return 4 - if model_key in {"ltx23-one-stage", "ltx23-two-stage"}: + if model_key in { + "ltx2", + "ltx23-ti2v-two-stage", + "ltx23-one-stage", + "ltx23-two-stage", + }: return 2 return 1 @@ -297,16 +322,16 @@ def model_nightly_case_id(model_key: str) -> str: def print_model_catalog(): """Print preset order, model path, and whether each preset maps to nightly.""" print() - print("=" * 95) + print("=" * CATALOG_TABLE_WIDTH) print("MODEL PRESETS — Nightly-aligned first, skill-only extras after") - print("=" * 95) - print(f"{'Preset':<17} {'Nightly':<28} {'Model Path':<46} {'GPUs':>4}") - print("-" * 95) + print("=" * CATALOG_TABLE_WIDTH) + print(f"{'Preset':<24} {'Nightly':<28} {'Model Path':<46} {'GPUs':>4}") + print("-" * CATALOG_TABLE_WIDTH) for model_key, cfg in MODELS.items(): print( - f"{model_key:<17} {model_nightly_case_id(model_key):<28} {cfg['path']:<46} {required_gpus_for_model(model_key):>4}" + f"{model_key:<24} {model_nightly_case_id(model_key):<28} {cfg['path']:<46} {required_gpus_for_model(model_key):>4}" ) - print("-" * 112) + print("-" * CATALOG_TABLE_WIDTH) print( "Nightly column shows the comparison_configs.json case id; '-' means skill-only." ) @@ -498,15 +523,15 @@ def run_benchmark_once( def print_results_table(results: list[dict]): """Print a compact table for one or more benchmark runs.""" print() - print("=" * 80) + print("=" * RESULTS_TABLE_WIDTH) print("BENCHMARK RESULTS — Denoise Latency (primary metric ★)") print("(Models and params match benchmark-and-profile.md)") - print("=" * 80) + print("=" * RESULTS_TABLE_WIDTH) print( - f"{'Model':<14} {'Nightly':<24} {'Label':<12} {'Denoise(s)':>12} {'E2E(s)':>10} {'Peak Mem(GB)':>14}" + f"{'Model':<24} {'Nightly':<28} {'Label':<12} {'Denoise(s)':>12} {'E2E(s)':>10} {'Peak Mem(GB)':>14}" ) - print("-" * 92) + print("-" * RESULTS_TABLE_WIDTH) for result in results: denoise_s = result.get("denoise_latency_s") @@ -516,10 +541,10 @@ def print_results_table(results: list[dict]): e2e_text = f"{e2e_s:.2f}" if isinstance(e2e_s, float) else "n/a" mem_text = f"{peak_mem:.1f}" if isinstance(peak_mem, float) else "n/a" print( - f"{result['model']:<14} {model_nightly_case_id(result['model']):<24} {result['label']:<12} {denoise_text:>12} {e2e_text:>10} {mem_text:>14}" + f"{result['model']:<24} {model_nightly_case_id(result['model']):<28} {result['label']:<12} {denoise_text:>12} {e2e_text:>10} {mem_text:>14}" ) - print("-" * 92) + print("-" * RESULTS_TABLE_WIDTH) print() print( "★ Denoise latency = sum of stages ending with DenoisingStage plus any RefinementStage." @@ -538,7 +563,7 @@ def main(): choices=list(MODELS.keys()), help="Model to benchmark (default: flux)", ) - parser.add_argument("--all", action="store_true", help="Benchmark all 14 models") + parser.add_argument("--all", action="store_true", help="Benchmark all 15 models") parser.add_argument( "--list-models", action="store_true", diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-modelopt-quant/SKILL.md b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-modelopt-quant/SKILL.md index 8fbcbef12..f3298fad8 100644 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-modelopt-quant/SKILL.md +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-modelopt-quant/SKILL.md @@ -71,6 +71,27 @@ Before writing CLI examples, re-read the active branch's `docs/diffusion/quantiz B200 CI coverage can include loose BF16-vs-quantized quality checks. Inspect the active branch's `run_suite.py` before assuming they are part of the suite; mainline and feature branches may differ. Those checks are intended to catch blank, corrupted, or obviously divergent images, not exact image parity. +Mainline documentation now uses `lmsys/*` for the five converted ModelOpt +checkpoint repos; the FLUX.2 NVFP4 raw export remains +`black-forest-labs/FLUX.2-dev-NVFP4`. Do not use older `BBuf/*` examples unless +you are explicitly testing a historical branch. + +## Open PR Watchlist + +As of 2026-05-02, these related SGLang PRs were open. Treat them as future +support or migration work until they merge and the docs/CI matrix is updated. + +- #23155 adds Qwen Image ModelOpt FP8 support. +- #23199 adds HunyuanVideo ModelOpt FP8 support. +- #23373 adds a runtime quantization flag; keep PTQ/export workflows separate from runtime quant examples until the CLI behavior is merged. +- #24024 adds transformer FP8-cast compatibility mode. +- #24186 re-enables B200 multimodal CI with NVFP4 fixes for FLUX.2 and Wan2.2. + +Do not expand the validated matrix beyond FLUX.1, FLUX.2, and Wan2.2 solely +because one of these PRs exists. Add a row only after the exact checkpoint, +loader path, accuracy check, and benchmark scope are validated on the active +branch. + ## Documentation Maintenance - Keep the validated ModelOpt support matrix in `docs/diffusion/quantization.md`. diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-performance/SKILL.md b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-performance/SKILL.md index a66232ada..fd4934a4c 100644 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-performance/SKILL.md +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-performance/SKILL.md @@ -54,7 +54,7 @@ These options **trade output quality** for speed or VRAM savings. Results will d | Option | CLI Flag / Env Var | What It Does | Speedup | Quality Impact / Limitations | |---|---|---|---|---| | **Approximate Attention** | `--attention-backend sage_attn` / `sage_attn_3` / `sliding_tile_attn` / `video_sparse_attn` / `sparse_video_gen_2_attn` / `vmoba_attn` / `sla_attn` / `sage_sla_attn` | Replaces exact attention with approximate or sparse variants. `sage_attn`: INT8/FP8 quantized Q·K; `sliding_tile_attn`: spatial-temporal tile skipping; others: model-specific sparse patterns. | ~1.5–2x on attention (varies by backend) | Quality degradation varies by backend and model. `sage_attn` is the most general; sparse backends (`sliding_tile_attn`, `video_sparse_attn`, etc.) are video-model-specific and may require config files (e.g. `--mask-strategy-file-path` for STA). Requires corresponding packages installed. | -| **Cache-DiT** | `SGLANG_CACHE_DIT_ENABLED=true` + `--cache-dit-config ` | Caches intermediate residuals across denoising steps and skips redundant computations via a Selective Computation Mask (SCM). | ~1.5–2x on supported models | Quality depends on SCM config. Incompatible with `--dit-layerwise-offload`. Requires correct per-model config YAML. | +| **Cache-DiT** | Native: `SGLANG_CACHE_DIT_ENABLED=true` plus `SGLANG_CACHE_DIT_*` env vars. Diffusers backend: `--backend diffusers --cache-dit-config ` | Caches intermediate residuals across denoising steps and skips redundant computations via DBCache, TaylorSeer, and optional SCM. | ~1.5-2x on supported models | Quality depends on cache policy. Incompatible with `--dit-layerwise-offload`. Do not pass `--cache-dit-config` for native SGLang tuning unless you are intentionally using the diffusers backend flow. | | **Quantized Models (Nunchaku / SVDQuant)** | `--enable-svdquant --transformer-weights-path ` + optional `--quantization-precision int4\|nvfp4`, `--quantization-rank 32` | W4A4-style quantization via [Nunchaku](https://nunchaku.tech). Reduces DiT weight memory by ~4x. Precision/rank can be auto-inferred from weight filename or set explicitly. | ~1.5–2x compute speedup | Lossy quantization; quality depends on rank and precision. Requires pre-quantized weights. Ampere (SM8x) or SM12x only (no Hopper SM90). Higher rank = better quality but more memory. | | **Pre-quantized Transformer Override** | `--transformer-path ` / `--transformer-weights-path ` | Load a quantized transformer component or raw transformer weights. For converted ModelOpt FP8/NVFP4 directories, prefer `--transformer-path`; use `--transformer-weights-path` for weight-only artifacts the model loader expects. | ~1.3–1.5x compute (dtype dependent) | Requires a validated quantized transformer override, such as one produced by the ModelOpt helper tools. Quality is usually slightly worse than BF16 and depends on the format, fallback layers, and calibration scope. | | **Component Precision Override** | `--dit-precision fp16`, `--vae-precision fp16\|bf16` | On-the-fly dtype conversion for individual components. E.g. convert a BF16 model to FP16 at load time, or run VAE in BF16 instead of FP32. | Reduces memory; FP16 can be faster on some GPUs | May affect numerical stability. VAE is FP32 by default for accuracy; lowering it is lossy. DiT defaults to BF16. | @@ -82,21 +82,35 @@ For Wan2.2 specifically: - for pure latency tuning, benchmark pure Ulysses too, for example `--ulysses-degree=4 --ring-degree=1` on 4 GPUs - on 8 GPUs, compare pure `--ulysses-degree=8` against `--enable-cfg-parallel --ulysses-degree=4` -### Nightly-aligned model, single GPU: LTX-2 two-stage +### Nightly-aligned model, 2 GPUs: LTX-2 two-stage ```bash sglang generate --model-path Lightricks/LTX-2 \ --pipeline-class-name LTX2TwoStagePipeline \ - --prompt "A beautiful sunset over the ocean" \ - --negative-prompt "shaky, glitchy, low quality, worst quality, deformed, distorted, disfigured, motion smear, motion artifacts, fused fingers, bad anatomy, weird hand, ugly, transition, static." \ - --width 1536 --height 1024 \ - --num-frames 121 --fps 24 \ - --seed 1234 --num-gpus 1 \ + --prompt "A cat and a dog baking a cake together in a kitchen." \ + --width 768 --height 512 \ + --num-frames 121 --num-inference-steps 50 --guidance-scale 4.0 \ + --seed 42 --num-gpus 2 --enable-cfg-parallel \ --enable-torch-compile --warmup --save-output ``` Note: this generate recipe is aligned with the nightly comparison case `ltx2_twostage_t2v`. `LTX2TwoStagePipeline` is a native path and auto-resolves the spatial upsampler plus distilled LoRA from the same model snapshot unless you override them. +### Nightly-aligned model, 2 GPUs: LTX-2.3 TI2V two-stage + +```bash +sglang generate --model-path Lightricks/LTX-2.3 \ + --pipeline-class-name LTX2TwoStagePipeline \ + --prompt "The cat starts walking slowly towards the camera." \ + --image-path "${ASSET_DIR}/cat.png" \ + --width 768 --height 512 \ + --num-frames 121 --num-inference-steps 50 --guidance-scale 4.0 \ + --seed 42 --num-gpus 2 \ + --enable-torch-compile --warmup --save-output +``` + +Note: this matches the nightly comparison case `ltx2.3_twostage_ti2v_2gpus`. Download `${ASSET_DIR}/cat.png` with the benchmark/profile skill before running it. + ### Native baseline, 2 GPUs: LTX-2.3 one-stage ```bash @@ -112,7 +126,7 @@ sglang generate --model-path Lightricks/LTX-2.3 \ Note: use this as the native `LTX2Pipeline` baseline for `LTX-2.3`. It keeps the validated one-stage resolution and explicit `LTX-2.3` sampling defaults, and matches the `ltx23-one-stage` benchmark preset in `sglang-diffusion-benchmark-profile`. -### Benchmark target, 2 GPUs: LTX-2.3 two-stage +### Skill-only stress target, 2 GPUs: LTX-2.3 two-stage high resolution ```bash sglang generate --model-path Lightricks/LTX-2.3 \ @@ -126,7 +140,7 @@ sglang generate --model-path Lightricks/LTX-2.3 \ --enable-torch-compile --warmup --save-output ``` -Note: this is the recommended benchmark command for the `LTX-2.3` two-stage path. It uses the native `LTX2TwoStagePipeline` and matches the `ltx23-two-stage` benchmark preset in `sglang-diffusion-benchmark-profile`. +Note: this is a high-resolution stress target for the native `LTX-2.3` two-stage path. It matches the skill-only `ltx23-two-stage` benchmark preset, not a nightly comparison case. ### Maximum speed, image model, single GPU, lossless @@ -149,26 +163,64 @@ sglang generate --model-path \ --prompt "..." --save-output ``` -### Maximum speed, lossy (SageAttention + Cache-DiT) +### Maximum speed, lossy native path (SageAttention + Cache-DiT) ```bash SGLANG_CACHE_DIT_ENABLED=true sglang generate --model-path \ --attention-backend sage_attn \ - --cache-dit-config \ + --dit-layerwise-offload false \ --enable-torch-compile --warmup \ + --prompt "..." --save-output +``` + +Add native Cache-DiT knobs such as `SGLANG_CACHE_DIT_SCM_PRESET=medium`, +`SGLANG_CACHE_DIT_RDT=0.24`, or `SGLANG_CACHE_DIT_TAYLORSEER=true` only after +you have a BF16 baseline output to compare against. + +For a diffusers-backend Cache-DiT YAML/JSON config baseline, make the fallback +explicit: + +```bash +sglang generate --backend diffusers --model-path \ + --cache-dit-config \ --dit-layerwise-offload false \ --prompt "..." --save-output ``` --- +## Model-Specific Starting Points + +Use these as first commands to benchmark, not as universal winners. + +| Model family | First performance shape | Starting flags | Notes | +|---|---|---|---| +| FLUX.1 / FLUX.2 image | 1024x1024, 50 steps, 1 GPU | `--enable-torch-compile --warmup --dit-layerwise-offload false` | `black-forest-labs/FLUX.*` repos are gated; for FP8/NVFP4 use validated `--transformer-path` or `--transformer-weights-path` flows from the quant skill. | +| Qwen-Image / Qwen-Image-Edit | 1024x1024, 50 steps, 1 GPU | `--enable-torch-compile --warmup`; optionally native `SGLANG_CACHE_DIT_ENABLED=true` | Cache-DiT is lossy. For edit tasks, keep reference image, seed, and output size fixed. | +| Z-Image-Turbo | 1024x1024, 9 steps, guidance 4.0 | `--enable-torch-compile --warmup` | Mainline has Z-Image tanh/gate norm fusions; PR #21912 tracks FP8 plus CUDA Graph work. | +| Wan2.2 A14B T2V/I2V | 720p, 81 frames | Nightly: `--num-gpus 4 --enable-cfg-parallel --ulysses-degree 2 --text-encoder-cpu-offload --pin-cpu-memory` | For lowest latency, also benchmark pure Ulysses on the same GPUs. | +| Wan2.2 TI2V 5B | 720p, 81 frames, 1 GPU | `--enable-torch-compile --warmup` | Keep the input image and motion prompt fixed when comparing sparse attention or Cache-DiT. | +| LTX-2 / LTX-2.3 | 768x512, 121 frames, 2 GPUs | `--pipeline-class-name LTX2TwoStagePipeline --enable-torch-compile --warmup`; LTX-2 nightly also uses `--enable-cfg-parallel` | Use the benchmark/profile skill presets for exact nightly alignment. PRs #22441, #24025, and #23736 track additional LTX2 perf/parallel work. | +| HunyuanVideo | 848x480 or 720p class video | `--text-encoder-cpu-offload --pin-cpu-memory --enable-torch-compile --warmup` | Check VAE decode separately. GroupNorm+SiLU is default-eligible in mainline when wrapper guards pass; use `bench_group_norm_silu.py` when VAE residual blocks are hot. | +| JoyAI-Image-Edit | 1024-class TI2I, 40 steps, guidance 4.0 | Start from the CI/default TI2I path for `jdopensource/JoyAI-Image-Edit-Diffusers`; add `--warmup` and benchmark `--enable-torch-compile` separately | Newly supported image-edit path. Keep the input image, prompt, seed, and output size fixed; sequence shard is auto-enabled for Joy pipelines. | +| MOVA / Helios | Use the benchmark/profile presets first | `--enable-torch-compile --warmup`; pin offload flags explicitly | PR #20530 tracks MOVA fused RMSNorm+RoPE; PR #24059 tracks Helios fused norm modulation. | + +## Open PR Watchlist + +As of 2026-05-02, these performance PRs were open. Treat them as direction and +prior art until merged: + +- Fusion/kernel: #24025 LTX2 QK norm, #24059 Helios norm modulation, #24117 Z-Image packed QKV, #19488 Wan elementwise cross-block fusion, #19249 Z-Image gate/norm fusion, #20429 Qwen-Image layernorm/modulation, #20530 MOVA RMSNorm+RoPE. +- VAE/decode: #22531 LTX2 parallel VAE, #20927 batched tiled VAE decode. +- Runtime/parallel/cache: #22805 FLUX.2 packed QKV for A2A, #21742 hybrid attention schedule, #24053 USP replicated-prefix fix, #21613 TeaCache refactor, #24227 WanVideo TeaCache fix, #18764 dynamic batching, #24200 disaggregated diffusion. + ## Tips - **Benchmarking**: always use `--warmup` and look for the line ending with `(with warmup excluded)` for accurate timing. - **Perf dump**: use `--perf-dump-path result.json` to save structured metrics, then compare with `python python/sglang/multimodal_gen/benchmarks/compare_perf.py baseline.json result.json`. - **Offload tuning**: after the first request, the runtime logs peak GPU memory and which components could stay resident. Use this to decide which `--*-cpu-offload` flags to disable. -- **Backend selection**: `--backend sglang` (default, auto-detected) enables all native optimizations (fused kernels, SP, etc.). `--backend diffusers` falls back to vanilla Diffusers pipelines but supports `--cache-dit-config` and diffusers attention backends. +- **Backend selection**: `--backend sglang` (default, auto-detected) enables native optimizations (fused kernels, SP, native Cache-DiT env knobs, etc.). `--backend diffusers` falls back to Diffusers pipelines and is the path that accepts `--cache-dit-config` plus diffusers attention backend names. - **Wan2.2-I2V sizing**: explicit `--width/--height` on `Wan2.2-I2V-A14B` control the target area while preserving the condition-image aspect ratio. -- **Mainline diffusion fast paths**: before proposing a new kernel or overlap scheme, check `sglang-diffusion-benchmark-profile/existing-fast-paths.md`. It covers Z-Image residual-form modulation, fused diffusion `QK norm + RoPE`, and existing multi-GPU overlap families such as Ulysses / USP and turbo-layer async all-to-all. +- **Mainline diffusion fast paths**: before proposing a new kernel or overlap scheme, check `sglang-diffusion-benchmark-profile/existing-fast-paths.md`. It covers GroupNorm+SiLU, Z-Image residual-form modulation, fused diffusion `QK norm + RoPE`, packed QKV/NVFP4 expectations, and existing multi-GPU overlap families such as Ulysses / USP and turbo-layer async all-to-all. - **NVFP4 trace interpretation**: on FLUX.2 NVFP4 and Nunchaku-style checkpoints, packed QKV is expected. SGLang intentionally uses fused projection modules such as `to_qkv` / `to_added_qkv` instead of separate `to_q` / `to_k` / `to_v`, so a split-QKV trace usually means the quantized path did not engage rather than a brand new fusion opportunity. - **Hotspot workflow split**: use `sglang-diffusion-benchmark-profile` to prove and classify a slowdown with perf dumps plus `torch.profiler`; hand concrete kernel work to `sglang-diffusion-ako4all-kernel` or another specialized optimization skill instead of expanding the benchmark skill.