diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-add-model/SKILL.md b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-add-model/SKILL.md index fd786b0ce..284021205 100644 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-add-model/SKILL.md +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-add-model/SKILL.md @@ -64,7 +64,12 @@ points cannot silently diverge. | Adding a variant of an existing model that shares most logic | **Modular** — reuse existing stages, customize via PipelineConfig callbacks | | A specific pre-processing step needs special parallelism or profiling isolation | **Modular** — extract that step as a dedicated stage | -**Key principle (both styles)**: The stage(s) before `DenoisingStage` must produce a `Req` batch object with all the standard tensor fields that `DenoisingStage` expects (latents, timesteps, prompt_embeds, etc.). As long as this contract is met, the pipeline remains composable regardless of which style you use. +**Key principle (standard-denoise styles)**: For Hybrid and Modular pipelines, +the stage(s) before `DenoisingStage` must produce a `Req` batch object with all +the standard tensor fields that `DenoisingStage` expects (latents, timesteps, +prompt embeds, and model-specific conditioning). Native task-contract pipelines +may own a different denoise/decode contract; keep that divergence explicit and +covered by request-contract tests. --- @@ -115,7 +120,7 @@ Once you have the reference code, study it thoroughly: **Before creating any new files, check whether an existing pipeline or stage can be reused or extended.** Only create new pipelines/stages when the existing ones would require extensive modifications or when no similar implementation exists. Specifically: -1. **Compare the new model's architecture against existing pipelines** before creating files. Current native families include MiniMax-H3, LTX-2/2.3, HunyuanVideo/FastHunyuan, Wan/FastWan/TurboWan/LingBot World, MOVA, FLUX/FLUX.2/Klein, Z-Image, Qwen-Image/edit/layered, GLM-Image, SD3, Hunyuan3D, Helios, Cosmos3, SANA/SANA-WM, FireRed, ERNIE-Image, JoyAI, and Ideogram4. If the new model shares most of its structure with an existing one (e.g., same text encoders, similar latent format, compatible denoising loop), prefer: +1. **Compare the new model's architecture against existing pipelines** before creating files. Current native families include MiniMax-H3, Krea-2, LTX-2/2.3, HunyuanVideo/FastHunyuan, Wan/FastWan/TurboWan/LingBot World/LingBot Video MoE, MOVA, FLUX/FLUX.2/Klein, Z-Image, Qwen-Image/edit/layered, GLM-Image, SD3, Hunyuan3D, Helios, Cosmos3, SANA/SANA-WM, FireRed, ERNIE-Image, JoyAI, and Ideogram4. If the new model shares most of its structure with an existing one (e.g., same text encoders, similar latent format, compatible denoising loop), prefer: - Adding a new config variant to the existing pipeline rather than creating a new pipeline class - Reusing the existing `BeforeDenoisingStage` with minor parameter differences - Using `add_standard_t2i_stages()` / `add_standard_ti2i_stages()` / `add_standard_ti2v_stages()` if the model fits standard patterns @@ -582,6 +587,7 @@ After implementation, **you must verify that the generated output is not noise** | Hunyuan3D | `runtime/pipelines/hunyuan3d_pipeline.py` | `stages/model_specific_stages/hunyuan3d/` | `configs/pipeline_configs/hunyuan3d.py` | | SANA-WM | `runtime/pipelines/sana_wm_pipeline.py`, `sana_wm_realtime_pipeline.py` | `stages/model_specific_stages/sana_wm/` | `configs/pipeline_configs/sana_wm.py` | | LingBot World realtime | `runtime/pipelines/lingbot_world_causal_dmd_pipeline.py` | `stages/model_specific_stages/lingbot_world/` | `configs/pipeline_configs/lingbot_world.py` | +| Krea-2 | `runtime/pipelines/krea2.py` | `stages/model_specific_stages/krea2.py` | `configs/pipeline_configs/krea2.py` | ### Modular Style (when standard stages fit well) @@ -599,6 +605,7 @@ After implementation, **you must verify that the generated output is not noise** | Helios | `runtime/pipelines/helios_pipeline.py` | Video pipeline family with custom denoising and decoding stages | | FireRed/JoyAI image edit | `runtime/pipelines/qwen_image.py`, `runtime/pipelines/joy_image.py` | FireRed reuses Qwen edit-plus config; JoyAI has its own edit pipeline | | Wan | `runtime/pipelines/wan_pipeline.py` | Uses `add_standard_ti2v_stages()` | +| LingBot Video MoE 30B | `runtime/pipelines/lingbot_video_moe.py` | Uses a model-specific structured-JSON text-encoding stage, then standard latent/timestep preparation, denoising, and decoding | ### Native Task-Contract Style (coupled multimodal requests) @@ -612,7 +619,7 @@ After implementation, **you must verify that the generated output is not noise** Before submitting, verify: -**Common (both styles):** +**Common (all styles):** - [ ] **Pipeline file** exists at `runtime/pipelines/{model_name}.py` with `EntryClass` - [ ] **PipelineConfig** at `configs/pipeline_configs/{model_name}.py` - [ ] **SamplingParams** at `configs/sample/{model_name}.py` diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/SKILL.md b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/SKILL.md index 94f82b00d..ee9459409 100644 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/SKILL.md +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/SKILL.md @@ -23,6 +23,9 @@ Before running any benchmark, profiler, or kernel-validation command: - verify the repo is writable - export `HF_TOKEN` before using gated Hugging Face models such as `black-forest-labs/FLUX.*` - export `FLASHINFER_DISABLE_VERSION_CHECK=1` +- set `SGLANG_DIFFUSION_SYNC_STAGE_PROFILING=1` when comparing stage-level + denoise/decode timings; the preset helper sets it by default unless the + caller explicitly overrides it - choose idle GPU(s) before starting perf work ## Native Backend Gate @@ -45,7 +48,7 @@ If any benchmark, perf-dump, or `torch.profiler` command prints one of those sig - [benchmark-and-profile.md](benchmark-and-profile.md) — canonical denoise benchmark, perf dump, and `torch.profiler` workflow; uses checked-in nightly-aligned presets plus current-source extras such as MiniMax-H3 joint video/audio T2VA, FLUX.2 Klein, Cosmos3, Ideogram4, ERNIE/GLM/SANA image models, FastWan2.2, `LTX-2.3` one-stage/two-stage/HQ, HunyuanVideo, MOVA, Helios, JoyAI/FireRed image edit, and Hunyuan3D shape - [existing-fast-paths.md](existing-fast-paths.md) — map bottlenecks to existing fused kernels, packed QKV paths, fused `QK norm + RoPE`, distributed overlap patterns, and open optimization PRs before proposing new code - [scripts/diffusion_skill_env.py](scripts/diffusion_skill_env.py) — preflight helper: repo root discovery via `sglang.__file__`, write-access probe, benchmark/profile output directories, idle GPU selection -- [scripts/bench_diffusion_denoise.py](scripts/bench_diffusion_denoise.py) — end-to-end denoise benchmark preset runner via `sglang generate`; supports `--no-torch-compile`, forces the H3 preset to its eager consistency mode, validates nightly preset drift with `--validate-nightly-alignment`, and saves perf dumps by label for `compare_perf.py` +- [scripts/bench_diffusion_denoise.py](scripts/bench_diffusion_denoise.py) — end-to-end denoise benchmark preset runner via `sglang generate`; supports `--no-torch-compile`, forces the H3 preset to its eager consistency mode, enables synchronized stage attribution for perf dumps, validates nightly preset drift with `--validate-nightly-alignment`, and saves perf dumps by label for `compare_perf.py` ## Opportunity Discovery Rule @@ -58,6 +61,10 @@ Always rule out these existing families first: - SANA packed self-attention Q/K/V and cross-attention K/V GEMMs - MiniMax-H3 indexed modulation, fused QK norm + RoPE, packed Ulysses QKV, USP relayout, and batched TP AdaLN collectives +- bit-exact diffusion adaLN modulation and fused LayerNorm + modulation for + FLUX.1, GLM-Image, and SANA +- request-scoped `quality=high` DiT and VAE fast paths +- Wan causal-VAE cache/padding and DupUp3D data-movement fusions - fused diffusion `QK norm + RoPE` - LTX2 split RoPE - LTX2 residual-gate add @@ -67,6 +74,7 @@ Always rule out these existing families first: - Ulysses / USP attention overlap - turbo-layer async all-to-all overlap - `torch.compile` compute / communication reorder +- breakable CUDA graph capture for supported fixed-resolution pipelines - dual-stream diffusion execution If the user explicitly requires `torch.compile` to stay off, do not use the diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/benchmark-and-profile.md b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/benchmark-and-profile.md index 69ba1e0e5..2a1aba3d9 100644 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/benchmark-and-profile.md +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/benchmark-and-profile.md @@ -37,6 +37,9 @@ python3 "$ENV_PY" check-write-access >/dev/null export HF_TOKEN= # required for gated repos such as black-forest-labs/FLUX.* export FLASHINFER_DISABLE_VERSION_CHECK=1 +# Required for correctly attributed stage-level denoise/decode timings. The +# checked-in benchmark helper sets this by default unless you explicitly set 0. +export SGLANG_DIFFUSION_SYNC_STAGE_PROFILING=1 # Leave CUDA_VISIBLE_DEVICES unset to let the preset helper select the number # of idle GPUs it requires. For manual runs, set --count to that command's # exact --num-gpus value. @@ -83,6 +86,8 @@ Environment notes: - all commands below assume you are inside the configured diffusion container shell - export `HF_TOKEN` before any gated Hugging Face model run - export `FLASHINFER_DISABLE_VERSION_CHECK=1` before any benchmark or profiler run +- keep `SGLANG_DIFFUSION_SYNC_STAGE_PROFILING=1` for stage-level comparisons; + without it, asynchronous GPU work can be charged to a later stage - re-run `print-idle-gpus` before each perf command if GPU availability may have changed - keep benchmark commands within 4 GPUs or fewer @@ -130,6 +135,10 @@ PYTHONPATH=python python3 "$BENCH_PY" \ --output-dir "${BENCH_DIR}" ``` +The helper sets `SGLANG_DIFFUSION_SYNC_STAGE_PROFILING=1` for accurate stage +attribution. Set it to `0` explicitly only when collecting an e2e-only run and +do not compare its per-stage values with synchronized results. + Keep `torch.compile` off when the task requires it: ```bash @@ -500,10 +509,17 @@ Always keep: - denoise latency - end-to-end latency - peak GPU memory -- exact command line, model shape, dtype, and GPU topology +- exact command line, model shape, dtype, request `quality`, GPU topology, and + whether synchronized stage profiling was enabled Never keep a perf dump produced after a diffusers-backend fallback. +Stage durations are host wall times around asynchronous GPU launches unless +`SGLANG_DIFFUSION_SYNC_STAGE_PROFILING=1`. Without the sync, queued denoise +work can leak into the next blocking stage and inflate `DecodingStage` by 2-3x. +Use synchronized dumps for denoise/decode attribution and keep the setting +identical in every before/after pair. + ## `torch.profiler` Workflow ### 1. Establish the baseline @@ -604,9 +620,13 @@ the known mainline families. | `fused_inplace_qknorm_rope` missing, but separate qk norm plus rope show up | Check whether the fused diffusion `QK norm + RoPE` path should have engaged | | `to_q -> to_k -> to_v` on NVFP4 or Nunchaku FLUX-family checkpoints | Treat as a packed-QKV fast-path miss or checkpoint-format mismatch | | `zimage_rmsnorm_scale` or `zimage_rmsnorm_tanh_residual` missing on Z-Image | Check the bf16-native Triton eligibility guards before proposing a new fusion | +| FLUX.1, GLM-Image, or SANA shows separate LayerNorm plus adaLN elementwise kernels | Check the bit-exact `modulate_scale_shift` and `fused_layernorm_modulate` guards/self-test before proposing another norm fusion | +| `quality=high` shows the same FLUX/GLM DiT or FLUX-family/Wan VAE chain as `lossless` | Check whether the request-scoped quality gate mounted and whether every site passed its all-or-nothing compatibility checks | | LTX-2 split RoPE appears as a long PyTorch elementwise chain | Check the `apply_ltx2_split_rotary_emb` Triton path and its shape guards | +| Wan decode is dominated by causal `cat + pad + contiguous`, feature-cache copies, or `repeat_interleave + permute + add` | Check the bit-exact Wan causal-cache and DupUp3D data-movement kernels before writing a new decoder kernel | | masked attention spends time packing/unpacking Q/K/V | Check whether fused varlen USP pack/scatter should have engaged | | `all_to_all`, ring attention, or async A2A dominate | Classify against Ulysses, USP, or turbo-layer overlap first | +| Fixed-resolution image/video traces show many small launch gaps | Check supported breakable CUDA graph capture, declared warmup resolutions, and text buckets before adding a new graph mechanism | | H3 shows separate indexed gather + scale/shift, QK norm + RoPE, or three Q/K/V Ulysses relayouts | Check H3's indexed-modulation, fused QK-norm+RoPE, packed Ulysses-QKV, and USP relayout guards before writing a new kernel | | H3 TP traces show one AdaLN collective per block | Check the batched TP AdaLN projection/all-gather path in `minimax_h3.py` before attempting communication overlap | | split `fc1 -> gelu -> quant -> fc2.lora_down` on Nunchaku FLUX | Treat as a missing fused GELU MLP path | @@ -633,6 +653,7 @@ This skill intentionally stops here. It tells you whether you are looking at: - [ ] fixed-shape baseline perf dump saved - [ ] fixed-shape new perf dump saved +- [ ] request `quality` and `SGLANG_DIFFUSION_SYNC_STAGE_PROFILING` match - [ ] `compare_perf.py` table generated - [ ] one representative `torch.profiler` trace saved - [ ] hotspot classified against `existing-fast-paths.md` diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/existing-fast-paths.md b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/existing-fast-paths.md index d0eb4276b..7f8eee21a 100644 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/existing-fast-paths.md +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/existing-fast-paths.md @@ -11,10 +11,16 @@ framework-specific optimization workflow. - `python/sglang/multimodal_gen/runtime/layers/fused_scale_shift_gate.py` - `python/sglang/multimodal_gen/runtime/layers/rotary_embedding/utils.py` - `python/sglang/kernels/ops/diffusion/triton/scale_shift.py` +- `python/sglang/kernels/ops/diffusion/modulate_scale_shift.py` +- `python/sglang/kernels/ops/diffusion/fused_ln_modulate.py` +- `python/sglang/kernels/ops/diffusion/quality_gate.py` - `python/sglang/kernels/ops/diffusion/group_norm_silu.py` - `python/sglang/kernels/ops/diffusion/triton/group_norm_silu.py` +- `python/sglang/kernels/ops/diffusion/triton/group_norm_silu_twopass.py` - `python/sglang/kernels/ops/diffusion/triton/norm.py` - `python/sglang/kernels/ops/diffusion/triton/rmsnorm_onepass.py` +- `python/sglang/kernels/ops/diffusion/triton/layernorm_modulate.py` +- `python/sglang/kernels/ops/diffusion/triton/native_bf16_rmsnorm.py` - `python/sglang/kernels/ops/diffusion/triton/zimage_native_norm.py` - `python/sglang/kernels/ops/diffusion/triton/rotary.py` - `python/sglang/kernels/ops/diffusion/triton/ltx2_rotary.py` @@ -26,13 +32,24 @@ framework-specific optimization workflow. - `python/sglang/kernels/ops/diffusion/residual_gate_add.py` - `python/sglang/kernels/jit/csrc/diffusion/residual_gate_add.cuh` - `python/sglang/kernels/ops/diffusion/triton/varlen_pack_pad.py` +- `python/sglang/kernels/ops/diffusion/triton/wan_causal_cache.py` - `python/sglang/kernels/ops/diffusion/cutedsl/scale_residual_norm_scale_shift.py` +- `python/sglang/multimodal_gen/runtime/models/vaes/fast_path_gate.py` +- `python/sglang/multimodal_gen/runtime/models/vaes/flux2_vae_cuda_opt.py` +- `python/sglang/multimodal_gen/runtime/models/vaes/wan_vae_cuda_opt.py` +- `python/sglang/multimodal_gen/runtime/breakable_cuda_graph/runner.py` - `test/registered/kernels/ops/diffusion/test_qwen_image_modulation.py` - `test/registered/kernels/ops/diffusion/test_group_norm_silu.py` - `test/registered/kernels/ops/diffusion/test_residual_gate_add.py` - `test/registered/kernels/ops/diffusion/test_varlen_pack_pad.py` - `test/registered/kernels/ops/diffusion/test_varlen_uspattn_equivalence.py` -- `test/registered/kernels/ops/diffusion/test_zimage_native_norm.py` +- `test/registered/kernels/ops/diffusion/test_native_bf16_rmsnorm.py` +- `test/registered/kernels/ops/diffusion/test_flux_ln_modulate.py` +- `test/registered/kernels/ops/diffusion/test_glm_image_ln_modulate.py` +- `test/registered/kernels/ops/diffusion/test_sana_ln_modulate.py` +- `test/registered/kernels/ops/diffusion/test_quality_gate.py` +- `test/registered/kernels/ops/diffusion/test_wan_causal_cache.py` +- `test/registered/kernels/ops/diffusion/test_stage_profiler_sync.py` - `test/registered/kernels/benchmark/diffusion/bench_qwen_image_modulation.py` - `test/registered/kernels/benchmark/diffusion/bench_group_norm_silu.py` - `test/registered/kernels/benchmark/diffusion/bench_residual_gate_add.py` @@ -60,33 +77,72 @@ framework-specific optimization workflow. - Constraints: `D % 256 == 0` and `D <= 8192`. `x/residual/gate/scale/shift` must pass shape and stride validation. Dtypes limited to fp16/bf16/fp32. - Behavior: CuTe DSL compilation cached by `(dtype, ndim, D, norm_type)`. `None` tensors replaced by scalar placeholders. If constraints fail, `layernorm.py` warns and falls back to native PyTorch. -3. Z-Image bf16-native RMSNorm modulation (Triton) +3. Bit-exact adaLN modulation and LayerNorm + modulation +- Kernels: `modulate_scale_shift`, `fused_layernorm_modulate`, and + `fused_qk_head_layernorm`. +- Locations: `modulate_scale_shift.py`, `triton/layernorm_modulate.py`, + `runtime/models/dits/flux.py`, `glm_image.py`, and `sana.py`. +- Use cases: + - `x * (1 + scale[:, None]) + shift[:, None]` as one JIT CUDA launch. + - BF16 `LayerNorm(x) * (1 + scale) + shift` as one Triton launch that + reproduces the active aten BF16 reduction and rounding order. + - GLM-Image per-head Q/K LayerNorm with the same aten-compatible reduction. +- Constraints: the JIT modulation path requires aligned contiguous CUDA + fp16/bf16 BLC inputs with `[B, D]` scale/shift. The Triton LayerNorm path is + BF16-specific and only claims bit-exactness for its guarded aten dispatch; + FLUX/GLM/SANA run a live eager equality check and fail closed on mismatch. +- Validation: `test_flux_ln_modulate.py`, `test_glm_image_ln_modulate.py`, + `test_sana_ln_modulate.py`, `test_modulate_scale_shift.py`, and + `test_fused_ln_modulate.py`. +- Workflow rule: if these models show separate norm and modulation kernels, + check dtype, alignment, shape, BCG/compile context, and the one-time equality + self-test before proposing another fusion. + +4. Request-scoped `quality=high` fusion gates +- Locations: `quality_gate.py`, `fused_ln_modulate.py`, `denoising.py`, + `decoding.py`, `fast_path_gate.py`, `flux2_vae_cuda_opt.py`, and + `wan_vae_cuda_opt.py`. +- Behavior: `quality="lossless"` is the default exact reference path. + `quality="high"` may mount model-owned, validated but non-bit-exact DiT + fusions and decode-scoped VAE rewrites. Mounting is all-or-nothing per + transformer/fusion family; VAE gates reset after every decode. +- Current families include FLUX affine-folded LN+modulate / fused GELU sites, + GLM-Image fused GELU sites, generic KL VAE decoder rewrites used by + FLUX.1/FLUX.2/Z-Image/SD3, and Wan VAE RMSNorm+SiLU. +- Do not confuse request `--quality` with `--output-quality`, which controls + output-file compression rather than model math. +- Validation: `test_quality_gate.py`, `test_fused_ln_modulate.py`, + `test_flux2_vae_fastpath.py`, `test_wan_vae_fastpath.py`, and + `test_vae_fast_path_gate.py`. + +5. Z-Image bf16-native RMSNorm modulation (Triton) - Kernels: `zimage_rmsnorm_scale`, `zimage_rmsnorm_tanh_residual` -- Locations: `triton/zimage_native_norm.py`, `zimage.py` +- Locations: `triton/native_bf16_rmsnorm.py`, compatibility exports in + `triton/zimage_native_norm.py`, and `zimage.py` - Use cases: - `y = rmsnorm(x) * scale` - `y = residual + tanh(gate) * rmsnorm(x)` - Constraints: CUDA bf16 tensors, contiguous weights, flattenable row strides, compatible modulation row counts, and `D <= 8192`. -- Validation: `test/registered/kernels/ops/diffusion/test_zimage_native_norm.py` +- Validation: `test/registered/kernels/ops/diffusion/test_native_bf16_rmsnorm.py` - Behavior: the kernels preserve Z-Image's native bf16 arithmetic. They return `None` when an eligibility guard fails, and the runtime wrapper executes the native PyTorch formula. -4. Triton LayerNorm/RMSNorm fusion +6. Triton LayerNorm/RMSNorm fusion - Kernels: `rms_norm_fn`, `layer_norm_fn`, `norm_infer` - Locations: `triton/norm.py`, `layernorm.py` - Use cases: fp32 RMSNorm with residual/dropout/rowscale/x1 branches, and inference-friendly `norm_infer`. - Constraints: last dim must be contiguous, and `N * element_size < 64KB`. - Validation: `test/registered/kernels/ops/layernorm/test_rmsnorm.py`. -5. Triton one-pass RMSNorm (small hidden size fast path) +7. Triton one-pass RMSNorm (small hidden size fast path) - Kernel: `triton_one_pass_rms_norm` - Locations: `triton/rmsnorm_onepass.py`, `layernorm.py` - Use case: `hidden_size <= 128` in `RMSNorm.forward_cuda`. - `torch.compile` note: keep this path behind the custom-op wrapper in `rmsnorm_onepass.py`; direct `wrap_triton` can recompile on dynamic row counts. -6. Triton RoPE fusion +8. Triton RoPE fusion - Kernel: `apply_rotary_embedding` - Locations: `triton/rotary.py`, `rotary_embedding/utils.py` - Use case: GPT-J style RoPE when not Neox. @@ -94,14 +150,14 @@ framework-specific optimization workflow. - NPU fallback: `npu_fallback.apply_rotary_embedding_native`. - Validation: `test/registered/kernels/ops/attention/test_rope.py`. -7. LTX2 split RoPE fusion +9. LTX2 split RoPE fusion - Kernel: `apply_ltx2_split_rotary_emb` - Locations: `triton/ltx2_rotary.py`, `runtime/models/dits/ltx_2.py` - Use case: LTX-2 split rotary embedding over `[B, S, num_heads * head_dim]` with separate `cos` and `sin` tensors. - Constraints: `cos` and `sin` shapes must match `[B, H, S, head_dim / 2]`, and `inner_dim == H * head_dim`. - Workflow rule: if LTX-2 traces show a large split-RoPE PyTorch chain, check whether the LTX2-specific Triton path was disabled by shape or dtype before proposing a new RoPE kernel. -8. LTX2 residual-gate add fusion +10. LTX2 residual-gate add fusion - Kernel: `diffusion_residual_gate_add` - Locations: `diffusion/residual_gate_add.py`, `csrc/diffusion/residual_gate_add.cuh`, `runtime/models/dits/ltx_2.py` - Use case: `residual + update * gate` in LTX2 self-attention, prompt cross-attention, audio/video cross-attention, and feed-forward residual updates. @@ -111,7 +167,7 @@ framework-specific optimization workflow. - Microbench: `test/registered/kernels/benchmark/diffusion/bench_residual_gate_add.py`. - Workflow rule: if LTX2 traces show repeated elementwise `mul` + `add` ladders around attention or MLP residuals, check whether this existing CUDA path was disabled by shape, dtype, contiguity, or a prior runtime failure before proposing another elementwise fusion. -9. MiniMax-H3 indexed AdaLN modulation and gated residual fusion +11. MiniMax-H3 indexed AdaLN modulation and gated residual fusion - Kernels: `indexed_scale_shift_bf16_`, `indexed_gate_bf16_` - Locations: `triton/indexed_modulation.py`, `runtime/models/dits/minimax_h3.py` - Use cases: H3's packed video/audio/text rows select per-token modulation with `combined_indices`; the Triton paths replace `index_select` plus scale/shift or gated residual chains in place. @@ -119,7 +175,7 @@ framework-specific optimization workflow. - Numerical contract: the kernels explicitly reproduce H3's eager BF16 rounding boundaries. Do not replace them with a mathematically equivalent contraction without the H3 consistency check. - Workflow rule: if H3 traces show `index_select` plus elementwise ladders around every block, check dtype, contiguity, and input-reuse eligibility before designing another modulation kernel. -10. MiniMax-H3 packed Ulysses QKV and output relayout +12. MiniMax-H3 packed Ulysses QKV and output relayout - Kernels: `pack_qkv_destination_major`, `usp_merge_heads` - Locations: `triton/ulysses_qkv.py`, `usp_relayout.py`, `runtime/layers/usp.py`, `runtime/models/dits/minimax_h3.py` - Use cases: one destination-major QKV pack plus one collective replaces three separately prepared Ulysses input exchanges; the output JIT kernel replaces `permute(...).contiguous()` when merging gathered heads. @@ -127,7 +183,7 @@ framework-specific optimization workflow. - Related transport: 2-rank, peer-accessible CUDA groups can use the existing IPC A2A transport; larger or unsupported groups fall back to the normal collective path. - Workflow rule: if an H3 Ulysses trace has three Q/K/V preparation ladders or a large output `permute + contiguous`, first prove why these existing guards missed. -11. HunyuanVideo / LTX upsampler GroupNorm + SiLU fusion +13. HunyuanVideo / LTX upsampler GroupNorm + SiLU fusion - Kernel: `triton_group_norm_silu` - Locations: `diffusion/group_norm_silu.py`, `triton/group_norm_silu.py`, `runtime/models/vaes/hunyuanvae.py`, `runtime/models/upsampler/latent_upsampler.py` - Use case: `activation(group_norm(x))` when the activation is non-inplace `nn.SiLU` and the GroupNorm is affine. @@ -136,6 +192,18 @@ framework-specific optimization workflow. - Validation: `test/registered/kernels/ops/diffusion/test_group_norm_silu.py`. - Microbench: `test/registered/kernels/benchmark/diffusion/bench_group_norm_silu.py`. +14. Wan causal-VAE data-movement fusion +- Kernels: `cat_pad_channels_last_3d` and `dup_up3d_add`. +- Locations: `triton/wan_causal_cache.py` and + `runtime/models/vaes/wanvae.py`. +- Use cases: build causal Conv3d input plus the next compact feature cache in + one channels-last-3D pass, and fuse `main + DupUp3D(src)` without + materializing `repeat_interleave + permute().contiguous()` intermediates. +- Numerical contract: these are bit-exact data-movement / same-order-add + replacements and run independently of the `quality=high` Wan RMSNorm+SiLU + path. Unsupported layouts or padding fall back to the aten chain. +- Validation: `test/registered/kernels/ops/diffusion/test_wan_causal_cache.py`. + **Faster CUDA Kernel Usage Points** 1. sgl-kernel RMSNorm and fused add RMSNorm @@ -218,10 +286,15 @@ framework-specific optimization workflow. **Common Entry Points in Diffusion Models** - AdaLN modulation: `LayerNormScaleShift`, `RMSNormScaleShift`, `ScaleResidual*` in `layernorm.py`. +- Bit-exact adaLN modulation / LayerNorm folding: `modulate_scale_shift` and + `fused_layernorm_modulate` through `flux.py`, `glm_image.py`, and `sana.py`. +- Request-scoped high-quality acceleration: `QualityGatedFusion` in + `quality_gate.py`, `_maybe_toggle_quality_fusions` in `denoising.py`, and + `use_vae_fast_path` in `decoding.py`. - Qwen-Image gating: `fuse_layernorm_scale_shift_gate_select01_kernel` and `fuse_residual_layernorm_scale_shift_gate_select01_kernel` through `fused_scale_shift_gate.py` and `qwen_image.py`. - Z-Image native norm modulation: `zimage_rmsnorm_scale` and - `zimage_rmsnorm_tanh_mul_add` in `zimage.py`, backed by - `triton/zimage_native_norm.py`. + `zimage_rmsnorm_tanh_mul_add` in `zimage.py`, backed by the shared + `triton/native_bf16_rmsnorm.py` kernels. - HunyuanVideo VAE and LTX upsampler GroupNorm+SiLU: `apply_group_norm_silu` in `hunyuanvae.py` and `latent_upsampler.py`; default-eligible when wrapper guards pass. - MiniMax-H3 indexed modulation: `_modulate_scale_shift` and `_modulate_gate` in `minimax_h3.py`, backed by `triton/indexed_modulation.py`. - MiniMax-H3 Ulysses relayout: `_usp_input_all_to_all_packed_qkv` and `usp_merge_heads` through `runtime/layers/usp.py`. @@ -229,6 +302,8 @@ framework-specific optimization workflow. - QK norm + RoPE: `apply_qk_norm_rope` in `layernorm.py`; use this path when the model wants fused attention prep instead of separate QK norm and RoPE calls. - LTX2 split RoPE: `apply_ltx2_split_rotary_emb` in `ltx_2.py`. - LTX2 residual-gate add: `_ltx2_residual_gate_add` in `ltx_2.py` wraps the CUDA `diffusion_residual_gate_add` custom op for attention, cross-attention, and MLP residual updates. +- Wan causal VAE: `cat_pad_channels_last_3d` and `dup_up3d_add` in + `wanvae.py`, backed by `triton/wan_causal_cache.py`. - Varlen USP attention: `fused_pack_qkv` and `fused_scatter_to_padded` in `attention/layer.py`. - SANA packed projections: `to_qkv` and `to_kv` in `sana.py`. - Nunchaku fused GELU MLP: `_fused_gelu_mlp` in `flux.py` for quantized FLUX-family checkpoints. @@ -238,10 +313,20 @@ framework-specific optimization workflow. **Existing Overlap / Communication Families** - Ulysses / USP attention: treat `all_to_all`, `ring_attn`, and head / sequence reshards as an existing distributed attention family, not a new overlap idea. +- Cross-node SP: current server args support `--nnodes`, `--node-rank`, and + `--dist-init-addr`. Prefer node-local Ulysses multiplied by cross-node Ring; + keep encoders replicated and verify each model's Ring admission before + treating cross-node transport as a new framework gap. - MiniMax-H3 TP AdaLN: the DiT stacks every block's TP-local AdaLN projection and performs one batched all-gather before the block loop when `_can_batch_block_adaln()` passes. One all-gather per block indicates that this existing batching path missed. - MiniMax-H3 final projections: H3 removes dead text/padding rows before the final TP column gathers and combines video/audio for the SP row gather. Preserve that ordering when optimizing output communication. - Turbo-layer async all-to-all: `all_to_all_single(..., async_op=True)` plus staged waits already form an existing overlap family in `turbo_layer.py`. - TorchInductor compute / communication reorder: `torch._inductor.config.reorder_for_compute_comm_overlap = True` can already partially overlap compiled denoise traces. +- Breakable CUDA graph: `runtime/breakable_cuda_graph/runner.py` captures + fixed-resolution DiT segments around eager attention/collectives for + supported pipelines. It is mutually exclusive with `torch.compile` and + Cache-DiT, requires every served resolution in `--warmup-resolutions`, and + uses `--bcg-text-buckets` for prompt signatures. Check this path before + proposing a second graph-capture mechanism for launch-bound traces. - Dual-stream diffusion models: `use_dual_stream = True` in models such as `hunyuan3d.py` is an existing overlap family. - Workflow rule: if a hotspot is communication-heavy, rule out these in-repo overlap families before proposing a brand new overlap design. @@ -278,6 +363,9 @@ relying on any file path, flag, or claim about whether the work has merged. **Constraints and Fallbacks** - `scale_shift` Triton requires CUDA + contiguous `x`. NPU swaps to native. +- Bit-exact BF16 LayerNorm+modulate requires the guarded aten-compatible shape + and a successful live equality check; `quality=high` affine folding is a + separate non-bit-exact path. - CuTe DSL fused norms require `D % 256 == 0` and `D <= 8192`. - Triton norm kernels error on feature size >= 64KB. - FlashAttention requires fp16/bf16 and SM80+; otherwise SDPA. diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py index 2279e9619..d2e31843d 100755 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py @@ -665,7 +665,11 @@ def _expected_nightly_cli_args(case: dict) -> dict[str, str]: serve_args = shlex.split(case["frameworks"]["sglang"].get("serve_args", "")) parsed_serve_args = _parse_cli_args(serve_args) for flag, value in parsed_serve_args.items(): - if flag in {"enable-torch-compile", "warmup-mode"}: + # Nightly's comparison driver still owns its legacy ``--warmup`` + # switch. It is not a valid ``sglang generate`` flag after the + # warmup-mode migration, so exclude both spellings from preset drift + # validation. + if flag in {"enable-torch-compile", "warmup", "warmup-mode"}: continue expected[flag] = _normalize_cli_value(value) @@ -834,6 +838,11 @@ def run_benchmark_once( env = os.environ.copy() env.setdefault("FLASHINFER_DISABLE_VERSION_CHECK", "1") + # Perf dumps are consumed as stage-attributed denoise measurements. Drain + # the device queue at stage boundaries so asynchronous denoise work cannot + # leak into a later stage (most visibly DecodingStage). An explicit 0 in + # the caller's environment still opts out for e2e-only experiments. + env.setdefault("SGLANG_DIFFUSION_SYNC_STAGE_PROFILING", "1") cfg = MODELS[model_key] for key, value in cfg.get("env", {}).items(): env.setdefault(key, str(value)) diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-modelopt-quant/SKILL.md b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-modelopt-quant/SKILL.md index e76e5b192..3936f4c26 100644 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-modelopt-quant/SKILL.md +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-modelopt-quant/SKILL.md @@ -23,7 +23,12 @@ This skill owns the ModelOpt-to-SGLang bridge. It is not a generic kernel-tuning - Benchmark only when BF16 and quantized commands are identical except for the checkpoint override being tested. - For diffusion FP8, keep `dit_cpu_offload=false`. `dit_layerwise_offload=true` is valid on the fixed path when you want lower DiT residency. - For multi-transformer pipelines, use per-component overrides when different components need different checkpoints. -- For B200 NVFP4 validation, keep backend-sensitive environment variables explicit. Wan2.2 NVFP4 is commonly validated with `SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND=cudnn`; benchmark the default CUTLASS path separately if that is what you are evaluating. +- For B200 NVFP4 validation, keep backend-sensitive environment variables + explicit. The current default is FlashInfer TensorRT-LLM + (`flashinfer_trtllm`); high-resolution Qwen Image can favor + `SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND=cutlass`, while 1024x1024 can + remain BF16-faster. Benchmark the exact shape instead of assuming one backend + or quantized checkpoint wins. - When a branch is missing the validated helper tools, refresh `python/sglang/multimodal_gen/tools/build_modelopt_fp8_transformer.py`, `python/sglang/multimodal_gen/tools/build_modelopt_nvfp4_transformer.py`, and `python/sglang/multimodal_gen/tools/compare_diffusion_trajectory_similarity.py` instead of inventing one-off scripts elsewhere. - After validating a new ModelOpt quant path, update the ModelOpt support matrix in `docs/docs/sglang-diffusion/quantization.mdx` before closing the task. @@ -35,6 +40,7 @@ Read these sources before changing code: - ModelOpt quantization entrypoint: `examples/diffusers/quantization/quantize.py` - ModelOpt diffusers quant presets: `examples/diffusers/quantization/config.py` - SGLang diffusion quant runtime: + - `python/sglang/multimodal_gen/runtime/layers/quantization/modelopt_fp8.py` - `python/sglang/multimodal_gen/runtime/layers/quantization/modelopt_quant.py` - `python/sglang/multimodal_gen/runtime/utils/quantization_utils.py` - `python/sglang/multimodal_gen/runtime/loader/transformer_load_utils.py` @@ -65,7 +71,8 @@ This repo now contains: Validated documentation and CI coverage currently center on these ModelOpt diffusion transformer override families: - FP8: FLUX.1-dev, FLUX.2-dev, Wan2.2, HunyuanVideo, Qwen Image, Qwen Image Edit -- NVFP4: FLUX.1-dev, FLUX.2-dev, Wan2.2 +- NVFP4: FLUX.1-dev, FLUX.2-dev, Wan2.2, Qwen Image, Qwen Image 2512, + Qwen Image Edit, Qwen Image Edit 2511 Treat a new family, a new precision, or a new checkpoint layout as unsupported until it has a documented matrix row and a matching validation story. Current B200 CI also contains an Ideogram4 NVFP4 native load case @@ -77,8 +84,8 @@ Before writing CLI examples, re-read the active branch's `docs/docs/sglang-diffu B200 CI coverage can include loose BF16-vs-quantized quality checks. Inspect the active branch's `run_suite.py` before assuming they are part of the suite; mainline and feature branches may differ. Those checks are intended to catch blank, corrupted, or obviously divergent images, not exact image parity. -Mainline documentation now uses `lmsys/*` for the eight converted ModelOpt -checkpoint repos; the FLUX.2 NVFP4 raw export remains +Mainline documentation now tracks thirteen published ModelOpt checkpoints. +Twelve live under `lmsys/*`; the FLUX.2 NVFP4 raw export remains `black-forest-labs/FLUX.2-dev-NVFP4`. Do not use older `BBuf/*` examples unless you are explicitly testing a historical branch. @@ -305,10 +312,23 @@ sglang generate \ --save-output ``` +Full ModelOpt Diffusers repo example (current Qwen Image NVFP4 path): + +```bash +sglang generate \ + --model-path lmsys/qwen-image-2512-modelopt-nvfp4-sglang \ + --prompt "" \ + --seed \ + --save-output +``` + Guideline: - use the global `--transformer-path` only when the model effectively has one transformer override to apply - use per-component overrides when different backbones need different checkpoints +- use `--model-path` directly for published full ModelOpt Diffusers repos such + as the Qwen Image NVFP4 family; this is different from a transformer-only + override - the preferred CLI form is `---path` - config-expanded forms such as `--component_paths.transformer_2=...` also resolve to the same internal override map @@ -411,6 +431,7 @@ When documenting results: | File | Role | | --- | --- | | `runtime/layers/quantization/__init__.py` | registers diffusion quant methods | +| `runtime/layers/quantization/modelopt_fp8.py` | static per-tensor ModelOpt FP8 path used by flat `quant_method=modelopt` exports | | `runtime/layers/quantization/modelopt_quant.py` | ModelOpt FP8 and NVFP4 runtime loading | | `runtime/utils/quantization_utils.py` | resolves flat ModelOpt configs and reconstructs NVFP4 config from metadata | | `runtime/loader/transformer_load_utils.py` | guards incompatible FP8 offload modes | diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-performance/SKILL.md b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-performance/SKILL.md index 92d921e47..4e4862bec 100644 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-performance/SKILL.md +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-performance/SKILL.md @@ -32,10 +32,13 @@ These options are intended to preserve output quality. In practice, some paths ( | Option | CLI Flag / Env Var | What It Does | Speedup | Limitations / Notes | |---|---|---|---|---| +| **Performance Mode** | `--performance-mode auto\|speed\|memory\|manual` (`--mode` alias) | Applies model-aware residency, FSDP/CFG, and compile defaults without overriding explicit flags. `auto` is the safe default; `speed` favors GPU residency; `memory` favors offload; `manual` leaves performance args explicit. | Fastest way to establish a sensible deployment baseline | `speed` may OOM and enables `torch.compile` only when the model deployment config allows it. Explicit offload/FSDP/parallelism/compile flags win. Use `manual` for controlled A/B benchmarks. | | **torch.compile** | `--enable-torch-compile` | Applies `torch.compile` to the DiT forward pass, fusing ops and reducing kernel launch overhead. | ~1.2–1.5x on denoising | First request is slow (compilation). May cause minor precision drifts due to [PyTorch issue #145213](https://github.com/pytorch/pytorch/issues/145213). Pair with `--warmup-mode request` for best results. | +| **Breakable CUDA Graph** | `--enable-breakable-cuda-graph --warmup-resolutions ` plus optional `--bcg-text-buckets ...` | Captures fixed-resolution DiT segments while leaving attention/collectives eager, reducing launch overhead on supported pipelines. | Large on launch-bound paths; merged SANA and LTX-2 cases show material e2e gains | Mutually exclusive with `torch.compile` and Cache-DiT; BCG takes priority. Every served resolution must be declared for warmup capture. Current support is model-specific (Ideogram4, LTX-2, MiniMax-H3, Qwen-Image, SANA1.5, Z-Image, GLM-Image); benchmark before keeping it. | | **Warmup** | `--warmup-mode request` | Runs dummy forward passes to warm up CUDA caches, JIT, and `torch.compile`. Eliminates cold-start penalty. | Removes first-request latency spike | Adds startup time. Without `--warmup-resolutions`, warmup happens on first request. | | **Warmup Resolutions** | `--warmup-resolutions 256x256 720x720` | Pre-compiles and warms up specific resolutions at server startup (instead of lazily on first request). | Faster first request per resolution | Each resolution adds to startup time. Serving mode only; useful when you know your target resolutions in advance. | | **Multi-GPU (SP)** | `--num-gpus N --ulysses-degree N` | Sequence parallelism across GPUs. Shards sequence tokens (not frames) to minimize padding. | Near-linear scaling with N GPUs | Requires NCCL; inter-GPU bandwidth matters. `ulysses_degree * ring_degree = sp_degree`. For Wan2.2 video, start by benchmarking pure Ulysses before assuming a mixed Ulysses/Ring layout is fastest. | +| **Cross-node SP** | `--nnodes`, `--node-rank`, `--dist-init-addr` with total `--num-gpus`; combine node-local Ulysses with cross-node Ring | Extends sequence parallel groups across multiple nodes. | Capacity and long-sequence scaling beyond one host | Prefer Ulysses within a node and Ring across nodes; all-to-all is usually the least cross-node-friendly. Use `--encoder-parallel replicate` today and verify the model's Ring admission and determinism. MiniMax-H3 is the current end-to-end validated recipe. | | **CFG Parallel** | `--enable-cfg-parallel` | Runs conditional and unconditional CFG branches in parallel across GPUs. For CFG models on multi-GPU, benchmark this against pure Ulysses on your topology instead of assuming one always wins. | Often faster than pure SP for CFG models | Requires `num_gpus >= 2`. Halves the Ulysses group size (e.g. 8 GPU → two 4-GPU groups). Only for models that use CFG. Nightly coverage configs may intentionally use smaller Ulysses groups to keep ring behavior exercised; that does not automatically make them the lowest-latency choice. | | **Layerwise Offload** | `--dit-layerwise-offload` | Async layer-by-layer H2D prefetch with compute overlap. Only ~2 DiT layers reside on GPU at a time, dramatically reducing VRAM. For some video models the copy stream can be almost fully hidden behind compute. | Saves VRAM (40 GB → ~11 GB for Wan A14B); can be near-zero speed cost on the right workload | Enabled by default for Wan/MOVA video models. Incompatible with Cache-DiT. For **image models** or highly parallelized setups (many GPUs, small per-GPU compute), the copy stream may not be fully hidden and can cause slowdown. | | **Offload Prefetch Size** | `--dit-offload-prefetch-size F` | Fine-grained control over layerwise offload: how many layers to prefetch ahead. `0.0` = 1 layer (min VRAM), `0.1` = 10% of layers, `≥1` = absolute layer count. | Tune for cases where default offload has copy stream interference (e.g. image models). 0.05–0.1 is a good starting point. | Values ≥ 0.5 approach no-offload VRAM with worse performance. Use lower values when copy overlap is weak; disable offload when memory allows and latency dominates. | @@ -53,8 +56,13 @@ These options **trade output quality** for speed or VRAM savings. Results will d | Option | CLI Flag / Env Var | What It Does | Speedup | Quality Impact / Limitations | |---|---|---|---|---| +| **Request Quality Fast Paths** | `--quality high` (`lossless` is default) | Mounts model-owned accelerated DiT/VAE paths that are validated for high quality but are not bit-exact to the reference path. | Model- and shape-specific | Support is per model and may be a no-op. Keep `--quality lossless` as the A/B ground truth. Do not confuse this with `--output-quality`, which controls file compression. | | **Approximate Attention** | `--attention-backend sage_attn` / `sage_attn_3` / `sliding_tile_attn` / `video_sparse_attn` / `sparse_video_gen_2_attn` / `vmoba_attn` / `sla_attn` / `sage_sla_attn` | Replaces exact attention with approximate or sparse variants. `sage_attn`: INT8/FP8 quantized Q·K; `sliding_tile_attn`: spatial-temporal tile skipping; others: model-specific sparse patterns. | ~1.5–2x on attention (varies by backend) | Quality degradation varies by backend and model. `sage_attn` is the most general; sparse backends (`sliding_tile_attn`, `video_sparse_attn`, etc.) are video-model-specific and may require config files (e.g. `--mask-strategy-file-path` for STA). Requires corresponding packages installed. | | **Cache-DiT** | Native: `SGLANG_CACHE_DIT_ENABLED=true` plus `SGLANG_CACHE_DIT_*` env vars. Diffusers backend: `--backend diffusers --cache-dit-config ` | Caches intermediate residuals across denoising steps and skips redundant computations via DBCache, TaylorSeer, and optional SCM. | ~1.5-2x on supported models | Quality depends on cache policy. Incompatible with `--dit-layerwise-offload`. Do not pass `--cache-dit-config` for native SGLang tuning unless you are intentionally using the diffusers backend flow. | +| **TeaCache** | `--enable-teacache` (uses model sampling presets) | Reuses residuals when adjacent denoising steps are sufficiently similar. | Model- and threshold-dependent | Approximate and model-specific. Mutually exclusive with Spectrum. Fix prompt/seed/shape/steps and validate temporal consistency, not only single frames. | +| **Spectrum** | `--enable-spectrum` plus optional `--spectrum-*` controls | Forecasts DiT features and skips selected denoising steps. | Defaults target an accuracy/speed tradeoff; aggressive windows can be much faster | Native `sglang generate` only for FLUX.1, Wan, HunyuanVideo, and SD3; not FLUX.2 or server requests. Mutually exclusive with TeaCache. `--debug` adds shadow validation and is not representative latency. | +| **Progressive Resolution** | `--progressive-mode dct_rewind --progressive-levels N --progressive-delta D` | Runs early denoising at lower latent resolution, then spectrally upsamples and switches to the target resolution. | Model- and schedule-dependent | Approximate and pipeline-specific. Keep the switch schedule fixed and compare detail, composition, and temporal stability. | +| **Causal KV-Cache Quantization** | `--kv-cache-quant int4\|int2` plus optional `--kv-cache-quant-*` controls | Compresses completed causal KV-cache chunks with Quant-VideoGen PRQ while keeping the mutable/current chunk and recent chunks in BF16. | Primarily a long-session memory saving | Currently limited to LingBot World realtime causal serving; requires `quant-videogen`. INT4 is the starting point; INT2 saves more memory with more error. It quantizes cache state, not checkpoint weights. | | **Quantized Models (Nunchaku / SVDQuant)** | `--enable-svdquant --transformer-weights-path ` + optional `--quantization-precision int4\|nvfp4`, `--quantization-rank 32` | W4A4-style quantization via [Nunchaku](https://nunchaku.tech). Reduces DiT weight memory by ~4x. Precision/rank can be auto-inferred from weight filename or set explicitly. | ~1.5–2x compute speedup | Lossy quantization; quality depends on rank and precision. Requires pre-quantized weights. Ampere (SM8x) or SM12x only (no Hopper SM90). Higher rank = better quality but more memory. | | **Pre-quantized Transformer Override** | `--transformer-path ` / `--transformer-weights-path ` | Load a quantized transformer component or raw transformer weights. For converted ModelOpt FP8/NVFP4 directories, prefer `--transformer-path`; use `--transformer-weights-path` for weight-only artifacts the model loader expects. | ~1.3–1.5x compute (dtype dependent) | Requires a validated quantized transformer override, such as one produced by the ModelOpt helper tools. Quality is usually slightly worse than BF16 and depends on the format, fallback layers, and calibration scope. | | **Component Precision Override** | `--dit-precision fp16`, `--vae-precision fp16\|bf16` | On-the-fly dtype conversion for individual components. E.g. convert a BF16 model to FP16 at load time, or run VAE in BF16 instead of FP32. | Reduces memory; FP16 can be faster on some GPUs | May affect numerical stability. VAE is FP32 by default for accuracy; lowering it is lossy. DiT defaults to BF16. | @@ -216,6 +224,37 @@ sglang generate --model-path \ Note: for image models, per-layer compute is smaller, so layerwise offload may not fully hide H2D transfer. Disable DiT layerwise and CPU offload if VRAM allows; otherwise a large image DiT can stay resident on CPU and make the denoise loop H2D-bound. +### Launch-bound fixed-resolution path: Breakable CUDA Graph + +```bash +sglang serve --model-path Efficient-Large-Model/SANA1.5_1.6B_1024px_diffusers \ + --performance-mode speed \ + --enable-torch-compile false \ + --enable-breakable-cuda-graph \ + --warmup-resolutions 1024x1024 \ + --port 30010 +``` + +Keep `torch.compile` off, declare every production resolution, and benchmark +the exact prompt-length distribution. Add `--bcg-text-buckets` only when the +default buckets create excessive padding or miss a served prompt signature. + +### Compare request-scoped high-quality fast paths + +```bash +sglang generate --model-path \ + --quality lossless --prompt "..." --seed 42 \ + --perf-dump-path baseline.json --save-output + +sglang generate --model-path \ + --quality high --prompt "..." --seed 42 \ + --perf-dump-path quality-high.json --save-output +``` + +Keep every other flag fixed and compare the generated artifact as well as the +perf dumps. If the model has no registered quality-gated sites, `high` may be a +no-op. + ### Image-edit baselines: JoyAI and FireRed ```bash @@ -317,6 +356,7 @@ Use these as first commands to benchmark, not as universal winners. | FLUX.1 / FLUX.2 image | 1024x1024, runtime-default steps/guidance, 1 GPU | `--enable-torch-compile --warmup-mode request --dit-layerwise-offload false` | `black-forest-labs/FLUX.*` repos are gated; for FP8/NVFP4 use validated `--transformer-path` or `--transformer-weights-path` flows from the quant skill. | | FLUX.2 Klein / Klein Base | 1024x1024, runtime-default steps/guidance, 1 GPU | `--enable-torch-compile --warmup-mode request --dit-layerwise-offload false` | Current registry has `black-forest-labs/FLUX.2-klein-4B`, `FLUX.2-klein-9B`, and base variants. Klein is step-distilled; Klein Base is not. | | Qwen-Image / Qwen-Image-Edit | 1024x1024, runtime-default steps/guidance, 1 GPU | `--enable-torch-compile --warmup-mode request`; optionally native `SGLANG_CACHE_DIT_ENABLED=true` | Cache-DiT is lossy. For edit tasks, keep reference image, seed, and output size fixed. | +| Krea-2 | 1024x1024, distilled `oss_turbo` defaults (8 steps, guidance 1.0) | `--performance-mode speed --warmup-mode request` | Native `krea/Krea-2` text-to-image path with Qwen3-VL text conditioning. The repo may require HF access; keep the 8-step distilled baseline separate from non-turbo sampling experiments. | | Z-Image / Z-Image-Turbo | 1024x1024, runtime-default steps/guidance, 1 GPU | `--enable-torch-compile --warmup-mode request` | Keep base Z-Image separate from Turbo: base uses 50-step CFG defaults, Turbo uses 9-step zero-CFG defaults. Mainline has bf16-native Triton RMSNorm scale and tanh-residual fusions. | | Wan2.2 A14B T2V/I2V | 1280x720, 81 frames | Nightly: `--num-gpus 4 --enable-cfg-parallel --ulysses-degree 2 --text-encoder-cpu-offload --pin-cpu-memory` | For lowest latency, also benchmark pure Ulysses on the same GPUs. | | Wan2.2 TI2V 5B | 1280x720, 81 frames, 1 GPU | `--enable-torch-compile --warmup-mode request` | Keep the input image and motion prompt fixed when comparing sparse attention or Cache-DiT. | @@ -329,7 +369,8 @@ Use these as first commands to benchmark, not as universal winners. | JoyAI-Image-Edit | 1024-class TI2I, 40 steps, guidance 4.0 | `--backend=sglang --num-gpus 2 --enable-cfg-parallel --ulysses-degree 1 --enable-torch-compile --warmup-mode request --dit-layerwise-offload false --dit-cpu-offload false` | Newly supported image-edit path. Keep the input image, prompt, seed, and output size fixed; 2-GPU CFG parallel is the validated H100 starting point. | | FireRed-Image-Edit 1.0 / 1.1 | 1024x1024 image edit, 40 steps, guidance 4.0 | `--backend=sglang --num-gpus 2 --enable-cfg-parallel --ulysses-degree 1 --enable-torch-compile --warmup-mode request --dit-layerwise-offload false --dit-cpu-offload false` | Uses the native `QwenImageEditPlusPipeline` path. 2-GPU CFG parallel is the validated H100 starting point; benchmark 1.0 and 1.1 separately because checkpoint differences can change denoise latency. | | Hunyuan3D-2 shape | Shape generation, 50 steps, guidance 5.0 | `--backend=sglang --enable-torch-compile --warmup-mode request --dit-layerwise-offload false --dit-cpu-offload false` | Focus on `Hunyuan3DShapeDenoisingStage`; keep mesh export/paint timings separate from denoise. | -| MOVA / Helios / LingBot World | Use the benchmark/profile presets or server test cases first | `--enable-torch-compile --warmup-mode request`; pin offload and topology flags explicitly | These video/realtime families have model-specific stages and condition handling. Keep prompt/image/action inputs fixed and prefer perf dumps over wall time alone. | +| LingBot Video MoE 30B | 384x640, 17 frames, 12 steps for the current GPU case | `--model-path robbyant/lingbot-video-moe-30b-a3b --text-encoder-cpu-offload` | Native T2V path. Prompts are structured JSON captions, not raw free text; keep that contract when comparing latency or quality. | +| MOVA / Helios / LingBot World | Use the benchmark/profile presets or server test cases first | `--enable-torch-compile --warmup-mode request`; pin offload and topology flags explicitly | These video/realtime families have model-specific stages and condition handling. For LingBot World causal serving, keep `--kv-cache-quant off` as the exact cache baseline before testing INT4/INT2. | ## Historical PR Watchlist @@ -344,10 +385,13 @@ about whether the work has merged: ## Tips - **Benchmarking**: always use `--warmup-mode request` and look for the line ending with `(with warmup excluded)` for accurate timing. +- **Preset vs experiment control**: start with `--performance-mode auto` or + `speed` for deployment, but use `--performance-mode manual` and pin the + relevant residency/parallelism flags for controlled A/B claims. - **Perf dump**: use `--perf-dump-path result.json` to save structured metrics, then compare with `python python/sglang/multimodal_gen/benchmarks/compare_perf.py baseline.json result.json`. - **Offload tuning**: after the first request, the runtime logs peak GPU memory and which components could stay resident. Use this to decide which `--*-cpu-offload` flags to disable. - **Backend selection**: `--backend sglang` (default, auto-detected) enables native optimizations (fused kernels, SP, native Cache-DiT env knobs, etc.). `--backend diffusers` falls back to Diffusers pipelines and is the path that accepts `--cache-dit-config` plus diffusers attention backend names. - **Wan2.2-I2V sizing**: explicit `--width/--height` on `Wan2.2-I2V-A14B` control the target area while preserving the condition-image aspect ratio. -- **Mainline diffusion fast paths**: before proposing a new kernel or overlap scheme, check `sglang-diffusion-benchmark-profile/existing-fast-paths.md`. It covers H3 indexed modulation, fused QK norm + RoPE, packed Ulysses QKV/USP relayout and batched TP AdaLN, plus GroupNorm+SiLU, Z-Image bf16-native Triton norm modulation, LTX2 split RoPE, LTX2 residual-gate add, varlen USP pack/scatter, packed QKV/NVFP4 expectations, and existing multi-GPU overlap families such as Ulysses / USP and turbo-layer async all-to-all. +- **Mainline diffusion fast paths**: before proposing a new kernel or overlap scheme, check `sglang-diffusion-benchmark-profile/existing-fast-paths.md`. It covers H3 indexed modulation, fused QK norm + RoPE, packed Ulysses QKV/USP relayout and batched TP AdaLN; FLUX/GLM/SANA bit-exact LayerNorm+modulate; request-scoped quality gates; Wan causal-VAE data movement; GroupNorm+SiLU, Z-Image bf16-native norm modulation, LTX2 split RoPE/residual-gate add, varlen USP pack/scatter, packed QKV/NVFP4, breakable CUDA graph, and existing distributed overlap families. - **NVFP4 trace interpretation**: on FLUX.2 NVFP4 and Nunchaku-style checkpoints, packed QKV is expected. SGLang intentionally uses fused projection modules such as `to_qkv` / `to_added_qkv` instead of separate `to_q` / `to_k` / `to_v`, so a split-QKV trace usually means the quantized path did not engage rather than a brand new fusion opportunity. - **Hotspot workflow split**: use `sglang-diffusion-benchmark-profile` to prove and classify a slowdown with perf dumps plus `torch.profiler`; hand concrete kernel work off with the perf/profile evidence attached instead of expanding the benchmark skill.