[Diffusion] Add cumulative extra-high quality tier (#37422)
This commit is contained in:
Xiaoyu Zhang
@@ -45,7 +45,7 @@ ext/ JIT C++/CUDA extensions (Hunyuan3D raster/inpaint) — NOT kernels
|
||||
../../kda_kernels/ agent-generated implementations and their JIT CUDA sources
|
||||
```
|
||||
|
||||
## The two numerical contracts
|
||||
## Numerical contracts and quality policy
|
||||
|
||||
**Bit-exact (`torch.equal` vs the eager chain) → mounted unconditionally.**
|
||||
These kernels reproduce every aten rounding boundary, sometimes down to the
|
||||
@@ -59,11 +59,18 @@ themselves against the live eager chain on first sight via
|
||||
dispatch they replicate can change under them.
|
||||
|
||||
**Not bit-exact → quality-gated.** Mounted onto marked `nn.Module` sites only
|
||||
for `quality="high"` requests, at batch boundaries, all-or-nothing per
|
||||
transformer (`sites/quality_gate.py`). A plain fp32 single-pass norm fusion
|
||||
looks harmless and is not: on ERNIE-Image it moved the 50-step trajectory to
|
||||
PSNR 18.83 dB at `quality=high`, which is what motivated the bit-exact
|
||||
rewrite.
|
||||
for `quality="extra-high"` and `quality="high"` requests, at batch boundaries,
|
||||
all-or-nothing per transformer (`sites/quality_gate.py`). `extra-high` adds
|
||||
only these request-gated DiT/VAE fusions; `high` is cumulative and may also
|
||||
enable model-owned approximate paths such as Cache-DiT or a lower-precision
|
||||
decode. A plain fp32 single-pass norm fusion looks harmless and is not: on
|
||||
ERNIE-Image it moved the 50-step trajectory to PSNR 18.83 dB, which is what
|
||||
motivated the bit-exact rewrite.
|
||||
|
||||
**Model/checkpoint-native.** Generic close-contract kernels, sparse operators,
|
||||
and FP8/NVFP4 producers can belong to the selected model or deployment path.
|
||||
The request `quality` tier neither selects nor disables those independent
|
||||
choices.
|
||||
|
||||
SANA-Video's quality-gated linear-attention site keeps BF16 inputs for the
|
||||
first GEMM while requesting FP32 accumulation/output, then runs the second
|
||||
@@ -98,6 +105,7 @@ Several norms look interchangeable and are not. Start here.
|
||||
| `fused_layernorm_modulate` | Triton | bit-exact vs aten `vectorized_layer_norm` | bf16, `N % 4 == 0`, 16B-aligned |
|
||||
| `fused_norm_scale_shift` / `fused_scale_residual_norm_scale_shift` | CuTe-DSL | fp32 statistics, close | fp16/bf16/fp32, LN or RMS, many broadcast modes |
|
||||
| `flydsl_norm_scale_shift` / `flydsl_fused_residual_norm_scale_shift` | FlyDSL | close | **ROCm gfx950 only** |
|
||||
| `try_fused_scale_residual_norm_scale_shift_nvfp4` | JIT CUDA | matches the selected NVFP4 producer contract | Qwen residual LayerNorm/modulation + FC1 NVFP4 quantization |
|
||||
| `fuse_layernorm_scale_shift_gate_select01_kernel` | Triton | close | per-token select between two modulation rows (Qwen-Image) |
|
||||
| `norm_infer` / `rms_norm_fn` | Triton (+torch/NPU/MPS fallbacks) | close | the generic entry point; use when nothing above fits |
|
||||
|
||||
@@ -131,6 +139,8 @@ tensor copy per residual site.
|
||||
|---|---|---|
|
||||
| `fused_inplace_qknorm_rope` | JIT CUDA | one bf16 rounding step vs split baseline; `round_norm_before_rope=True` makes it exact; supports compact and full-width NeoX/interleaved caches |
|
||||
| `fused_qknorm_rope_pack_kv` | JIT CUDA | as above, also packs prefix K/V |
|
||||
| `try_fused_flux2_qkv_epilogue` | JIT CUDA | bit-exact vs the selected BF16 chain | FLUX.2 QK RMSNorm + RoPE + joint QKV packing |
|
||||
| `try_fused_qwen_qkv_epilogue` | JIT CUDA | bit-exact vs the selected BF16 chain | Qwen-Image QK RMSNorm + RoPE + joint QKV writes; SM100+ |
|
||||
| `fused_rope_rotate_half_bitexact` | Triton | bit-exact (elementwise only) |
|
||||
| `fused_interleaved_rope_fp64` | JIT CUDA | bit-exact vs paired SANA-Video fp64 RoPE |
|
||||
| `fused_inplace_helios_qk_rope` | JIT CUDA | bit-exact paired in-place RoPE for Helios' transposed frequency layout |
|
||||
@@ -139,13 +149,16 @@ tensor copy per residual site.
|
||||
| `apply_rotary_embedding` | Triton (+fallbacks) | close; the generic entry point |
|
||||
| `hunyuan_qkv_rope_pack` | Triton | bit-exact; packs QKV and applies RoPE in one pass |
|
||||
|
||||
### Data movement (all bit-exact by construction)
|
||||
### Data movement and quantized layout producers
|
||||
|
||||
`usp_merge_heads`, `pack_qkv_destination_major`, `fused_pack_qkv`,
|
||||
`fused_pack_segmented_qkv`, `fused_scatter_to_padded`,
|
||||
`fused_causal_conv3d_cat_pad_cuda`,
|
||||
`cat_pad_channels_last_3d`, `dup_up3d_add`, `fused_temb_table_slices`,
|
||||
`ltx2_ada_values9`.
|
||||
and `ltx2_ada_values9` are bit-exact data movement or same-order arithmetic.
|
||||
`try_flux2_token_cat_fp8` and `try_flux2_token_cat_nvfp4` fuse branch
|
||||
concatenation directly into the quantized representation selected by the
|
||||
FLUX.2 checkpoint path.
|
||||
|
||||
`fused_temb_table_slices` is worth knowing about: the eager
|
||||
`(table + temb.float()).chunk(6, dim=2)` materializes ~8 GB of fp32 at
|
||||
@@ -172,7 +185,7 @@ inspecting model modules is its whole job.
|
||||
3. Give it a `can_use_*` predicate; raise, don't return `None`.
|
||||
4. State the numerical contract in the module docstring, including which
|
||||
shapes it was verified on.
|
||||
5. If it is not bit-exact, gate it through `sites/`. Do not mount it by
|
||||
default.
|
||||
5. If it is not bit-exact, gate it through `sites/`. It must mount for both
|
||||
`extra-high` and `high`, never for the default `lossless` path.
|
||||
6. Test it in the domain suite (`test/registered/kernels/ops/diffusion/`), and
|
||||
the model wiring in `test_model_fast_paths.py`.
|
||||
|
||||
@@ -656,15 +656,17 @@ BCG validation must prove all of the following:
|
||||
|
||||
For a non-bit-exact optimization, integrate through the request-scoped site
|
||||
framework under `sglang.kernels.ops.diffusion.sites`. Mark sites during model
|
||||
construction and let `QualityGatedFusion` mount them only for
|
||||
`quality="high"`; `quality="lossless"` must keep the original code path.
|
||||
construction and let `QualityGatedFusion` mount them for both
|
||||
`quality="extra-high"` and `quality="high"`; `quality="lossless"` must keep
|
||||
the original code path. A high-only sparse, caching, or other approximate
|
||||
path must remain outside this fusion gate.
|
||||
Eligibility must be all-or-nothing for coupled sites and fail closed on dtype,
|
||||
shape, layout, backend, BCG, or compile incompatibility. Add clean site-level
|
||||
guard/parity tests and a model wiring test instead of embedding request-policy
|
||||
branches throughout the DiT.
|
||||
|
||||
Finally, use the benchmark/profile skill's `--quality-bcg-matrix` to run
|
||||
same-GPU ABBA pairs for Eager/BCG at lossless/high. Report denoise and saved
|
||||
same-GPU ABBA pairs for Eager/BCG at lossless/extra-high/high. Report denoise and saved
|
||||
request e2e separately, require at least 1.5% repeated mean e2e improvement for
|
||||
an optimization PR, attach profile and generated-media A/B evidence, then
|
||||
delete the task-owned checkpoint cache and verify zero residual weight files
|
||||
|
||||
+7
-7
@@ -9,7 +9,7 @@ Use this skill when measuring denoise performance, finding the slow op, checking
|
||||
|
||||
This skill is diagnosis-first. It owns:
|
||||
- checked-in denoise benchmark presets
|
||||
- same-GPU quality/BCG applicability checks with repeated lossless and high rows
|
||||
- same-GPU quality/BCG applicability checks with repeated lossless, extra-high, and high rows
|
||||
- perf dump collection and before/after comparison
|
||||
- `torch.profiler` trace capture and quick hotspot ranking
|
||||
- mapping hot kernels back to known fast paths and fusion families
|
||||
@@ -65,7 +65,7 @@ Always rule out these existing families first:
|
||||
- Z-Image bf16-native Triton RMSNorm scale/tanh-residual modulation
|
||||
- SANA packed self-attention Q/K/V and cross-attention K/V GEMMs
|
||||
- SANA-Video's packed projections and request-scoped BF16-input linear
|
||||
attention at `quality=high`; keep the second attention GEMM in FP32 and
|
||||
attention at `quality=extra-high` or `quality=high`; keep the second attention GEMM in FP32 and
|
||||
compare against `quality=lossless` before changing its precision further
|
||||
- SANA-Video reuse of SANA's bit-exact bias/activation, residual-gate, and
|
||||
LayerNorm-modulation fast paths before adding video-only kernels
|
||||
@@ -73,7 +73,7 @@ Always rule out these existing families first:
|
||||
USP relayout, and batched TP AdaLN collectives
|
||||
- bit-exact diffusion adaLN modulation and fused LayerNorm + modulation for
|
||||
FLUX.1, GLM-Image, and SANA
|
||||
- request-scoped `quality=high` DiT and VAE fast paths
|
||||
- request-scoped DiT and VAE fast paths at `quality=extra-high` or `quality=high`
|
||||
- Wan causal-VAE cache/padding and DupUp3D data-movement fusions
|
||||
- fused diffusion `QK norm + RoPE`
|
||||
- LTX2 split RoPE
|
||||
@@ -94,9 +94,9 @@ controlled comparator, never for the eager ground truth. The legacy
|
||||
`--no-torch-compile` spelling remains accepted but is redundant.
|
||||
|
||||
For kernel/BCG discovery, run `--quality-bcg-matrix`. It executes Eager/BCG as
|
||||
A-B-B-A at `lossless`, then repeats the pair at `high`, on one locked GPU set
|
||||
and one isolated checkpoint cache. The high+BCG rows are applicability checks,
|
||||
not presumed-valid performance cells. A BCG row is invalid unless the log
|
||||
A-B-B-A at `lossless`, then repeats the pair at `extra-high` and `high`, on
|
||||
one locked GPU set and one isolated checkpoint cache. The extra-high/high+BCG
|
||||
rows are applicability checks, not presumed-valid performance cells. A BCG row is invalid unless the log
|
||||
contains `[Diffusion BCG] captured` and contains no support-disable,
|
||||
capture-failure, serving-signature-miss, or late quality-fusion marker. In
|
||||
particular, a request-scoped DiT fusion mounted after lossless warmup capture
|
||||
@@ -120,7 +120,7 @@ its normal cleanup finally block without modifying the seed cache.
|
||||
|
||||
Keep prompt, negative prompt, seed, shape, steps, guidance, dtype, topology,
|
||||
and residency fixed. Lossless comparisons require byte-identical artifacts.
|
||||
For `quality=high`, report aggregate and worst-frame SSIM/PSNR; the repository
|
||||
For `quality=extra-high` and `quality=high`, report aggregate and worst-frame SSIM/PSNR; the repository
|
||||
defaults are 0.95/28 dB for images and 0.92/24 dB for video unless the model's
|
||||
checked-in consistency metadata defines a different threshold. A performance
|
||||
PR needs repeated saved-request e2e improvement of at least 1.5%, a
|
||||
|
||||
+15
-13
@@ -139,9 +139,9 @@ The helper defaults to eager. Add `--torch-compile` only for a labeled compile
|
||||
control. `--no-torch-compile` remains accepted for compatibility but is no
|
||||
longer required.
|
||||
|
||||
Run one explicit quality or BCG comparator with `--quality lossless|high` and
|
||||
Run one explicit quality or BCG comparator with `--quality {lossless,extra-high,high}` and
|
||||
`--breakable-cuda-graph`. BCG and `torch.compile` are intentionally mutually
|
||||
exclusive in this helper. A high+BCG command is only a compatibility probe:
|
||||
exclusive in this helper. An extra-high/high+BCG command is only a compatibility probe:
|
||||
it is invalid if request-scoped DiT fusions mount after the lossless warmup
|
||||
graphs were captured. When a preset has explicit width and height, the helper
|
||||
declares that same `--warmup-resolutions` value automatically. Video presets
|
||||
@@ -150,21 +150,22 @@ with an explicit frame count also declare the matching `--warmup-num-frames`:
|
||||
```bash
|
||||
PYTHONPATH=python python3 "$BENCH_PY" \
|
||||
--model longcat-image \
|
||||
--quality high \
|
||||
--quality extra-high \
|
||||
--breakable-cuda-graph \
|
||||
--label bcg-high \
|
||||
--label bcg-extra-high \
|
||||
--output-dir "${BENCH_DIR}"
|
||||
```
|
||||
|
||||
For optimization discovery, use the full repeated matrix. It runs
|
||||
Eager/BCG/BCG/Eager at `lossless`, then the same sequence at `high`, while
|
||||
holding one GPU set and one isolated checkpoint cache. The high+BCG cells test
|
||||
Eager/BCG/BCG/Eager at `lossless`, then the same sequence at `extra-high` and
|
||||
`high`, while holding one GPU set and one isolated checkpoint cache. The
|
||||
extra-high/high+BCG cells test
|
||||
whether the combination is actually supported; do not average them when the
|
||||
runtime rejects the combination or the helper detects a late quality-fusion
|
||||
mount. The helper hashes every generated image, video, audio, or 3D mesh
|
||||
artifact. It first requires the two Eager rows at each quality to agree, then
|
||||
rejects any BCG row whose hash differs from that Eager reference. Cleanup occurs
|
||||
only after all eight runs, including on failure or interruption:
|
||||
only after all twelve runs, including on failure or interruption:
|
||||
|
||||
```bash
|
||||
MODEL_CACHE_ROOT=/path/to/task-owned/model-caches
|
||||
@@ -181,7 +182,7 @@ Before starting, confirm the chosen GPU set has no foreign process and remains
|
||||
unchanged through every run boundary. The helper rejects a BCG row unless its
|
||||
log contains `[Diffusion BCG] captured` and contains none of: support-gate
|
||||
disable, capture failure, `serving signature MISSED`, a message that no graph
|
||||
will be captured, or a request-scoped high-quality DiT fusion mounted after
|
||||
will be captured, or a request-scoped quality-gated DiT fusion mounted after
|
||||
capture. Do not average rejected rows with valid results.
|
||||
|
||||
BCG signatures include more than width and height. The helper maps an explicit
|
||||
@@ -319,13 +320,13 @@ Use the preset categories this way:
|
||||
| `qwen-edit-base` | `Qwen/Qwen-Image-Edit` | No | Covers the original native `QwenImageEditPipelineConfig`, which is distinct from the 2509/2511 edit-plus paths; public SGLang edit fixture, 1024x1024. |
|
||||
| `qwen-image-layered` | `Qwen/Qwen-Image-Layered` | No | Native layered-image path using the same public reference image and four-frame request as the GPU server case, at the registered 640x640 canvas. |
|
||||
| `stable-diffusion-3.5-medium` | `stabilityai/stable-diffusion-3.5-medium-diffusers` | No | Representative native `StableDiffusion3PipelineConfig` path at 1024x1024. The repository is gated, so export `HF_TOKEN`; an unauthenticated run is a recorded access blocker, not model evidence. |
|
||||
| `sana-video` | `Efficient-Large-Model/SANA-Video_2B_480p_diffusers` | No | CI-sized T2V baseline: 832x480, 17 frames, 8 steps, guidance 6.0. The BCG comparator declares the same 17-frame warmup shape. Compare `quality=lossless` and `quality=high`; high enables the BF16-input first linear-attention GEMM while retaining FP32 output and the FP32 second GEMM. |
|
||||
| `sana-video` | `Efficient-Large-Model/SANA-Video_2B_480p_diffusers` | No | CI-sized T2V baseline: 832x480, 17 frames, 8 steps, guidance 6.0. The BCG comparator declares the same 17-frame warmup shape. Compare all three tiers; `extra-high` and `high` enable the BF16-input first linear-attention GEMM while retaining FP32 output and the FP32 second GEMM. |
|
||||
| `sana-wm-bidirectional` | `Efficient-Large-Model/SANA-WM_bidirectional` | No | Dense two-stage TI2V baseline at the native 1280x704 shape, 49 frames, 16 fps, 20 steps, guidance 4.5, and a 48-frame forward/left action program. Uses the shared cat fixture. |
|
||||
| `sana-wm-streaming` | `Efficient-Large-Model/SANA-WM_streaming` | No | Matching offline chunk-causal two-stage baseline with the streaming DiT and chunked refiner enabled; uses the same shape, fixture, seed, and camera action for comparison. |
|
||||
| `lingbot-video-moe` | `robbyant/lingbot-video-moe-30b-a3b` | No | One-GPU eager baseline using the CI structured-JSON caption, 384x640, 17 frames, 12 steps, and text-encoder CPU offload. |
|
||||
| `lingbot-world` | `robbyant/lingbot-world-fast-diffusers` | No | One-H200 offline single-chunk profile for the registered causal DMD path: 832x480x9, four steps, guidance 1.0, the shared image fixture, and forward-camera actions for all nine frames. Keep stateful websocket latency as a separate metric. |
|
||||
| `lingbot-world-v2` | `robbyant/lingbot-world-v2-14b-causal-fast-diffusers` | No | Matching controlled single-chunk profile for the separately registered v2 checkpoint. The fixed shape, action program, and schedule make v1/v2 hotspot comparisons reproducible without presenting one-chunk e2e as stateful realtime latency. |
|
||||
| `fastwan21-t2v-1.3b` | `FastVideo/FastWan2.1-T2V-1.3B-Diffusers` | No | One-GPU 832x480, 61-frame, 3-step DMD baseline. The preset pins manual mode with a resident DiT so lossless/high comparisons do not measure an offload-policy change. |
|
||||
| `fastwan21-t2v-1.3b` | `FastVideo/FastWan2.1-T2V-1.3B-Diffusers` | No | One-GPU 832x480, 61-frame, 3-step DMD baseline. The preset pins manual mode with a resident DiT so lossless/extra-high/high comparisons do not measure an offload-policy change. |
|
||||
| `wan21-t2v-1.3b` | `Wan-AI/Wan2.1-T2V-1.3B-Diffusers` | No | Registered one-GPU 832x480, 81-frame Wan2.1 baseline at 50 steps and guidance 3.0. Keep it separate from FastWan and TurboWan because the longer schedule changes the end-to-end weight of VAE optimizations. |
|
||||
| `wan21-t2v-14b` | `Wan-AI/Wan2.1-T2V-14B-Diffusers` | No | Cookbook-aligned four-GPU CFG/Ulysses baseline at 832x480, 81 frames, 50 steps, and guidance 5.0. Text encoding stays CPU-offloaded as in the documented deployment command. |
|
||||
| `wan21-i2v-14b-480p` | `Wan-AI/Wan2.1-I2V-14B-480P-Diffusers` | No | Four-GPU CFG/Ulysses image-conditioned baseline at 832x480, 81 frames, 50 steps, and guidance 5.0. Uses the shared cat fixture and its motion prompt. |
|
||||
@@ -668,7 +669,8 @@ some generation failures are reported through the response payload without a
|
||||
nonzero process exit.
|
||||
|
||||
For `quality=lossless`, compare saved artifact hashes and require byte equality
|
||||
for a claimed lossless fast path or BCG change. For `quality=high`, keep the
|
||||
for a claimed lossless fast path or BCG change. For `quality=extra-high` and
|
||||
`quality=high`, keep the
|
||||
lossless artifact as ground truth and report both aggregate and worst-frame
|
||||
SSIM/PSNR. Repository defaults are SSIM 0.95 / PSNR 28 dB for images and SSIM
|
||||
0.92 / PSNR 24 dB for videos; checked-in model/hardware consistency metadata
|
||||
@@ -788,7 +790,7 @@ the known mainline families.
|
||||
| `to_q -> to_k -> to_v` on NVFP4 or Nunchaku FLUX-family checkpoints | Treat as a packed-QKV fast-path miss or checkpoint-format mismatch |
|
||||
| `rmsnorm_scale` or `rmsnorm_tanh_residual` missing on Z-Image | Check the bf16-native Triton eligibility guards before proposing a new fusion |
|
||||
| FLUX.1, GLM-Image, or SANA shows separate LayerNorm plus adaLN elementwise kernels | Check the bit-exact `modulate_scale_shift` and `fused_layernorm_modulate` guards/self-test before proposing another norm fusion |
|
||||
| `quality=high` shows the same FLUX/GLM DiT or FLUX-family/Wan VAE chain as `lossless` | Check whether the request-scoped quality gate mounted and whether every site passed its all-or-nothing compatibility checks |
|
||||
| `quality=extra-high` or `quality=high` shows the same FLUX/GLM DiT or FLUX-family/Wan VAE chain as `lossless` | Check whether the request-scoped quality gate mounted and whether every site passed its all-or-nothing compatibility checks |
|
||||
| LTX-2 split RoPE appears as a long PyTorch elementwise chain | Check the `apply_ltx2_split_rotary_emb` Triton path and its shape guards |
|
||||
| Wan decode is dominated by causal `cat + pad + contiguous`, feature-cache copies, or `repeat_interleave + permute + add` | Check the bit-exact Wan causal-cache and DupUp3D data-movement kernels before writing a new decoder kernel |
|
||||
| masked attention spends time packing/unpacking Q/K/V | Check whether fused varlen USP pack/scatter should have engaged |
|
||||
@@ -826,7 +828,7 @@ This skill intentionally stops here. It tells you whether you are looking at:
|
||||
- [ ] `compare_perf.py` table generated
|
||||
- [ ] one representative `torch.profiler` trace saved
|
||||
- [ ] hotspot classified against `existing-fast-paths.md`
|
||||
- [ ] lossless artifact hash is exact; high-quality aggregate and worst-frame SSIM/PSNR pass the checked-in threshold
|
||||
- [ ] lossless artifact hash is exact; extra-high/high aggregate and worst-frame SSIM/PSNR pass the checked-in threshold
|
||||
- [ ] reference image or start/middle/end video contact sheet checked visually
|
||||
- [ ] any PR claim has repeated saved-request e2e improvement >= 1.5%
|
||||
- [ ] task-owned checkpoint cache cleaned and ledger shows zero residual weight files
|
||||
|
||||
+24
-20
@@ -114,17 +114,20 @@ framework-specific optimization workflow.
|
||||
check dtype, alignment, shape, BCG/compile context, and the one-time equality
|
||||
self-test before proposing another fusion.
|
||||
|
||||
4. Request-scoped `quality=high` fusion gates
|
||||
4. Request-scoped fusion gates at `quality=extra-high` or `quality=high`
|
||||
- Locations: `quality_gate.py`, `fused_ln_modulate.py`, `denoising.py`,
|
||||
`decoding.py`, `fast_path_gate.py`, `flux2_vae_cuda_opt.py`, and
|
||||
`wan_vae_cuda_opt.py`.
|
||||
- Behavior: `quality="lossless"` is the default exact reference path.
|
||||
`quality="high"` may mount model-owned, validated but non-bit-exact DiT
|
||||
fusions and decode-scoped VAE rewrites. Mounting is all-or-nothing per
|
||||
`quality="extra-high"` and `quality="high"` mount the same validated but
|
||||
non-bit-exact DiT fusions and decode-scoped VAE rewrites. `high` is
|
||||
cumulative and may additionally enable model-owned approximate paths.
|
||||
Mounting is all-or-nothing per
|
||||
transformer/fusion family; VAE gates reset after every decode.
|
||||
- Current families include FLUX affine-folded LN+modulate / fused GELU sites,
|
||||
GLM/Qwen/Hunyuan/LTX fused GELU sites, LTX RMSNorm+modulate, Hunyuan QK
|
||||
RMSNorm, Ideogram gated RMSNorm, SANA-Video linear attention, generic KL VAE
|
||||
Wan cublasLt/NVFP4 GELU, Qwen added-QKV, GLM/Qwen/Hunyuan/LTX fused GELU,
|
||||
LTX RMSNorm+modulate, Hunyuan QK RMSNorm, Ideogram gated RMSNorm,
|
||||
LingBot RMSNorm, SANA-Video linear attention, generic KL VAE
|
||||
decoder rewrites used by FLUX.1/FLUX.2/Z-Image/SD3, and Wan VAE
|
||||
RMSNorm+SiLU.
|
||||
- Do not confuse request `--quality` with `--output-quality`, which controls
|
||||
@@ -218,7 +221,7 @@ framework-specific optimization workflow.
|
||||
one channels-last-3D pass, and fuse `main + DupUp3D(src)` without
|
||||
materializing `repeat_interleave + permute().contiguous()` intermediates.
|
||||
- Numerical contract: these are bit-exact data-movement / same-order-add
|
||||
replacements and run independently of the `quality=high` Wan RMSNorm+SiLU
|
||||
replacements and run independently of the request-gated Wan RMSNorm+SiLU
|
||||
path. Unsupported layouts or padding fall back to the aten chain.
|
||||
- Validation: `test/registered/kernels/ops/diffusion/test_wan_causal_cache.py`.
|
||||
|
||||
@@ -328,16 +331,16 @@ framework-specific optimization workflow.
|
||||
|
||||
**Request-Scoped DiT Fusions with Breakable CUDA Graphs**
|
||||
|
||||
- `quality=high` DiT sites are mounted at a request boundary. BCG warmup uses
|
||||
- DiT sites at `quality=extra-high` or `quality=high` are mounted at a request boundary. BCG warmup uses
|
||||
the model's lossless sampling default unless a quality-aware graph variant
|
||||
was captured explicitly.
|
||||
- A graph captured before the high-quality mount retains the lossless module
|
||||
branches. Replaying it after the mount silently bypasses the requested high
|
||||
- A graph captured before the request-quality mount retains the lossless module
|
||||
branches. Replaying it after the mount silently bypasses the requested fused
|
||||
kernels even when the tensor signature matches.
|
||||
- Workflow rule: a high+BCG cell is valid only when the model has no
|
||||
- Workflow rule: an extra-high/high+BCG cell is valid only when the model has no
|
||||
request-scoped DiT quality sites, or when logs prove those sites were mounted
|
||||
before the matching graph capture. A mount after `[Diffusion BCG] captured`
|
||||
invalidates the row; do not use its latency or output as high-quality
|
||||
invalidates the row; do not use its latency or output as request-quality
|
||||
evidence.
|
||||
|
||||
**Recent Model Audit Boundaries**
|
||||
@@ -363,10 +366,10 @@ framework-specific optimization workflow.
|
||||
- LingBot Video MoE's router implements sigmoid+bias grouped top-k in
|
||||
`multimodal_gen/runtime/layers/moe.py`. Check parameter and output-order
|
||||
compatibility with `srt/layers/moe/topk.py::biased_grouped_topk` before
|
||||
writing a new router kernel. Its released eager path still expands RMSNorm
|
||||
into `pow/mean/rsqrt` chains. #35969 is a measured `quality=high` candidate
|
||||
that dispatches existing Triton row kernels by weight dtype and hidden size;
|
||||
it is not current-main behavior until the PR merges.
|
||||
writing a new router kernel. Current main mounts fused Triton RMSNorm row
|
||||
kernels by weight dtype and hidden size for `quality=extra-high` and
|
||||
`quality=high`; check the quality-site guards before treating an expanded
|
||||
`pow/mean/rsqrt` chain as a new opportunity.
|
||||
- LTX-2.5 reuses the mature LTX-2 DiT paths. Treat the optional diffusion
|
||||
decoder separately: confirm NATTEN `na3d` is active, then inspect its
|
||||
per-block 3D RoPE construction and split QKV/SwiGLU projections.
|
||||
@@ -379,7 +382,7 @@ framework-specific optimization workflow.
|
||||
- AdaLN modulation: `LayerNormScaleShift`, `RMSNormScaleShift`, `ScaleResidual*` in `layernorm.py`.
|
||||
- Bit-exact adaLN modulation / LayerNorm folding: `modulate_scale_shift` and
|
||||
`fused_layernorm_modulate` through `flux.py`, `glm_image.py`, and `sana.py`.
|
||||
- Request-scoped high-quality acceleration: `QualityGatedFusion` in
|
||||
- Request-scoped extra-high/high acceleration: `QualityGatedFusion` in
|
||||
`quality_gate.py`, `_maybe_toggle_quality_fusions` in `denoising.py`, and
|
||||
`use_vae_fast_path` in `decoding.py`.
|
||||
- Bit-exact first-sight verify/disable: `BitExactFusionGate` in
|
||||
@@ -394,7 +397,7 @@ framework-specific optimization workflow.
|
||||
- QK norm: `apply_qk_norm` used in `flux.py`, `flux_2.py`, `qwen_image.py`, `zimage.py`, `wanvideo.py`, `ltx_2.py`, `hunyuanvideo.py`.
|
||||
- QK norm + RoPE: `apply_qk_norm_rope` in `layernorm.py`; use this path when the model wants fused attention prep instead of separate QK norm and RoPE calls.
|
||||
- LTX2 split RoPE: `apply_ltx2_split_rotary_emb` in `ltx_2.py`.
|
||||
- LTX2 RMSNorm+modulate and FFN GELU epilogue under `quality="high"`:
|
||||
- LTX2 RMSNorm+modulate and FFN GELU epilogue under `quality="extra-high"` and `quality="high"`:
|
||||
`mark_ltx2_rms_norm_modulate_site` / `fused_ltx2_rms_norm_modulate` in
|
||||
`kernels/ops/diffusion/sites/ltx2_rmsnorm_modulate_site.py` (mount-based
|
||||
`QualityGatedFusion`, not a first-sight `BitExactFusionGate` — the fused
|
||||
@@ -468,8 +471,9 @@ relying on any file path, flag, or claim about whether the work has merged.
|
||||
- #34584 Wan TI2V modulation/RoPE; #34616 FLUX2; #34617 Hunyuan;
|
||||
#34619 GLM; #34620 ERNIE; #34928 SANA; #34932 Cosmos3; #35728
|
||||
SANA-Video linear attention.
|
||||
- #35961 SANA-Video lossless shared-kernel reuse and #35969 LingBot
|
||||
`quality=high` RMSNorm are open candidates, not current-main fast paths.
|
||||
- SANA-Video shared-kernel reuse and LingBot request-gated RMSNorm are now
|
||||
current-main fast paths; verify the source tree before treating their
|
||||
historical PRs as open work.
|
||||
- VAE and decode-side acceleration:
|
||||
- #22531 LTX2 parallel VAE support and #20927 batched tiled VAE decode (draft).
|
||||
- Attention, communication, and runtime scheduling:
|
||||
@@ -492,7 +496,7 @@ relying on any file path, flag, or claim about whether the work has merged.
|
||||
**Constraints and Fallbacks**
|
||||
- `scale_shift` Triton requires CUDA + contiguous `x`. NPU swaps to native.
|
||||
- Bit-exact BF16 LayerNorm+modulate requires the guarded aten-compatible shape
|
||||
and a successful live equality check; `quality=high` affine folding is a
|
||||
and a successful live equality check; request-gated affine folding is a
|
||||
separate non-bit-exact path.
|
||||
- CuTe DSL fused norms require `D % 256 == 0` and `D <= 8192`.
|
||||
- Triton norm kernels error on feature size >= 64KB.
|
||||
|
||||
+13
-7
@@ -16,8 +16,9 @@ Usage:
|
||||
# Opt in to a compile control (presets are eager by default)
|
||||
python3 python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py --model flux --torch-compile
|
||||
|
||||
# Check Eager/BCG at lossless/high on one GPU set; high+BCG is invalid when
|
||||
# request-scoped DiT fusions mount only after lossless graph capture.
|
||||
# Check Eager/BCG at every request quality on one GPU set; extra-high/high
|
||||
# + BCG are invalid when request-scoped DiT fusions mount only after the
|
||||
# lossless graph capture.
|
||||
python3 python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py --model sana-video --quality-bcg-matrix --model-cache-root /task/model-caches --cleanup-model-cache
|
||||
|
||||
# Clean an isolated model cache even if the run fails or is interrupted
|
||||
@@ -80,14 +81,14 @@ DIFFUSERS_FALLBACK_SIGNALS = (
|
||||
"using diffusers backend",
|
||||
"loaded diffusers pipeline",
|
||||
)
|
||||
BENCHMARK_QUALITY_LEVELS = ("lossless", "high")
|
||||
BENCHMARK_QUALITY_LEVELS = ("lossless", "extra-high", "high")
|
||||
BCG_CAPTURE_SIGNAL = "[diffusion bcg] captured"
|
||||
BCG_INVALID_SIGNALS = (
|
||||
"[diffusion bcg] capture failed",
|
||||
"[diffusion bcg] disabled",
|
||||
"[diffusion bcg] serving signature missed",
|
||||
"no graph will be captured",
|
||||
"quality='high' cannot be used with breakable cuda graphs",
|
||||
"cannot be used with breakable cuda graphs",
|
||||
)
|
||||
BCG_LATE_QUALITY_FUSION_SIGNAL = "quality fusion mounted after BCG capture"
|
||||
QUALITY_BCG_ABBA_MATRIX = (
|
||||
@@ -95,6 +96,10 @@ QUALITY_BCG_ABBA_MATRIX = (
|
||||
("bcg-lossless-a", "lossless", True),
|
||||
("bcg-lossless-b", "lossless", True),
|
||||
("eager-lossless-b", "lossless", False),
|
||||
("eager-extra-high-a", "extra-high", False),
|
||||
("bcg-extra-high-a", "extra-high", True),
|
||||
("bcg-extra-high-b", "extra-high", True),
|
||||
("eager-extra-high-b", "extra-high", False),
|
||||
("eager-high-a", "high", False),
|
||||
("bcg-high-a", "high", True),
|
||||
("bcg-high-b", "high", True),
|
||||
@@ -1838,11 +1843,11 @@ def _run_benchmark_once_impl(
|
||||
if BCG_CAPTURE_SIGNAL in lower_line:
|
||||
bcg_capture_detected = True
|
||||
if (
|
||||
quality == "high"
|
||||
quality in {"extra-high", "high"}
|
||||
and breakable_cuda_graph
|
||||
and bcg_capture_detected
|
||||
and "mounted " in lower_line
|
||||
and "for quality=high" in lower_line
|
||||
and f"for quality={quality}" in lower_line
|
||||
):
|
||||
bcg_invalid_signals.add(BCG_LATE_QUALITY_FUSION_SIGNAL)
|
||||
bcg_invalid_signals.update(
|
||||
@@ -2256,7 +2261,8 @@ def main():
|
||||
"--quality-bcg-matrix",
|
||||
action="store_true",
|
||||
help=(
|
||||
"Run lossless/high Eager-vs-BCG as two ABBA pairs on one GPU set "
|
||||
"Run lossless/extra-high/high Eager-vs-BCG as three ABBA pairs "
|
||||
"on one GPU set "
|
||||
"and one task-owned model cache."
|
||||
),
|
||||
)
|
||||
|
||||
@@ -62,7 +62,7 @@ These options **trade output quality** for speed or VRAM savings. Results will d
|
||||
|
||||
| Option | CLI Flag / Env Var | What It Does | Speedup | Quality Impact / Limitations |
|
||||
|---|---|---|---|---|
|
||||
| **Request Quality Fast Paths** | `--quality high` (`lossless` is default) | Mounts model-owned accelerated DiT/VAE paths that are validated for high quality but are not bit-exact to the reference path. | Model- and shape-specific | Support is per model and may be a no-op. Keep `--quality lossless` as the A/B ground truth. Report aggregate and worst-frame SSIM/PSNR; defaults are 0.95/28 dB for images and 0.92/24 dB for video unless checked-in model metadata overrides them. Do not confuse this with `--output-quality`, which controls file compression. |
|
||||
| **Request Quality Fast Paths** | `--quality {extra-high,high}` (`lossless` is default) | `extra-high` mounts only request-gated DiT/VAE fusions. `high` includes that complete set and may add model-owned approximate paths such as Cache-DiT or lower-precision decode. | Model- and shape-specific | Support is per model and may be a no-op. Keep `--quality lossless` as the A/B ground truth, then compare `extra-high` before `high` to isolate fusion wins. Report aggregate and worst-frame SSIM/PSNR for every non-bit-exact path; defaults are 0.95/28 dB for images and 0.92/24 dB for video unless checked-in model metadata overrides them. Do not confuse this with `--output-quality`, which controls file compression. |
|
||||
| **Approximate Attention** | Server-wide: `--attention-backend sage_attn` / `sage_attn_3` / `sliding_tile_attn` / `video_sparse_attn` / `sparse_video_gen_2_attn` / `vmoba_attn` / `sla_attn` / `sage_sla_attn`. Per-request (dense drop-ins only): `--attention-backend-override sage_attn` sampling param / API `extra_body` — valid values `fa`, `torch_sdpa`, `sage_attn`, `sage_attn_3`; rejected (with a log) under BCG, torch.compile, sparse server backends, or a non-ring-capable target with ring parallelism. | Replaces exact attention with approximate or sparse variants. `sage_attn`: INT8/FP8 quantized Q·K; `sliding_tile_attn`: spatial-temporal tile skipping; others: model-specific sparse patterns. | ~1.5–2x on attention (varies by backend) | Quality degradation varies by backend and model. `sage_attn` is the most general; sparse backends (`sliding_tile_attn`, `video_sparse_attn`, etc.) are video-model-specific, may require config files (e.g. `--mask-strategy-file-path` for STA), and are server-level only. Requires corresponding packages installed. |
|
||||
| **Cache-DiT** | Native: per-request `--enable-cache-dit true\|false` + `--cache-dit-params <json>` (sampling params; also via API `extra_body`). `SGLANG_CACHE_DIT_ENABLED` / `SGLANG_CACHE_DIT_*` env vars are the server-wide defaults for requests that leave them unset. Diffusers backend: `--backend diffusers --cache-dit-config <yaml-or-json>` | Caches intermediate residuals across denoising steps and skips redundant computations via DBCache, TaylorSeer, and optional SCM. | ~1.5-2x on supported models | Quality depends on cache policy. Compatible with `--dit-layerwise-offload`: skipped blocks are not streamed, and the first layer after a skip may sync-load. Models that touch every layer before the block loop (for example a full-stack AdaLN prepass) must keep that prepass off while caching. Do not pass `--cache-dit-config` for native SGLang tuning unless you are intentionally using the diffusers backend flow. |
|
||||
| **CFG Gating** | Per-request `--cfg-gate-step 0.5` (sampling param; also via API `extra_body`). `SGLANG_DIFFUSION_CFG_GATE_STEP` is the server-wide default (1.0 = off). | After the given fraction of denoising steps, reuses the cached cond-uncond residual instead of running the unconditional branch each step. | Up to ~2x on the gated tail of CFG models (skips one of two branches) | Lossy; no-op without classifier-free guidance or with `--enable-cfg-parallel`. Lower fractions gate earlier and drift more. |
|
||||
@@ -252,7 +252,7 @@ For video, also match the captured frame and conditioning shape; `WxH` alone
|
||||
does not prove replay.
|
||||
|
||||
For a repeated discovery sweep, use the benchmark/profile helper. This runs
|
||||
lossless and high-quality Eager/BCG ABBA pairs on one GPU set, then deletes the
|
||||
lossless, extra-high, and high Eager/BCG ABBA pairs on one GPU set, then deletes the
|
||||
model group cache once:
|
||||
|
||||
```bash
|
||||
@@ -262,20 +262,25 @@ python3 python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-p
|
||||
--cleanup-model-cache
|
||||
```
|
||||
|
||||
### Compare request-scoped high-quality fast paths
|
||||
### Compare cumulative request-quality fast paths
|
||||
|
||||
```bash
|
||||
sglang generate --model-path <MODEL> \
|
||||
--quality lossless --prompt "..." --seed 42 \
|
||||
--perf-dump-path baseline.json --save-output
|
||||
|
||||
sglang generate --model-path <MODEL> \
|
||||
--quality extra-high --prompt "..." --seed 42 \
|
||||
--perf-dump-path quality-extra-high.json --save-output
|
||||
|
||||
sglang generate --model-path <MODEL> \
|
||||
--quality high --prompt "..." --seed 42 \
|
||||
--perf-dump-path quality-high.json --save-output
|
||||
```
|
||||
|
||||
Keep every other flag fixed and compare the generated artifact as well as the
|
||||
perf dumps. If the model has no registered quality-gated sites, `high` may be a
|
||||
perf dumps. `high` must retain every fusion observed under `extra-high`. If the
|
||||
model has no registered request-gated or high-only sites, either tier may be a
|
||||
no-op.
|
||||
|
||||
### Image-edit baselines: JoyAI and FireRed
|
||||
@@ -384,7 +389,7 @@ Use these as first commands to benchmark, not as universal winners.
|
||||
| Z-Image / Z-Image-Turbo | 1024x1024, runtime-default steps/guidance, 1 GPU | `--enable-torch-compile --warmup-mode request` | Keep base Z-Image separate from Turbo: base uses 50-step CFG defaults, Turbo uses 9-step zero-CFG defaults. Mainline has bf16-native Triton RMSNorm scale and tanh-residual fusions. |
|
||||
| Wan2.2 A14B T2V/I2V | 1280x720, 81 frames | Nightly: `--num-gpus 4 --enable-cfg-parallel --ulysses-degree 2 --text-encoder-cpu-offload --pin-cpu-memory` | For lowest latency, also benchmark pure Ulysses on the same GPUs. |
|
||||
| Wan2.2 TI2V 5B | 1280x720, 81 frames, 1 GPU | `--enable-torch-compile --warmup-mode request` | Keep the input image and motion prompt fixed when comparing sparse attention or Cache-DiT. |
|
||||
| Wan2.1 / FastWan / TurboWan variants | 480p or 720p video, family defaults | Compare `--quality lossless` with `--quality high`, then try `--enable-torch-compile --warmup-mode request`; add `--ulysses-degree` / CFG parallel only after measuring | `quality=high` mounts the Wan FFN cublasLt GELU epilogue and the Wan VAE RMSNorm+SiLU fast path when their guards pass; validate video quality against lossless. Current registry includes Wan2.1, FastWan2.1, FastWan2.2 TI2V, TurboWan2.1, TurboWan2.2 I2V, and Wan2.1-Fun InP. Use the compatibility matrix and benchmark presets before choosing topology. |
|
||||
| Wan2.1 / FastWan / TurboWan variants | 480p or 720p video, family defaults | Compare `--quality lossless`, `--quality extra-high`, and `--quality high`, then try `--enable-torch-compile --warmup-mode request`; add `--ulysses-degree` / CFG parallel only after measuring | `extra-high` and `high` mount the Wan FFN cublasLt/NVFP4 GELU epilogues and the Wan VAE RMSNorm+SiLU fast path when their guards pass; validate video quality against lossless. Current registry includes Wan2.1, FastWan2.1, FastWan2.2 TI2V, TurboWan2.1, TurboWan2.2 I2V, and Wan2.1-Fun InP. Use the compatibility matrix and benchmark presets before choosing topology. |
|
||||
| Cosmos3 Nano / Super | T2I: 1024x1024 with `--num-frames 1`; T2V/I2V: 480p/720p video | Start with `--performance-mode auto --warmup-mode request`; use `SGLANG_DISABLE_COSMOS3_GUARDRAILS=1` only for benchmark isolation, and compare compile separately | One checkpoint serves T2I/T2V/I2V. Mode is request-driven: `num_frames == 1` means T2I, `--image-path` means I2V. On GPUs with at least 120 GiB available, auto mode keeps the Cosmos3 DiT and VAE resident for every checkpoint in the family; a 1xH200 832x480x9f, 4-step eager ABBA reduced e2e from 1.576 to 0.428 seconds with exact output parity. Cosmos3 runs one DiT per pipeline, so component offload above that threshold only buys a DiT copy out to host memory and back per request -- it cost Cosmos3-Super 720p 81f T2V ~4s of ~115s on 2xH200. |
|
||||
| Cosmos3 Edge / distilled Super | Edge T2I: 640x640, 35 steps, 1 GPU; distilled Super T2I: 640x640, fixed 4-step schedule, 4 GPUs | Start eager with `--performance-mode manual`; use `SGLANG_DISABLE_COSMOS3_GUARDRAILS=1` only for benchmark isolation | Edge is trained for 256p/480p shapes. Distilled checkpoints own their sigma schedule and force guidance 1.0; do not override steps or flow shift. Do not retry the closed experimental Cosmos BCG path without a new lifecycle design. |
|
||||
| Ideogram 4 FP8/NVFP4 | 1024x1024, native preset defaults | `--enable-torch-compile --warmup-mode request` | Do not set `--num-inference-steps` or `--guidance-scale` directly unless you also update the Ideogram preset; sampling params derive them from `preset`. |
|
||||
@@ -397,7 +402,7 @@ Use these as first commands to benchmark, not as universal winners.
|
||||
| JoyAI-Image-Edit | 1024-class TI2I, 40 steps, guidance 4.0 | `--backend=sglang --num-gpus 2 --enable-cfg-parallel --ulysses-degree 1 --enable-torch-compile --warmup-mode request --dit-layerwise-offload false --dit-cpu-offload false` | Newly supported image-edit path. Keep the input image, prompt, seed, and output size fixed; 2-GPU CFG parallel is the validated H100 starting point. |
|
||||
| FireRed-Image-Edit 1.0 / 1.1 | 1024x1024 image edit, 40 steps, guidance 4.0 | `--backend=sglang --num-gpus 2 --enable-cfg-parallel --ulysses-degree 1 --enable-torch-compile --warmup-mode request --dit-layerwise-offload false --dit-cpu-offload false` | Uses the native `QwenImageEditPlusPipeline` path. 2-GPU CFG parallel is the validated H100 starting point; benchmark 1.0 and 1.1 separately because checkpoint differences can change denoise latency. |
|
||||
| Hunyuan3D-2 shape | Shape generation, 50 steps, guidance 5.0 | `--backend=sglang --enable-torch-compile --warmup-mode request --dit-layerwise-offload false --dit-cpu-offload false` | Focus on `Hunyuan3DShapeDenoisingStage`; keep mesh export/paint timings separate from denoise. |
|
||||
| LingBot Video MoE 30B | 384x640, 17 frames, 12 steps for the current GPU case | `--model-path robbyant/lingbot-video-moe-30b-a3b --text-encoder-cpu-offload` | Native T2V path. Prompts are structured JSON captions, not raw free text; keep that contract when comparing latency or quality. Main still expands RMSNorm into PyTorch reduction chains; #35969 is an open `quality=high` Triton-dispatch candidate, not a current-main option until merged. |
|
||||
| LingBot Video MoE 30B | 384x640, 17 frames, 12 steps for the current GPU case | `--model-path robbyant/lingbot-video-moe-30b-a3b --text-encoder-cpu-offload` | Native T2V path. Prompts are structured JSON captions, not raw free text; keep that contract when comparing latency or quality. Current main can mount the fused Triton RMSNorm path at `quality=extra-high` or `quality=high`; keep `lossless` as the reference. |
|
||||
| MOVA / Helios / LingBot World | Use the benchmark/profile presets or server test cases first | `--enable-torch-compile --warmup-mode request`; pin offload and topology flags explicitly | These video/realtime families have model-specific stages and condition handling. For LingBot World causal serving, keep `--kv-cache-quant off` as the exact cache baseline before testing INT4/INT2. |
|
||||
|
||||
## Historical PR Watchlist
|
||||
|
||||
@@ -99,7 +99,8 @@ class SGLDiffusionServerAPI:
|
||||
seed: Random seed for reproducible generation
|
||||
enable_teacache: Enable TEA cache acceleration
|
||||
response_format: Response format ("b64_json" or "url")
|
||||
quality: Image quality ("auto", "standard", "hd") - only for generation
|
||||
quality: Request optimization tier ("auto", "lossless",
|
||||
"extra-high", "high") - only for generation
|
||||
style: Image style ("vivid" or "natural") - only for generation
|
||||
background: Background type ("auto", "transparent", "opaque")
|
||||
output_format: Output format ("png", "jpeg", "webp")
|
||||
|
||||
@@ -51,10 +51,17 @@ def generate_request_id() -> str:
|
||||
return str(uuid.uuid4())
|
||||
|
||||
|
||||
# Validated request-level quality levels. "lossless" is the exact reference
|
||||
# path (bit-exact against the CI golden outputs); "high" opts into validated
|
||||
# accelerated paths whose quality is guaranteed but not bit-exact.
|
||||
QUALITY_LEVELS: tuple[str, ...] = ("lossless", "high")
|
||||
# Validated request-level quality levels, ordered from the strictest numerical
|
||||
# contract to the broadest optimization set. "lossless" keeps the exact
|
||||
# reference path; "extra-high" adds only request-gated kernel fusions; "high"
|
||||
# is cumulative and may also enable model-owned approximate optimizations.
|
||||
QUALITY_LEVELS: tuple[str, ...] = ("lossless", "extra-high", "high")
|
||||
KERNEL_FUSION_QUALITY_LEVELS = frozenset({"extra-high", "high"})
|
||||
|
||||
|
||||
def quality_allows_kernel_fusions(quality: str) -> bool:
|
||||
"""Return whether a quality level includes request-gated kernel fusions."""
|
||||
return quality in KERNEL_FUSION_QUALITY_LEVELS
|
||||
|
||||
|
||||
def _sanitize_filename(name: str, replacement: str = "_", max_length: int = 150) -> str:
|
||||
@@ -141,15 +148,15 @@ class SamplingParams:
|
||||
# - "lossless" (default): the exact reference path. Output is expected to
|
||||
# be bit-identical to the HF reference implementation and to pass the
|
||||
# CI golden/ground-truth comparisons.
|
||||
# - "high": opt into validated accelerated paths. Quality stays
|
||||
# guaranteed (the intent is to back every such path with mathematical
|
||||
# acceptance thresholds, e.g. PSNR > 25 against the reference), but
|
||||
# the output is no longer bit-exact versus the HF reference or the CI
|
||||
# ground truth.
|
||||
# - "extra-high": add only validated kernel fusions. These may change
|
||||
# half-precision rounding order, so output is not bit-exact versus the
|
||||
# reference, but this tier does not itself enable sparse or approximate
|
||||
# optimizations.
|
||||
# - "high": include every "extra-high" fusion and allow model-owned
|
||||
# approximate optimizations such as sparse computation or feature
|
||||
# caching. These paths require model-specific quality validation.
|
||||
#
|
||||
# Models that support "high" must validate the deployment and workload
|
||||
# explicitly. It intentionally participates in the dynamic-batch
|
||||
# signature.
|
||||
# It intentionally participates in the dynamic-batch signature.
|
||||
quality: str = "lossless"
|
||||
|
||||
# Frame interpolation
|
||||
@@ -1120,10 +1127,12 @@ class SamplingParams:
|
||||
help=(
|
||||
"Request-level quality: 'lossless' (default) keeps the exact "
|
||||
"reference path, bit-exact against the reference "
|
||||
"implementation; 'high' opts into the model-owned validated "
|
||||
"accelerated path, whose quality stays guaranteed but is not "
|
||||
"bit-exact. Support and validated deployment constraints are "
|
||||
"model-specific."
|
||||
"implementation; 'extra-high' adds only request-gated kernel "
|
||||
"fusions and does not itself enable sparse or approximate "
|
||||
"optimization; 'high' includes every extra-high fusion and "
|
||||
"may also enable "
|
||||
"model-owned approximate paths. Support and validated "
|
||||
"deployment constraints are model-specific."
|
||||
),
|
||||
)
|
||||
add_argument(
|
||||
|
||||
@@ -169,10 +169,11 @@ def _flux_norm_modulate(
|
||||
"""``norm(x) * (1 + scale) + shift`` for the FLUX adaLN sites.
|
||||
|
||||
Priority: (1) the bit-exact single-kernel LN+modulate -- lossless, so it
|
||||
needs no quality gate and also supersedes the ``quality="high"`` affine
|
||||
needs no quality gate and also supersedes the request-gated affine
|
||||
fold wherever it verifies; (2) when the site is mounted
|
||||
(``quality="high"``) and the bit-exact kernel is unavailable, the
|
||||
modulate folded into the LN affine (one aten kernel; not bit-exact);
|
||||
(``quality="extra-high"`` or ``"high"``) and the bit-exact kernel is
|
||||
unavailable, the modulate folded into the LN affine (one aten kernel; not
|
||||
bit-exact);
|
||||
(3) affine-free LayerNorm + the bit-exact fused modulate.
|
||||
"""
|
||||
out = _flux_fused_ln_modulate(norm, x, scale, shift)
|
||||
@@ -387,7 +388,7 @@ class FluxGELU(nn.Module):
|
||||
prefix=f"{prefix}.proj" if prefix else "proj",
|
||||
)
|
||||
self.gelu = nn.GELU(approximate="tanh")
|
||||
# quality="high" fusion site: up-proj GEMM + tanh-GELU in the cublasLt
|
||||
# extra-high/high fusion site: up-proj GEMM + tanh-GELU in the cublasLt
|
||||
# epilogue. Off by default; mounted per batch by the denoising stage.
|
||||
mark_fused_gelu_site(self, "proj")
|
||||
|
||||
@@ -407,7 +408,7 @@ class FluxFusedGELUProj(nn.Module):
|
||||
``approximate="tanh"`` that keeps the ``net.0.proj`` parameter path. The
|
||||
default path is the bit-exact reference (plain Linear + tanh-GELU); the
|
||||
cublasLt GELU epilogue is mounted per batch by the denoising stage for
|
||||
quality="high" requests only.
|
||||
requests with ``quality="extra-high"`` or ``quality="high"`` only.
|
||||
"""
|
||||
|
||||
def __init__(self, proj: nn.Linear):
|
||||
@@ -790,7 +791,7 @@ class FluxSingleTransformerBlock(nn.Module):
|
||||
prefix=f"{prefix}.proj_mlp" if prefix else "proj_mlp",
|
||||
)
|
||||
self.act_mlp = nn.GELU(approximate="tanh")
|
||||
# quality="high" fusion site: proj_mlp GEMM + tanh-GELU in the
|
||||
# extra-high/high fusion site: proj_mlp GEMM + tanh-GELU in the
|
||||
# cublasLt epilogue (mounted per batch by the denoising stage).
|
||||
mark_fused_gelu_site(self, "proj_mlp")
|
||||
proj_out_cls = (
|
||||
@@ -954,7 +955,7 @@ class FluxTransformerBlock(nn.Module):
|
||||
|
||||
self.norm2 = LayerNorm(dim, eps=1e-6, elementwise_affine=False)
|
||||
self.norm2_context = LayerNorm(dim, eps=1e-6, elementwise_affine=False)
|
||||
# quality="high" site: the norm2/norm2_context modulate folds into the
|
||||
# extra-high/high site: norm2/norm2_context modulate folds into the
|
||||
# LN affine when mounted.
|
||||
mark_fused_ln_modulate_site(self)
|
||||
|
||||
@@ -1009,7 +1010,7 @@ class FluxTransformerBlock(nn.Module):
|
||||
activation_fn="gelu-approximate",
|
||||
)
|
||||
# Re-home each FF's tanh-GELU up-projection onto a marked
|
||||
# quality="high" fusion site (bit-exact reference by default).
|
||||
# extra-high/high fusion site (bit-exact reference by default).
|
||||
self.ff.net[0] = FluxFusedGELUProj(self.ff.net[0].proj)
|
||||
self.ff_context.net[0] = FluxFusedGELUProj(self.ff_context.net[0].proj)
|
||||
|
||||
|
||||
@@ -447,7 +447,7 @@ class GlmImageGELU(nn.Module):
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.proj" if prefix else "proj",
|
||||
)
|
||||
# quality="high" fusion site: up-proj GEMM + tanh-GELU in the cublasLt
|
||||
# extra-high/high fusion site: up-proj GEMM + tanh-GELU in cublasLt
|
||||
# epilogue. Off by default; mounted per batch by the denoising stage.
|
||||
mark_fused_gelu_site(self, "proj")
|
||||
|
||||
|
||||
@@ -421,7 +421,7 @@ def _norm_scale(
|
||||
norm: Ideogram4RMSNorm,
|
||||
enable_fused: bool,
|
||||
) -> torch.Tensor:
|
||||
"""``RMSNorm(x) * (1 + scale)``, fused for ``quality="high"`` batches."""
|
||||
"""``RMSNorm(x) * (1 + scale)``, fused at extra-high or high quality."""
|
||||
if enable_fused:
|
||||
y = fused_rmsnorm_scale(
|
||||
x,
|
||||
@@ -455,7 +455,7 @@ def _gate_residual(
|
||||
norm: Ideogram4RMSNorm,
|
||||
enable_fused: bool,
|
||||
) -> torch.Tensor:
|
||||
"""``residual + tanh(gate) * RMSNorm(x)``, fused for ``quality="high"``."""
|
||||
"""``residual + tanh(gate) * RMSNorm(x)``, fused at extra-high or high."""
|
||||
if enable_fused:
|
||||
y = fused_rmsnorm_tanh_residual(
|
||||
x,
|
||||
@@ -511,7 +511,7 @@ class Ideogram4TransformerBlock(nn.Module):
|
||||
self.ffn_norm1 = Ideogram4RMSNorm(hidden_size, eps=norm_eps)
|
||||
self.attention_norm2 = Ideogram4RMSNorm(hidden_size, eps=norm_eps)
|
||||
self.ffn_norm2 = Ideogram4RMSNorm(hidden_size, eps=norm_eps)
|
||||
# quality="high" fusion sites: each RMSNorm modulate/gate chain
|
||||
# extra-high/high fusion sites: each RMSNorm modulate/gate chain
|
||||
# collapses into one Triton kernel (Z-Image bf16-native suite). Off by
|
||||
# default (bit-exact reference path); mounted per batch by the
|
||||
# denoising stage.
|
||||
|
||||
@@ -222,7 +222,7 @@ class _LongCatFFN(nn.Module):
|
||||
]
|
||||
)
|
||||
self.act = nn.GELU(approximate="tanh")
|
||||
# quality="high" site: up-proj GEMM + tanh-GELU cublasLt epilogue. Off by
|
||||
# extra-high/high site: up-proj GEMM + tanh-GELU epilogue. Off by
|
||||
# default; the denoising stage mounts it per batch. The ModuleDict holds
|
||||
# `proj` in _modules, so getattr resolves it for the fusion helper.
|
||||
mark_fused_gelu_site(self.net[0], "proj")
|
||||
@@ -511,7 +511,7 @@ class _SingleTransformerBlock(nn.Module):
|
||||
prefix=f"{prefix}.proj_mlp",
|
||||
)
|
||||
self.act_mlp = nn.GELU(approximate="tanh")
|
||||
# quality="high" site: proj_mlp GEMM + tanh-GELU cublasLt epilogue,
|
||||
# extra-high/high site: proj_mlp GEMM + tanh-GELU epilogue,
|
||||
# mounted per batch by the denoising stage; off (bit-exact) by default.
|
||||
mark_fused_gelu_site(self, "proj_mlp")
|
||||
# proj_out: RowParallelLinear reduces sharded [attn | mlp] concat via
|
||||
|
||||
@@ -198,7 +198,7 @@ def _ltx2_rms_norm_modulate(
|
||||
"""``rms_norm(x) * (1 + scale) + shift`` for the LTX-2 adaLN sites.
|
||||
|
||||
Folds the weightless RMSNorm and the modulate into one kernel when the
|
||||
``quality="high"`` fusion is mounted on ``block`` and the per-call guard
|
||||
request-gated fusion is mounted on ``block`` and the per-call guard
|
||||
passes; otherwise the verbatim eager reference chain (the ``lossless``
|
||||
default). The fused kernel is not bit-exact (<=1 bf16 ULP) so it is gated
|
||||
on the request-scoped mount rather than a runtime self-check.
|
||||
|
||||
@@ -824,7 +824,7 @@ class QwenImageCrossAttention(nn.Module):
|
||||
)
|
||||
if self._unquantized_added_qkv_is_packed:
|
||||
# Packing changes BF16 GEMM reduction association. Keep it
|
||||
# off for lossless requests and mount it for quality=high.
|
||||
# off for lossless and mount it at extra-high or high.
|
||||
mark_qwen_image_added_qkv_site(self)
|
||||
else:
|
||||
self.add_q_proj = ColumnParallelLinear(
|
||||
@@ -1171,7 +1171,7 @@ class QwenImageGELU(nn.Module):
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.proj",
|
||||
)
|
||||
# quality="high" fusion site: up-proj GEMM + tanh-GELU in the cublasLt
|
||||
# Extra-high-or-higher fusion site: up-proj GEMM + tanh-GELU in cublasLt
|
||||
# epilogue. Off by default; mounted per batch by the denoising stage.
|
||||
mark_fused_gelu_site(self, "proj")
|
||||
|
||||
|
||||
@@ -8,8 +8,9 @@ decoder module family (``ResnetBlock2D`` GroupNorm+SiLU chains,
|
||||
|
||||
All rewrites are mathematically exact re-associations of the original
|
||||
operators. Wrappers are installed once at VAE load and dispatch on a
|
||||
decode-scoped :class:`VaeFastPathGate`: ``quality == "high"`` runs the fast
|
||||
paths, the ``"lossless"`` default runs the original module path bit-for-bit.
|
||||
decode-scoped :class:`VaeFastPathGate`: ``quality="extra-high"`` and
|
||||
``quality="high"`` run the fast paths, while the ``"lossless"`` default runs
|
||||
the original module path bit-for-bit.
|
||||
|
||||
- channels_last: run the decoder in NHWC so cuDNN convs skip the transpose
|
||||
kernels; parameter layout is swapped at decode entry to match the gate.
|
||||
|
||||
@@ -3,10 +3,10 @@
|
||||
|
||||
Fuses every decoder ``WanRMS_norm -> SiLU`` chain into one Triton kernel on
|
||||
the channels_last_3d layout. Wrappers are installed once at VAE load and
|
||||
dispatch on a decode-scoped :class:`VaeFastPathGate`: ``quality == "high"``
|
||||
runs the fused kernel (not bitwise-identical to aten, hence gated), the
|
||||
``"lossless"`` default runs the original module path bit-for-bit. Install is
|
||||
all-or-nothing and fail-closed.
|
||||
dispatch on a decode-scoped :class:`VaeFastPathGate`: ``quality="extra-high"``
|
||||
and ``quality="high"`` run the fused kernel (not bitwise-identical to aten,
|
||||
hence gated), while the ``"lossless"`` default runs the original module path
|
||||
bit-for-bit. Install is all-or-nothing and fail-closed.
|
||||
"""
|
||||
|
||||
import torch
|
||||
|
||||
@@ -10,6 +10,9 @@ import weakref
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
|
||||
from sglang.multimodal_gen.configs.sample.sampling_params import (
|
||||
quality_allows_kernel_fusions,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.distributed import (
|
||||
get_decode_parallel_world_size,
|
||||
get_local_torch_device,
|
||||
@@ -333,7 +336,10 @@ class DecodingStage(PipelineStage):
|
||||
assert vae is not None
|
||||
self.vae = vae
|
||||
|
||||
with use_vae_fast_path(vae, batch.sampling_params.quality == "high"):
|
||||
with use_vae_fast_path(
|
||||
vae,
|
||||
quality_allows_kernel_fusions(batch.sampling_params.quality),
|
||||
):
|
||||
frames = self.decode(batch.latents, server_args, vae_dtype=vae_dtype)
|
||||
|
||||
# decode trajectory latents if needed
|
||||
|
||||
@@ -49,6 +49,9 @@ from sglang.multimodal_gen.configs.pipeline_configs.flux import (
|
||||
FluxPipelineConfig,
|
||||
)
|
||||
from sglang.multimodal_gen.configs.pipeline_configs.zimage import ZImagePipelineConfig
|
||||
from sglang.multimodal_gen.configs.sample.sampling_params import (
|
||||
quality_allows_kernel_fusions,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.breakable_cuda_graph import (
|
||||
prompt_padding as bcg_utils,
|
||||
)
|
||||
@@ -326,7 +329,7 @@ class DenoisingStage(PipelineStage, RolloutDenoisingMixin):
|
||||
self._cache_dit_request_overrides: dict[str, Any] = {}
|
||||
# Overrides key the mounted hooks were built from; None when unmounted.
|
||||
self._cache_dit_active_key: tuple | None = None
|
||||
# Whether request-scoped quality="high" fusions are currently mounted.
|
||||
# Whether request-scoped extra-high-or-higher fusions are mounted.
|
||||
self._quality_fusions_mounted = False
|
||||
self._torch_compile_registry = CompiledModuleRegistry()
|
||||
# Breakable CUDA graph runners, one per transformer module (lazy).
|
||||
@@ -666,17 +669,18 @@ class DenoisingStage(PipelineStage, RolloutDenoisingMixin):
|
||||
return stage_backend
|
||||
|
||||
def _maybe_toggle_quality_fusions(self, batch: Req) -> None:
|
||||
"""Mount/unmount the ``quality="high"`` fusions for this batch.
|
||||
"""Mount/unmount request-gated kernel fusions for this batch.
|
||||
|
||||
These fusions are numerically equivalent only at half-precision
|
||||
rounding level (not bit-exact), so they are mounted for
|
||||
``quality="high"`` requests and unmounted otherwise. The
|
||||
``"lossless"`` default runs the reference path bit-for-bit. ``quality``
|
||||
participates in the dynamic-batch signature, making this transition
|
||||
safe at the batch boundary. Mounting is all-or-nothing per transformer
|
||||
and fusion family; models without marked sites are no-ops.
|
||||
rounding level (not bit-exact), so they are mounted for both
|
||||
``quality="extra-high"`` and ``quality="high"``. The ``"lossless"``
|
||||
default runs the reference path bit-for-bit. ``quality`` participates
|
||||
in the dynamic-batch signature, making this transition safe at the
|
||||
batch boundary. Mounting is all-or-nothing per transformer and fusion
|
||||
family; models without marked sites are no-ops.
|
||||
"""
|
||||
want = getattr(batch.sampling_params, "quality", "lossless") == "high"
|
||||
quality = getattr(batch.sampling_params, "quality", "lossless")
|
||||
want = quality_allows_kernel_fusions(quality)
|
||||
if want == self._quality_fusions_mounted:
|
||||
return
|
||||
mounted_fusions: set[str] = set()
|
||||
@@ -694,7 +698,7 @@ class DenoisingStage(PipelineStage, RolloutDenoisingMixin):
|
||||
unmount(transformer)
|
||||
descriptions = ", ".join(sorted(mounted_fusions))
|
||||
raise ValueError(
|
||||
"quality='high' cannot be used with breakable CUDA graphs for "
|
||||
f"quality={quality!r} cannot be used with breakable CUDA graphs for "
|
||||
f"this model because its request-scoped DiT fusions "
|
||||
f"({descriptions}) do not match the lossless warmup graphs. "
|
||||
"Disable breakable CUDA graphs or use quality='lossless'."
|
||||
@@ -702,7 +706,7 @@ class DenoisingStage(PipelineStage, RolloutDenoisingMixin):
|
||||
|
||||
self._quality_fusions_mounted = want
|
||||
for description in sorted(mounted_fusions):
|
||||
logger.info("Mounted %s for quality=high", description)
|
||||
logger.info("Mounted %s for quality=%s", description, quality)
|
||||
|
||||
def _cache_dit_dual_model_name(self) -> str:
|
||||
return "wan2.2"
|
||||
|
||||
@@ -685,8 +685,8 @@ class CudaPlatformBase(Platform):
|
||||
"""Install the quality-gated FLUX.2 / AutoencoderKL / Wan VAE decoder
|
||||
fast paths.
|
||||
|
||||
Requests with quality == "high" run the fast paths; the "lossless"
|
||||
default runs the original module path bit-for-bit. See
|
||||
Requests with quality="extra-high" or "high" run the fast paths; the
|
||||
"lossless" default runs the original module path bit-for-bit. See
|
||||
flux2_vae_cuda_opt and wan_vae_cuda_opt for details.
|
||||
"""
|
||||
try:
|
||||
|
||||
@@ -70,22 +70,26 @@ class TestQualityFusionBCGCompatibility(unittest.TestCase):
|
||||
def _batch(quality: str):
|
||||
return SimpleNamespace(sampling_params=SimpleNamespace(quality=quality))
|
||||
|
||||
def test_rejects_high_when_dit_fusion_would_replace_captured_graph(self):
|
||||
unmounted = []
|
||||
handlers = (
|
||||
(
|
||||
"test fusion",
|
||||
lambda _: True,
|
||||
lambda transformer: unmounted.append(transformer),
|
||||
),
|
||||
)
|
||||
def test_rejects_fusion_levels_when_they_would_replace_captured_graph(self):
|
||||
for quality in ("extra-high", "high"):
|
||||
with self.subTest(quality=quality):
|
||||
unmounted = []
|
||||
handlers = (
|
||||
(
|
||||
"test fusion",
|
||||
lambda _: True,
|
||||
lambda transformer: unmounted.append(transformer),
|
||||
),
|
||||
)
|
||||
|
||||
with patch.object(denoising_module, "_QUALITY_FUSION_HANDLERS", handlers):
|
||||
with self.assertRaisesRegex(ValueError, "lossless warmup graphs"):
|
||||
self.stage._maybe_toggle_quality_fusions(self._batch("high"))
|
||||
with patch.object(
|
||||
denoising_module, "_QUALITY_FUSION_HANDLERS", handlers
|
||||
):
|
||||
with self.assertRaisesRegex(ValueError, "lossless warmup graphs"):
|
||||
self.stage._maybe_toggle_quality_fusions(self._batch(quality))
|
||||
|
||||
self.assertEqual(unmounted, [self.stage.transformer])
|
||||
self.assertFalse(self.stage._quality_fusions_mounted)
|
||||
self.assertEqual(unmounted, [self.stage.transformer])
|
||||
self.assertFalse(self.stage._quality_fusions_mounted)
|
||||
|
||||
def test_allows_high_when_model_has_no_dit_quality_fusions(self):
|
||||
handlers = (("test fusion", lambda _: False, lambda _: None),)
|
||||
@@ -95,6 +99,21 @@ class TestQualityFusionBCGCompatibility(unittest.TestCase):
|
||||
|
||||
self.assertTrue(self.stage._quality_fusions_mounted)
|
||||
|
||||
def test_high_keeps_extra_high_fusions_mounted(self):
|
||||
self.stage.server_args.enable_breakable_cuda_graph = False
|
||||
mounted = []
|
||||
handlers = (
|
||||
("test fusion", lambda _: mounted.append(True) or True, lambda _: None),
|
||||
)
|
||||
|
||||
with patch.object(denoising_module, "_QUALITY_FUSION_HANDLERS", handlers):
|
||||
self.stage._maybe_toggle_quality_fusions(self._batch("extra-high"))
|
||||
self.assertTrue(self.stage._quality_fusions_mounted)
|
||||
self.stage._maybe_toggle_quality_fusions(self._batch("high"))
|
||||
|
||||
self.assertEqual(mounted, [True])
|
||||
self.assertTrue(self.stage._quality_fusions_mounted)
|
||||
|
||||
|
||||
def _fake_cache_dit_batch(*, is_warmup: bool) -> SimpleNamespace:
|
||||
return SimpleNamespace(
|
||||
|
||||
@@ -257,6 +257,12 @@ class TestDiffusionBenchmarkSkill(unittest.TestCase):
|
||||
self.assertIn("--quality=high", high_cmd)
|
||||
self.assertNotIn("--enable-breakable-cuda-graph", high_cmd)
|
||||
|
||||
extra_high_cmd = module.build_sglang_cmd(
|
||||
"longcat-image", quality="extra-high"
|
||||
)
|
||||
self.assertIn("--quality=extra-high", extra_high_cmd)
|
||||
self.assertNotIn("--enable-breakable-cuda-graph", extra_high_cmd)
|
||||
|
||||
bcg_cmd = module.build_sglang_cmd(
|
||||
"longcat-image",
|
||||
breakable_cuda_graph=True,
|
||||
@@ -538,6 +544,37 @@ class TestDiffusionBenchmarkSkill(unittest.TestCase):
|
||||
[module.BCG_LATE_QUALITY_FUSION_SIGNAL],
|
||||
)
|
||||
|
||||
def test_extra_high_bcg_rejects_quality_fusion_mounted_after_capture(self):
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
temp_root = Path(tmpdir)
|
||||
module = _load_benchmark_module(temp_root)
|
||||
output_dir = temp_root / "outputs"
|
||||
output_dir.mkdir()
|
||||
|
||||
with patch.object(module.subprocess, "Popen") as popen:
|
||||
popen.return_value.stdout = iter(
|
||||
(
|
||||
"[Diffusion BCG] captured 3 segment(s)\n",
|
||||
"Mounted Qwen fused added-QKV for quality=extra-high\n",
|
||||
)
|
||||
)
|
||||
popen.return_value.wait.return_value = 0
|
||||
result = module._run_benchmark_once_impl(
|
||||
"longcat-image",
|
||||
"bcg-extra-high",
|
||||
output_dir,
|
||||
warmup=False,
|
||||
quality="extra-high",
|
||||
breakable_cuda_graph=True,
|
||||
cuda_visible_devices="0",
|
||||
)
|
||||
|
||||
self.assertTrue(result["error"])
|
||||
self.assertEqual(
|
||||
result["bcg_invalid_signals"],
|
||||
[module.BCG_LATE_QUALITY_FUSION_SIGNAL],
|
||||
)
|
||||
|
||||
def test_quality_bcg_matrix_reuses_one_gpu_set_and_cleans_once(self):
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
temp_root = Path(tmpdir)
|
||||
@@ -564,7 +601,7 @@ class TestDiffusionBenchmarkSkill(unittest.TestCase):
|
||||
cleanup_model_cache=True,
|
||||
)
|
||||
|
||||
self.assertEqual(len(results), 8)
|
||||
self.assertEqual(len(results), 12)
|
||||
self.assertEqual(
|
||||
[
|
||||
(call[2]["quality"], call[2]["breakable_cuda_graph"])
|
||||
|
||||
@@ -23,6 +23,7 @@ from sglang.multimodal_gen.runtime.managers.memory_managers.component_residency
|
||||
LAYERWISE_OFFLOAD,
|
||||
RESIDENT,
|
||||
)
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.stages.denoising import DenoisingStage
|
||||
from sglang.multimodal_gen.runtime.pipelines_core.stages.model_specific_stages.minimax_h3.release_metadata import (
|
||||
MiniMaxH3PartitionAdmissionStage,
|
||||
MiniMaxH3ReleaseMetadata,
|
||||
@@ -351,6 +352,31 @@ def test_high_quality_request_warns_when_bcg_suppresses_cache_dit():
|
||||
)
|
||||
|
||||
|
||||
def test_extra_high_quality_does_not_enable_h3_cache_dit():
|
||||
stage = MiniMaxH3DenoisingStage.__new__(MiniMaxH3DenoisingStage)
|
||||
stage.server_args = SimpleNamespace(enable_breakable_cuda_graph=False)
|
||||
stage._cache_dit_enabled = False
|
||||
stage._minimax_h3_cache_mode = None
|
||||
stage._minimax_h3_quality = "lossless"
|
||||
batch = SimpleNamespace(
|
||||
sampling_params=SimpleNamespace(
|
||||
quality="extra-high",
|
||||
_explicit_fields={"quality"},
|
||||
enable_cache_dit=None,
|
||||
cache_dit_params=None,
|
||||
)
|
||||
)
|
||||
|
||||
# Even a server-wide generic Cache-DiT default must not turn an explicit
|
||||
# fusion-only quality tier into an approximate H3 request.
|
||||
with patch.object(DenoisingStage, "_cache_dit_requested", return_value=True):
|
||||
stage._maybe_enable_cache_dit(50, batch)
|
||||
|
||||
assert stage._minimax_h3_quality == "extra-high"
|
||||
assert stage._minimax_h3_cache_mode is None
|
||||
assert not stage._cache_dit_enabled
|
||||
|
||||
|
||||
def test_quality_admission_fails_closed_outside_validated_request():
|
||||
metadata = MiniMaxH3ReleaseMetadata.from_model_index(
|
||||
{
|
||||
@@ -405,6 +431,9 @@ def test_quality_admission_fails_closed_outside_validated_request():
|
||||
server_args.attention_backend = "sage_attn"
|
||||
assert stage.forward(batch, server_args) is batch
|
||||
|
||||
batch.sampling_params.quality = "extra-high"
|
||||
assert stage.forward(batch, server_args) is batch
|
||||
|
||||
batch.sampling_params.quality = "ultra"
|
||||
server_args.attention_backend = None
|
||||
with pytest.raises(ValueError, match="quality must be one of"):
|
||||
|
||||
@@ -58,6 +58,7 @@ def test_runtime_sampling_quality_preserves_the_openai_default():
|
||||
assert _runtime_sampling_quality(None) is None
|
||||
assert _runtime_sampling_quality("auto") is None
|
||||
assert _runtime_sampling_quality("lossless") == "lossless"
|
||||
assert _runtime_sampling_quality("extra-high") == "extra-high"
|
||||
assert _runtime_sampling_quality("high") == "high"
|
||||
|
||||
|
||||
|
||||
@@ -149,6 +149,13 @@ class TestDiffusionPrecisionConsistency(unittest.TestCase):
|
||||
),
|
||||
torch.bfloat16,
|
||||
)
|
||||
self.assertEqual(
|
||||
resolve_decode_precision(
|
||||
self._server_args(vae_decode_precision_high="bf16"),
|
||||
quality="extra-high",
|
||||
),
|
||||
torch.float16,
|
||||
)
|
||||
self.assertEqual(
|
||||
resolve_decode_precision(
|
||||
self._server_args(vae_decode_precision_high="bf16"),
|
||||
|
||||
@@ -26,8 +26,10 @@ from sglang.multimodal_gen.configs.sample.glmimage import (
|
||||
)
|
||||
from sglang.multimodal_gen.configs.sample.qwenimage import QwenImageSamplingParams
|
||||
from sglang.multimodal_gen.configs.sample.sampling_params import (
|
||||
QUALITY_LEVELS,
|
||||
SamplingParams,
|
||||
_json_safe,
|
||||
quality_allows_kernel_fusions,
|
||||
)
|
||||
from sglang.multimodal_gen.configs.sample.spectrum import SpectrumParams
|
||||
from sglang.multimodal_gen.configs.sample.teacache import TeaCacheParams
|
||||
@@ -52,9 +54,15 @@ class TestSamplingParamsValidate(unittest.TestCase):
|
||||
def test_quality_defaults_to_lossless(self):
|
||||
self.assertEqual(SamplingParams().quality, "lossless")
|
||||
|
||||
def test_quality_accepts_the_two_validated_levels(self):
|
||||
self.assertEqual(SamplingParams(quality="lossless").quality, "lossless")
|
||||
self.assertEqual(SamplingParams(quality="high").quality, "high")
|
||||
def test_quality_levels_are_cumulative(self):
|
||||
self.assertEqual(QUALITY_LEVELS, ("lossless", "extra-high", "high"))
|
||||
for quality in QUALITY_LEVELS:
|
||||
with self.subTest(quality=quality):
|
||||
self.assertEqual(SamplingParams(quality=quality).quality, quality)
|
||||
|
||||
self.assertFalse(quality_allows_kernel_fusions("lossless"))
|
||||
self.assertTrue(quality_allows_kernel_fusions("extra-high"))
|
||||
self.assertTrue(quality_allows_kernel_fusions("high"))
|
||||
|
||||
def test_quality_rejects_invalid_values(self):
|
||||
for bad in ("ultra", "draft", "fast", "", True, 1):
|
||||
@@ -338,9 +346,11 @@ class TestSamplingParamsCliArgs(unittest.TestCase):
|
||||
|
||||
def test_quality_is_request_scoped_cli_arg(self):
|
||||
self.assertNotIn("quality", self._parse_cli_kwargs([]))
|
||||
self.assertEqual(
|
||||
self._parse_cli_kwargs(["--quality", "high"])["quality"], "high"
|
||||
)
|
||||
for quality in ("extra-high", "high"):
|
||||
with self.subTest(quality=quality):
|
||||
self.assertEqual(
|
||||
self._parse_cli_kwargs(["--quality", quality])["quality"], quality
|
||||
)
|
||||
|
||||
def test_get_cli_args_maps_spectrum_prefixed_flags(self):
|
||||
kwargs = self._parse_cli_kwargs(
|
||||
|
||||
Reference in New Issue
Block a user