From b64fd800d4cbaa6ee885aa21ad37100bfbab9627 Mon Sep 17 00:00:00 2001 From: Xiaoyu Zhang <1182563586@qq.com> Date: Mon, 3 Aug 2026 12:44:01 +0800 Subject: [PATCH] docs(diffusion): update skills for MiniMax-H3 (#33282) --- .../sglang-diffusion-add-model/SKILL.md | 42 ++++++- .../references/testing-and-accuracy.md | 13 ++ .../SKILL.md | 10 +- .../benchmark-and-profile.md | 118 +++++++++++++++++- .../existing-fast-paths.md | 32 ++++- .../scripts/bench_diffusion_denoise.py | 43 ++++++- .../sglang-diffusion-modelopt-quant/SKILL.md | 16 +++ .../sglang-diffusion-performance/SKILL.md | 61 ++++++++- 8 files changed, 325 insertions(+), 10 deletions(-) diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-add-model/SKILL.md b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-add-model/SKILL.md index 495557132..94a00d3dd 100644 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-add-model/SKILL.md +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-add-model/SKILL.md @@ -7,7 +7,7 @@ description: Use when adding a new diffusion model or Diffusers pipeline to SGLa Use this skill when adding a new diffusion model or pipeline variant to `sglang.multimodal_gen`. -## Two Pipeline Styles +## Three Pipeline Styles ### Style A: Hybrid Monolithic Pipeline (Recommended) @@ -33,11 +33,32 @@ This style is appropriate when: See existing Modular examples: `QwenImagePipeline` (uses `add_standard_t2i_stages`), `FluxPipeline`, `WanPipeline`, `SanaPipeline`, `StableDiffusion3Pipeline`, and `ZImagePipeline`. +### Style C: Native Task-Contract Pipeline + +Use this only when one checkpoint exposes multiple tightly coupled modalities +or request profiles that cannot be represented safely by generic image/video +sampling fields. MiniMax-H3 is the reference: it selects FL2VA or Ref2VA +weights from one root model ID, validates canonical `task` / `conditions` / +`target` requests before queueing, packs text/video/audio tokens into one +denoise sequence, and returns synchronized video plus audio. + +This style still uses `ComposedPipelineBase`, but owns a model-specific chain +under `stages/model_specific_stages//`. Keep request validation, media +materialization, packed-sequence construction, per-modality encode/decode, and +presentation as explicit stages. Do not force coupled state into the standard +`DenoisingStage` / `DecodingStage` contract just to resemble a simpler model. + +Choose this style only with source evidence that the public API, scheduler, or +joint latent state needs it. Preserve one canonical request object from API +admission through offline generation and server execution so the two entry +points cannot silently diverge. + ### How to Choose | Situation | Recommended Style | |-----------|-------------------| | Model has unique/complex pre-processing (VLM captioning, AR token generation, custom latent packing, etc.) | **Hybrid** — consolidate into a BeforeDenoisingStage | +| Model jointly denoises multiple modalities or exposes partitioned task contracts from one root checkpoint | **Native task contract** — use MiniMax-H3 as the reference and keep model-specific stages explicit | | Model fits neatly into standard text-to-image or text+image-to-image pattern | **Modular** — use `add_standard_t2i_stages()` / `add_standard_ti2i_stages()` | | Porting a Diffusers pipeline with many custom steps | **Hybrid** — copy the `__call__` logic into a single stage | | Adding a variant of an existing model that shares most logic | **Modular** — reuse existing stages, customize via PipelineConfig callbacks | @@ -94,7 +115,7 @@ Once you have the reference code, study it thoroughly: **Before creating any new files, check whether an existing pipeline or stage can be reused or extended.** Only create new pipelines/stages when the existing ones would require extensive modifications or when no similar implementation exists. Specifically: -1. **Compare the new model's architecture against existing pipelines** before creating files. Current native families include LTX-2/2.3, HunyuanVideo/FastHunyuan, Wan/FastWan/TurboWan/LingBot World, MOVA, FLUX/FLUX.2/Klein, Z-Image, Qwen-Image/edit/layered, GLM-Image, SD3, Hunyuan3D, Helios, Cosmos3, SANA/SANA-WM, FireRed, ERNIE-Image, JoyAI, and Ideogram4. If the new model shares most of its structure with an existing one (e.g., same text encoders, similar latent format, compatible denoising loop), prefer: +1. **Compare the new model's architecture against existing pipelines** before creating files. Current native families include MiniMax-H3, LTX-2/2.3, HunyuanVideo/FastHunyuan, Wan/FastWan/TurboWan/LingBot World, MOVA, FLUX/FLUX.2/Klein, Z-Image, Qwen-Image/edit/layered, GLM-Image, SD3, Hunyuan3D, Helios, Cosmos3, SANA/SANA-WM, FireRed, ERNIE-Image, JoyAI, and Ideogram4. If the new model shares most of its structure with an existing one (e.g., same text encoders, similar latent format, compatible denoising loop), prefer: - Adding a new config variant to the existing pipeline rather than creating a new pipeline class - Reusing the existing `BeforeDenoisingStage` with minor parameter differences - Using `add_standard_t2i_stages()` / `add_standard_ti2i_stages()` / `add_standard_ti2v_stages()` if the model fits standard patterns @@ -579,6 +600,12 @@ After implementation, **you must verify that the generated output is not noise** | FireRed/JoyAI image edit | `runtime/pipelines/qwen_image.py`, `runtime/pipelines/joy_image.py` | FireRed reuses Qwen edit-plus config; JoyAI has its own edit pipeline | | Wan | `runtime/pipelines/wan_pipeline.py` | Uses `add_standard_ti2v_stages()` | +### Native Task-Contract Style (coupled multimodal requests) + +| Model | Pipeline | Request / stage references | +|-------|----------|----------------------------| +| MiniMax-H3 | `runtime/pipelines/minimax_h3_pipeline.py` | `configs/sample/minimax_h3.py` owns the canonical request fields; `stages/model_specific_stages/minimax_h3/` owns admission, material I/O, packed video/audio/text denoising, separate video/audio VAE work, and synchronized presentation | + --- ## Checklist @@ -607,6 +634,17 @@ Before submitting, verify: - [ ] **BeforeDenoisingStage** at `stages/model_specific_stages/{model_name}.py` - [ ] `BeforeDenoisingStage.forward()` populates all fields needed by `DenoisingStage` +**Native task-contract style only:** + +- [ ] Root checkpoint plus variant selection maps to the intended partition; + do not require users to discover internal subdirectories +- [ ] Offline `generate` and HTTP serving lower through the same validated + request contract +- [ ] Task, condition role/order, target canvas/time, and output container are + rejected early when invalid +- [ ] Joint-modality correctness covers every output stream; a valid video is + insufficient when the model also generates audio or action data + ## Common Pitfalls 1. **`batch.sigmas` must be a Python list**, not a numpy array. Use `.tolist()` to convert. diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-add-model/references/testing-and-accuracy.md b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-add-model/references/testing-and-accuracy.md index 2a8f0c619..fe10dc695 100644 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-add-model/references/testing-and-accuracy.md +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-add-model/references/testing-and-accuracy.md @@ -33,6 +33,15 @@ B200-only groups are not currently inputs to the component-accuracy selector; add or identify a representative regular GPU case when that coverage is required. +Larger-topology smoke cases need the same explicit decision even when they are +not enrolled in component accuracy. MiniMax-H3's current +`MINIMAX_H3_FOUR_GPU_H100_CASES` is the reference: it exercises a real FL2VA +request with TP2 + Ulysses2, but deliberately disables component accuracy and +pipeline consistency because its native joint video/audio components do not +have a directly comparable Diffusers pipeline contract. Pair this GPU smoke +case with focused unit tests for request admission, packed-sequence layout, +denoise scheduling, media handling, and VAE parallel-mode rejection. + The component-accuracy harness compares SGLang components against Diffusers/HF reference components. This is stricter than pipeline-level inference. New GPU cases commonly fail here for one of three reasons: @@ -87,6 +96,10 @@ Tests should cover: - single-GPU inference producing non-noise output - multi-GPU inference if TP/SP is supported - relevant unit tests for new math, parsing, scheduling, or loader behavior +- every generated modality and delivery contract. For a joint model such as + MiniMax-H3, validate the MP4 video stream, synchronized audio stream, frame + rate/sample rate, and multi-output grouping; a visually valid frame sequence + alone is not sufficient For performance data: diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/SKILL.md b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/SKILL.md index dd57167fa..94f82b00d 100644 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/SKILL.md +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/SKILL.md @@ -42,10 +42,10 @@ If any benchmark, perf-dump, or `torch.profiler` command prints one of those sig ## Main Reference -- [benchmark-and-profile.md](benchmark-and-profile.md) — canonical denoise benchmark, perf dump, and `torch.profiler` workflow; uses checked-in nightly-aligned presets plus current-source extras such as FLUX.2 Klein, Cosmos3, Ideogram4, ERNIE/GLM/SANA image models, FastWan2.2, `LTX-2.3` one-stage/two-stage/HQ, HunyuanVideo, MOVA, Helios, JoyAI/FireRed image edit, and Hunyuan3D shape +- [benchmark-and-profile.md](benchmark-and-profile.md) — canonical denoise benchmark, perf dump, and `torch.profiler` workflow; uses checked-in nightly-aligned presets plus current-source extras such as MiniMax-H3 joint video/audio T2VA, FLUX.2 Klein, Cosmos3, Ideogram4, ERNIE/GLM/SANA image models, FastWan2.2, `LTX-2.3` one-stage/two-stage/HQ, HunyuanVideo, MOVA, Helios, JoyAI/FireRed image edit, and Hunyuan3D shape - [existing-fast-paths.md](existing-fast-paths.md) — map bottlenecks to existing fused kernels, packed QKV paths, fused `QK norm + RoPE`, distributed overlap patterns, and open optimization PRs before proposing new code - [scripts/diffusion_skill_env.py](scripts/diffusion_skill_env.py) — preflight helper: repo root discovery via `sglang.__file__`, write-access probe, benchmark/profile output directories, idle GPU selection -- [scripts/bench_diffusion_denoise.py](scripts/bench_diffusion_denoise.py) — end-to-end denoise benchmark preset runner via `sglang generate`; supports `--no-torch-compile`, validates nightly preset drift with `--validate-nightly-alignment`, and saves perf dumps by label for `compare_perf.py` +- [scripts/bench_diffusion_denoise.py](scripts/bench_diffusion_denoise.py) — end-to-end denoise benchmark preset runner via `sglang generate`; supports `--no-torch-compile`, forces the H3 preset to its eager consistency mode, validates nightly preset drift with `--validate-nightly-alignment`, and saves perf dumps by label for `compare_perf.py` ## Opportunity Discovery Rule @@ -56,6 +56,8 @@ Always rule out these existing families first: - LTX upsampler GroupNorm+SiLU - Z-Image bf16-native Triton RMSNorm scale/tanh-residual modulation - SANA packed self-attention Q/K/V and cross-attention K/V GEMMs +- MiniMax-H3 indexed modulation, fused QK norm + RoPE, packed Ulysses QKV, + USP relayout, and batched TP AdaLN collectives - fused diffusion `QK norm + RoPE` - LTX2 split RoPE - LTX2 residual-gate add @@ -72,6 +74,10 @@ default benchmark preset invocation unchanged. Either pass the checked-in benchmark helper its no-compile switch or run the equivalent manual command without `--enable-torch-compile`. +MiniMax-H3 is always an eager consistency case on current main. Use +`--model minimax-h3-t2va`; its preset writes the H3 request fields through a +generated config and suppresses the helper's global compile default. + For FLUX-family manual profiling runs with a quantized transformer override: - use `sglang generate` directly - pass the override as `--transformer-path ` diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/benchmark-and-profile.md b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/benchmark-and-profile.md index 791de59e9..c4274505c 100644 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/benchmark-and-profile.md +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/benchmark-and-profile.md @@ -37,11 +37,15 @@ python3 "$ENV_PY" check-write-access >/dev/null export HF_TOKEN= # required for gated repos such as black-forest-labs/FLUX.* export FLASHINFER_DISABLE_VERSION_CHECK=1 -export CUDA_VISIBLE_DEVICES=$(python3 "$ENV_PY" print-idle-gpus --count 1) +# Leave CUDA_VISIBLE_DEVICES unset to let the preset helper select the number +# of idle GPUs it requires. For manual runs, set --count to that command's +# exact --num-gpus value. ASSET_DIR=$(python3 "$ENV_PY" print-assets-dir --mkdir) BENCH_DIR=$(python3 "$ENV_PY" print-output-dir --kind benchmarks --mkdir) PROFILE_DIR=$(python3 "$ENV_PY" print-output-dir --kind profiles --mkdir) +CONFIG_DIR="${BENCH_DIR}/generated_configs" +mkdir -p "${CONFIG_DIR}" export PROFILE_DIR check() { @@ -163,6 +167,17 @@ PYTHONPATH=python python3 "$BENCH_PY" \ --output-dir "${BENCH_DIR}" ``` +Run the current-source MiniMax-H3 T2VA preset. The helper forces eager mode +for this model even when its global compile default is enabled: + +```bash +export CUDA_VISIBLE_DEVICES=$(python3 "$ENV_PY" print-idle-gpus --count 4) +PYTHONPATH=python python3 "$BENCH_PY" \ + --model minimax-h3-t2va \ + --label baseline \ + --output-dir "${BENCH_DIR}" +``` + Run the full preset sweep only when you have enough GPU time for both the nightly-aligned cases and the source-tracked extras: @@ -202,6 +217,7 @@ Use the preset categories this way: | `ideogram4-fp8` | `ideogram-ai/ideogram-4-fp8` | Yes: `ideogram4_fp8_t2i_2gpu` | Prompt, 1024x1024, seed 42, 2 GPUs, TP size 2, FlashAttention backend; sampling preset owns steps/guidance | | `cosmos3-super-t2v` | `nvidia/Cosmos3-Super` | Yes: `cosmos3_super_t2v_2gpu` | Prompt, 1280x720, 81 frames, seed 42, 2 GPUs, TP size 2, guardrails disabled for benchmark isolation | | `wan-i2v` | `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | Yes: `wan22_i2v_a14b_720p` | Nightly cat image and motion prompt, 1280x720, 81 frames, 4 GPUs, CFG parallel, Ulysses degree 2, text encoder CPU offload and pinned CPU memory | +| `minimax-h3-t2va` | `MiniMaxAI/MiniMax-H3` | No | Current-source H3 FL2VA-partition T2VA baseline: 1344x768 resolved canvas, 5 seconds / 124 frames at 24 fps, 50 joint video-audio steps, 4 GPUs, TP2 + Ulysses2, eager BF16/FP32. The helper writes H3's `task`, `conditions`, `target`, and audio/video flow shifts to a generated config. | | `ltx2` | `Lightricks/LTX-2` | No | Current-source two-stage LTX-2 preset with 2 GPUs, CFG parallel, 768x512, 121 frames | | `qwen-image` | `Qwen/Qwen-Image` | No | Current-source extra covering the base Qwen-Image native path, separate from the nightly `Qwen-Image-2512` case | | `qwen-edit-2509` | `Qwen/Qwen-Image-Edit-2509` | No | Current-source extra for the pre-2511 edit-plus path; uses the cat image, 1024x1024 | @@ -232,6 +248,92 @@ and **best latency tuning**: - do not assume that is the fastest topology - for pure latency tuning, benchmark pure Ulysses too, for example `--ulysses-degree=4 --ring-degree=1` on 4 GPUs, and on 8 GPUs compare pure `--ulysses-degree=8` against `--enable-cfg-parallel --ulysses-degree=4` +For MiniMax-H3, keep the native contract intact: + +- use the root model ID and select `fl2va` or `ref2va` with + `--model-variant`; do not point at a checkpoint subdirectory +- use eager BF16/FP32 for consistency ground truth; current H3 + `torch.compile` changes numerical output +- use Ulysses, not Ring, for H3's packed multi-segment attention; CFG parallel + is invalid because the released pipeline has one denoising branch +- keep the released overlapping tiled video-VAE decode. H3 rejects + `spatial`, `spatial_shard`, and patch decode modes after output mismatches + +### Manual command example: MiniMax-H3 T2VA + +Create `${CONFIG_DIR}/minimax-h3-t2va.json` with the model-specific request +fields below. The generic width, height, and frame flags are intentionally +absent because H3 resolves all three from `target`: + +```json +{ + "task": "t2va", + "conditions": [], + "target": { + "short_edge": 768, + "aspect_ratio": "16:9", + "duration_seconds": 5.0 + }, + "num_inference_steps": 50, + "flow_shift": 12.0, + "audio_flow_shift": 3.0 +} +``` + +Then run the same lossless 4-GPU H100 topology and 5-second shape used by the +source-tracked preset: + +```bash +sglang generate \ + --backend=sglang \ + --model-path=MiniMaxAI/MiniMax-H3 \ + --model-variant=fl2va \ + --config="${CONFIG_DIR}/minimax-h3-t2va.json" \ + --prompt="At night, while their owner sleeps in a bedroom, three cats march in loudly playing tiny brass instruments, then abruptly file out." \ + --seed=1101 --num-gpus=4 --tp-size=2 --ulysses-degree=2 \ + --performance-mode=speed --enable-torch-compile=false \ + --save-output --warmup \ + --perf-dump-path="${BENCH_DIR}/minimax-h3-t2va-baseline.json" +``` + +The benchmark helper creates this config automatically. For ModelScope, set +`SGLANG_USE_MODELSCOPE=true`, replace the root model ID with +`MiniMax/MiniMax-H3`, and keep the selected variant unchanged. +When `--output-dir` is provided, the helper places the generated config under +that directory's `generated_configs/` subdirectory so the run is self-contained. + +For a serving benchmark, use the driver maintained by the H3 cookbook after +launching the corresponding `sglang serve` command: + +```bash +python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ + --host 127.0.0.1 --port 30010 \ + --model MiniMaxAI/MiniMax-H3 \ + --dataset vbench --task text-to-video \ + --num-prompts 1 --max-concurrency 1 \ + --warmup-requests 1 --warmup-inference-steps 50 \ + --extra-body '{"task":"t2va","conditions":[],"target":{"short_edge":768,"aspect_ratio":"16:9","duration_seconds":5.0},"seconds":5,"flow_shift":12.0,"audio_flow_shift":3.0}' +``` + +H3 correctness is joint video/audio correctness. Use eager BF16/FP32 as the +only ground truth and keep prompt, seed, target, step count, shifts, partition, +and topology fixed. For a lossless kernel/runtime change: + +- compare decoded frames after frame-count and timestamp alignment; report at + least frame-wise PSNR/SSIM plus the worst frame, not only an average +- extract the 32 kHz stereo audio stream and compare channel order, sample + count, waveform error, and a time-aligned log-mel or spectral metric +- verify the MP4 contract remains H.264 video at 24 fps plus one AAC stereo + audio stream +- run the relevant kernel/unit exactness test when replacing an existing H3 + BF16 fast path. Do not hide a failed exact test behind a permissive + end-to-end perceptual threshold + +There is no source-wide universal perceptual threshold for arbitrary H3 +changes. Record the acceptance bounds before optimization and tighten them for +changes that claim to preserve eager math. Approximate Cache-DiT or FP8 runs +must be labeled separately and validated for both output modalities. + ### Manual command example: LTX-2 Two-Stage ```bash @@ -417,6 +519,18 @@ Keep model shape, seed, and GPU topology fixed for every comparison. Save one reference image or video before changing code. If the active task requires `torch.compile` off, add `--no-torch-compile` here too. +MiniMax-H3 always requires eager mode for consistency ground truth. The +`minimax-h3-t2va` helper preset enforces it, and manual H3 profile commands +must pass `--enable-torch-compile=false`. + +For H3, one `--profile-all-stages` trace separates text/condition encoding, +`MiniMaxH3DenoisingStage`, and the aggregate `MiniMaxH3DecodingStage`. The +decoding stage contains both video decode and rank-0 audio decode. If decoding +is hot, add temporary `record_function` or NVTX scopes around +`video_vae.decode_base` and `_decode_audio` in the H3 decoding stage, then +re-run the same all-stage profile. Do not attribute aggregate decoding time to +one VAE without those inner scopes. + ### 2. Capture a representative trace By default SGLang profiles the denoising stage. The default sampling window is @@ -493,6 +607,8 @@ the known mainline families. | LTX-2 split RoPE appears as a long PyTorch elementwise chain | Check the `apply_ltx2_split_rotary_emb` Triton path and its shape guards | | masked attention spends time packing/unpacking Q/K/V | Check whether fused varlen USP pack/scatter should have engaged | | `all_to_all`, ring attention, or async A2A dominate | Classify against Ulysses, USP, or turbo-layer overlap first | +| H3 shows separate indexed gather + scale/shift, QK norm + RoPE, or three Q/K/V Ulysses relayouts | Check H3's indexed-modulation, fused QK-norm+RoPE, packed Ulysses-QKV, and USP relayout guards before writing a new kernel | +| H3 TP traces show one AdaLN collective per block | Check the batched TP AdaLN projection/all-gather path in `minimax_h3.py` before attempting communication overlap | | split `fc1 -> gelu -> quant -> fc2.lora_down` on Nunchaku FLUX | Treat as a missing fused GELU MLP path | | attention kernels dominate | Confirm backend, topology, and shape guards before proposing a new kernel | diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/existing-fast-paths.md b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/existing-fast-paths.md index 7812e16c9..dd0b7bb98 100644 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/existing-fast-paths.md +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/existing-fast-paths.md @@ -18,6 +18,11 @@ framework-specific optimization workflow. - `python/sglang/kernels/ops/diffusion/triton/zimage_native_norm.py` - `python/sglang/kernels/ops/diffusion/triton/rotary.py` - `python/sglang/kernels/ops/diffusion/triton/ltx2_rotary.py` +- `python/sglang/kernels/ops/diffusion/triton/indexed_modulation.py` +- `python/sglang/kernels/ops/diffusion/triton/ulysses_qkv.py` +- `python/sglang/kernels/ops/diffusion/usp_relayout.py` +- `python/sglang/multimodal_gen/runtime/layers/usp.py` +- `python/sglang/multimodal_gen/runtime/models/dits/minimax_h3.py` - `python/sglang/kernels/ops/diffusion/residual_gate_add.py` - `python/sglang/kernels/jit/csrc/diffusion/residual_gate_add.cuh` - `python/sglang/kernels/ops/diffusion/triton/varlen_pack_pad.py` @@ -106,7 +111,23 @@ framework-specific optimization workflow. - Microbench: `test/registered/kernels/benchmark/diffusion/bench_residual_gate_add.py`. - Workflow rule: if LTX2 traces show repeated elementwise `mul` + `add` ladders around attention or MLP residuals, check whether this existing CUDA path was disabled by shape, dtype, contiguity, or a prior runtime failure before proposing another elementwise fusion. -9. HunyuanVideo / LTX upsampler GroupNorm + SiLU fusion +9. MiniMax-H3 indexed AdaLN modulation and gated residual fusion +- Kernels: `indexed_scale_shift_bf16_`, `indexed_gate_bf16_` +- Locations: `triton/indexed_modulation.py`, `runtime/models/dits/minimax_h3.py` +- Use cases: H3's packed video/audio/text rows select per-token modulation with `combined_indices`; the Triton paths replace `index_select` plus scale/shift or gated residual chains in place. +- Constraints: CUDA BF16 H3 tensors, BF16 modulation tensors, contiguous disposable inputs; the gated path also requires contiguous `other`. Unsupported shapes/dtypes retain the eager formula. +- Numerical contract: the kernels explicitly reproduce H3's eager BF16 rounding boundaries. Do not replace them with a mathematically equivalent contraction without the H3 consistency check. +- Workflow rule: if H3 traces show `index_select` plus elementwise ladders around every block, check dtype, contiguity, and input-reuse eligibility before designing another modulation kernel. + +10. MiniMax-H3 packed Ulysses QKV and output relayout +- Kernels: `pack_qkv_destination_major`, `usp_merge_heads` +- Locations: `triton/ulysses_qkv.py`, `usp_relayout.py`, `runtime/layers/usp.py`, `runtime/models/dits/minimax_h3.py` +- Use cases: one destination-major QKV pack plus one collective replaces three separately prepared Ulysses input exchanges; the output JIT kernel replaces `permute(...).contiguous()` when merging gathered heads. +- Constraints: packed QKV fast packing requires CUDA fp16/bf16 Q/K/V with matching dtypes, contiguous head dimension, and eager execution. `usp_merge_heads` requires a nonempty contiguous 5D CUDA fp16/bf16/fp32 tensor and is disabled inside `torch.compile`. +- Related transport: 2-rank, peer-accessible CUDA groups can use the existing IPC A2A transport; larger or unsupported groups fall back to the normal collective path. +- Workflow rule: if an H3 Ulysses trace has three Q/K/V preparation ladders or a large output `permute + contiguous`, first prove why these existing guards missed. + +11. HunyuanVideo / LTX upsampler GroupNorm + SiLU fusion - Kernel: `triton_group_norm_silu` - Locations: `diffusion/group_norm_silu.py`, `triton/group_norm_silu.py`, `runtime/models/vaes/hunyuanvae.py`, `runtime/models/upsampler/latent_upsampler.py` - Use case: `activation(group_norm(x))` when the activation is non-inplace `nn.SiLU` and the GroupNorm is affine. @@ -166,6 +187,11 @@ framework-specific optimization workflow. - Supported head dims: `64, 128, 256`. - Behavior: `apply_qk_norm_rope` prefers the fused JIT kernel when all guards pass; otherwise it falls back to `apply_qk_norm(...)` plus `apply_flashinfer_rope_qk_inplace(...)`. - Validation: `test/registered/kernels/ops/diffusion/test_qknorm_rope.py`. +- MiniMax-H3: the H3 DiT calls `fused_inplace_qknorm_rope` directly for BF16 + head dim 128 with 96 rotary dims, NeoX layout, and + `round_norm_before_rope=True`. This flag is part of H3's eager numerical + contract. Compiled execution deliberately falls back to separate eager + operations. - Workflow rule: treat LTX2 traces that miss the generic fused path as an enablement/shape-guard issue first, and check the separate LTX2 split-RoPE path before proposing new attention-prep kernels. **Nunchaku Fused GELU MLP** @@ -197,6 +223,8 @@ framework-specific optimization workflow. `zimage_rmsnorm_tanh_mul_add` in `zimage.py`, backed by `triton/zimage_native_norm.py`. - HunyuanVideo VAE and LTX upsampler GroupNorm+SiLU: `apply_group_norm_silu` in `hunyuanvae.py` and `latent_upsampler.py`; default-eligible when wrapper guards pass. +- MiniMax-H3 indexed modulation: `_modulate_scale_shift` and `_modulate_gate` in `minimax_h3.py`, backed by `triton/indexed_modulation.py`. +- MiniMax-H3 Ulysses relayout: `_usp_input_all_to_all_packed_qkv` and `usp_merge_heads` through `runtime/layers/usp.py`. - QK norm: `apply_qk_norm` used in `flux.py`, `flux_2.py`, `qwen_image.py`, `zimage.py`, `wanvideo.py`, `ltx_2.py`, `hunyuanvideo.py`. - QK norm + RoPE: `apply_qk_norm_rope` in `layernorm.py`; use this path when the model wants fused attention prep instead of separate QK norm and RoPE calls. - LTX2 split RoPE: `apply_ltx2_split_rotary_emb` in `ltx_2.py`. @@ -210,6 +238,8 @@ framework-specific optimization workflow. **Existing Overlap / Communication Families** - Ulysses / USP attention: treat `all_to_all`, `ring_attn`, and head / sequence reshards as an existing distributed attention family, not a new overlap idea. +- MiniMax-H3 TP AdaLN: the DiT stacks every block's TP-local AdaLN projection and performs one batched all-gather before the block loop when `_can_batch_block_adaln()` passes. One all-gather per block indicates that this existing batching path missed. +- MiniMax-H3 final projections: H3 removes dead text/padding rows before the final TP column gathers and combines video/audio for the SP row gather. Preserve that ordering when optimizing output communication. - Turbo-layer async all-to-all: `all_to_all_single(..., async_op=True)` plus staged waits already form an existing overlap family in `turbo_layer.py`. - TorchInductor compute / communication reorder: `torch._inductor.config.reorder_for_compute_comm_overlap = True` can already partially overlap compiled denoise traces. - Dual-stream diffusion models: `use_dual_stream = True` in models such as `hunyuan3d.py` is an existing overlap family. diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py index 0b2bb1223..b015d0014 100755 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py @@ -249,6 +249,38 @@ MODELS = { ], }, # Source-tracked extras from current registry / GPU test coverage. + # MiniMax-H3 owns its temporal canvas through target.duration_seconds, so + # the model-specific sampling fields are passed through --config instead + # of generic --width/--height/--num-frames flags. + "minimax-h3-t2va": { + "path": "MiniMaxAI/MiniMax-H3", + "prompt": "At night, while their owner sleeps in a bedroom, three cats march in loudly playing tiny brass instruments, then abruptly file out.", + "seed": 1101, + "config_overrides": { + "task": "t2va", + "conditions": [], + "target": { + "short_edge": 768, + "aspect_ratio": "16:9", + "duration_seconds": 5.0, + }, + "audio_flow_shift": 3.0, + "flow_shift": 12.0, + "num_inference_steps": 50, + }, + "extra_args": [ + "--model-variant=fl2va", + "--num-gpus=4", + "--tp-size=2", + "--ulysses-degree=2", + "--performance-mode=speed", + "--enable-torch-compile=false", + ], + # H3 eager BF16/FP32 is the consistency ground truth. Current + # torch.compile changes numerical output, so never add the global + # helper default --enable-torch-compile flag for this preset. + "force_eager": True, + }, "ltx2": { "path": "Lightricks/LTX-2", "prompt": "A cat and a dog baking a cake together in a kitchen.", @@ -730,6 +762,7 @@ def build_sglang_cmd( torch_compile: bool = True, seed: int = 42, save_output: bool = True, + artifact_dir: Optional[Path] = None, ) -> list[str]: """ Build the `sglang generate` command for the given model. @@ -756,9 +789,12 @@ def build_sglang_cmd( cmd.append(f"--image-path={cfg['image_path']}") if "config_overrides" in cfg: - config_dir = ensure_dir( - get_output_dir("benchmarks", REPO_ROOT) / "generated_configs" + config_root = ( + Path(artifact_dir) + if artifact_dir is not None + else get_output_dir("benchmarks", REPO_ROOT) ) + config_dir = ensure_dir(config_root / "generated_configs") config_path = config_dir / f"{model_key}.json" with open(config_path, "w") as f: json.dump(cfg["config_overrides"], f, indent=2, sort_keys=True) @@ -770,7 +806,7 @@ def build_sglang_cmd( cmd.append("--save-output") if warmup: cmd.append("--warmup") - if torch_compile: + if torch_compile and not cfg.get("force_eager", False): cmd.append("--enable-torch-compile") if perf_dump_path: cmd.extend(["--perf-dump-path", perf_dump_path]) @@ -793,6 +829,7 @@ def run_benchmark_once( perf_dump_path=str(perf_path), warmup=warmup, torch_compile=torch_compile, + artifact_dir=output_dir, ) env = os.environ.copy() diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-modelopt-quant/SKILL.md b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-modelopt-quant/SKILL.md index 3682562ce..5b99d73fa 100644 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-modelopt-quant/SKILL.md +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-modelopt-quant/SKILL.md @@ -82,6 +82,22 @@ checkpoint repos; the FLUX.2 NVFP4 raw export remains `black-forest-labs/FLUX.2-dev-NVFP4`. Do not use older `BBuf/*` examples unless you are explicitly testing a historical branch. +### MiniMax-H3 boundary + +MiniMax-H3 is current-main evidence for the separate online FP8 path, not a +validated ModelOpt PTQ/export family. Its verified B200/B300 serving recipe +loads the unquantized root checkpoint with `--quantization fp8` and preserves +the video/audio patch projections, timestep MLP, and final video/audio heads in +FP32. Do not add H3 to the ModelOpt support matrix or run the generic ModelOpt +converter until an exact H3 export, loader mapping, accuracy check, and +benchmark scope have been validated. + +If the user asks for current H3 online quantization, route the command and +quality caveats through `sglang-diffusion-performance` and the MiniMax-H3 +cookbook. Online FP8 is approximate and must be compared against eager +BF16/FP32 for both video and audio; combining it with Cache-DiT compounds two +approximations. + ## Related PR Watchlist These related SGLang PRs are useful as ModelOpt diffusion support history. diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-performance/SKILL.md b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-performance/SKILL.md index a9cf5f448..fe1869f42 100644 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-performance/SKILL.md +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-performance/SKILL.md @@ -64,6 +64,64 @@ These options **trade output quality** for speed or VRAM savings. Results will d ## Quick Recipes +### MiniMax-H3 first: lossless joint video/audio + +H3 has a stricter contract than the generic recipes below. Keep its DiT eager +for consistency ground truth, use Ulysses rather than Ring, do not enable CFG +parallel, and leave the released overlapping tiled video-VAE decode in place. + +Four H200 GPUs can keep the complete BF16/FP32 pipeline resident: + +```bash +sglang serve \ + --model-path MiniMaxAI/MiniMax-H3 \ + --model-variant fl2va \ + --num-gpus 4 \ + --ulysses-degree 4 \ + --performance-mode speed \ + --enable-torch-compile false \ + --port 30010 +``` + +On 4x H100 80 GB, start from the fastest measured lossless resident topology: + +```bash +sglang serve \ + --model-path MiniMaxAI/MiniMax-H3 \ + --model-variant fl2va \ + --num-gpus 4 \ + --tp-size 2 \ + --ulysses-degree 2 \ + --performance-mode speed \ + --enable-torch-compile false \ + --port 30010 +``` + +On B200/B300, the verified resident sweep uses 8 GPUs with Ulysses8. H3 also +has a verified 4x B200 FSDP-capacity path, but FSDP all-gathers are a memory +policy rather than the default latency choice. Benchmark the target topology +with the H3 driver from `sglang-diffusion-benchmark-profile`. + +Use the FL2VA partition for both `t2va` and `fl2va`; use +`--model-variant ref2va` for image/video/audio reference conditioning. The root +IDs are `MiniMaxAI/MiniMax-H3` on Hugging Face and `MiniMax/MiniMax-H3` on +ModelScope. Do not point `--model-path` at a partition subdirectory. + +Current H3 restrictions: + +- `torch.compile` is opt-in experimentation only because it changes numerical + output; it is not a lossless baseline +- Ring attention and CFG parallel are incompatible with the packed single + denoising branch +- SageAttention is rejected for the current packed multi-segment attention +- `--vae-config.parallel-decode-mode spatial`, `spatial_shard`, and patch VAE + decode are rejected after mismatches; use the default tiled recipe +- Breakable CUDA Graph is opt-in and signature-specific; the validated + 1344x768 Ref2VA capture uses `--bcg-text-buckets 5504`, but it did not show a + measured speedup +- the `quality=high|medium|low` Cache-DiT profiles and online FP8 are + approximate; keep them outside lossless comparisons + ### Maximum speed, video model, multi-GPU, lossless (Wan A14B, 8 GPUs) ```bash @@ -255,6 +313,7 @@ Use these as first commands to benchmark, not as universal winners. | Model family | First performance shape | Starting flags | Notes | |---|---|---|---| +| MiniMax-H3 | 1344x768 resolved canvas, 5 seconds / 124 frames at 24 fps, 50 joint video/audio steps | H200: `--num-gpus 4 --ulysses-degree 4 --performance-mode speed --enable-torch-compile false`; H100: TP2 + Ulysses2 | Root ID plus `--model-variant fl2va` for T2VA/FL2VA or `ref2va` for Ref2VA. Ulysses only; no Ring/CFG/SageAttention. Preserve tiled video-VAE decode. Profile joint denoise, video VAE, audio VAE/vocoder, encoder, and collectives separately. | | FLUX.1 / FLUX.2 image | 1024x1024, runtime-default steps/guidance, 1 GPU | `--enable-torch-compile --warmup --dit-layerwise-offload false` | `black-forest-labs/FLUX.*` repos are gated; for FP8/NVFP4 use validated `--transformer-path` or `--transformer-weights-path` flows from the quant skill. | | FLUX.2 Klein / Klein Base | 1024x1024, runtime-default steps/guidance, 1 GPU | `--enable-torch-compile --warmup --dit-layerwise-offload false` | Current registry has `black-forest-labs/FLUX.2-klein-4B`, `FLUX.2-klein-9B`, and base variants. Klein is step-distilled; Klein Base is not. | | Qwen-Image / Qwen-Image-Edit | 1024x1024, runtime-default steps/guidance, 1 GPU | `--enable-torch-compile --warmup`; optionally native `SGLANG_CACHE_DIT_ENABLED=true` | Cache-DiT is lossy. For edit tasks, keep reference image, seed, and output size fixed. | @@ -289,6 +348,6 @@ about whether the work has merged: - **Offload tuning**: after the first request, the runtime logs peak GPU memory and which components could stay resident. Use this to decide which `--*-cpu-offload` flags to disable. - **Backend selection**: `--backend sglang` (default, auto-detected) enables native optimizations (fused kernels, SP, native Cache-DiT env knobs, etc.). `--backend diffusers` falls back to Diffusers pipelines and is the path that accepts `--cache-dit-config` plus diffusers attention backend names. - **Wan2.2-I2V sizing**: explicit `--width/--height` on `Wan2.2-I2V-A14B` control the target area while preserving the condition-image aspect ratio. -- **Mainline diffusion fast paths**: before proposing a new kernel or overlap scheme, check `sglang-diffusion-benchmark-profile/existing-fast-paths.md`. It covers GroupNorm+SiLU, Z-Image bf16-native Triton norm modulation, fused diffusion `QK norm + RoPE`, LTX2 split RoPE, LTX2 residual-gate add, varlen USP pack/scatter, packed QKV/NVFP4 expectations, and existing multi-GPU overlap families such as Ulysses / USP and turbo-layer async all-to-all. +- **Mainline diffusion fast paths**: before proposing a new kernel or overlap scheme, check `sglang-diffusion-benchmark-profile/existing-fast-paths.md`. It covers H3 indexed modulation, fused QK norm + RoPE, packed Ulysses QKV/USP relayout and batched TP AdaLN, plus GroupNorm+SiLU, Z-Image bf16-native Triton norm modulation, LTX2 split RoPE, LTX2 residual-gate add, varlen USP pack/scatter, packed QKV/NVFP4 expectations, and existing multi-GPU overlap families such as Ulysses / USP and turbo-layer async all-to-all. - **NVFP4 trace interpretation**: on FLUX.2 NVFP4 and Nunchaku-style checkpoints, packed QKV is expected. SGLang intentionally uses fused projection modules such as `to_qkv` / `to_added_qkv` instead of separate `to_q` / `to_k` / `to_v`, so a split-QKV trace usually means the quantized path did not engage rather than a brand new fusion opportunity. - **Hotspot workflow split**: use `sglang-diffusion-benchmark-profile` to prove and classify a slowdown with perf dumps plus `torch.profiler`; hand concrete kernel work off with the perf/profile evidence attached instead of expanding the benchmark skill.