[SKILL] Add diffusion benchmark presets for edit and Hunyuan3D models (#24288)

Co-authored-by: BBuf Codex <bbuf-codex@users.noreply.github.com>
2026-05-05 08:18:12 +08:00
parent 4b487ca98b
commit 078f84d80d
4 changed files with 217 additions and 6 deletions
@@ -42,7 +42,7 @@ If any benchmark, perf-dump, or `torch.profiler` command prints one of those sig

 ## Main Reference

- [benchmark-and-profile.md](benchmark-and-profile.md) — canonical denoise benchmark, perf dump, and `torch.profiler` workflow; uses checked-in nightly-aligned presets plus skill-only stress recipes such as `LTX-2.3` one-stage/two-stage, HunyuanVideo, MOVA, and Helios
+- [benchmark-and-profile.md](benchmark-and-profile.md) — canonical denoise benchmark, perf dump, and `torch.profiler` workflow; uses checked-in nightly-aligned presets plus skill-only stress recipes such as `LTX-2.3` one-stage/two-stage, HunyuanVideo, MOVA, Helios, JoyAI/FireRed image edit, and Hunyuan3D shape
 - [existing-fast-paths.md](existing-fast-paths.md) — map bottlenecks to existing fused kernels, packed QKV paths, fused `QK norm + RoPE`, distributed overlap patterns, and open optimization PRs before proposing new code
 - [scripts/diffusion_skill_env.py](scripts/diffusion_skill_env.py) — preflight helper: repo root discovery via `sglang.__file__`, write-access probe, benchmark/profile output directories, idle GPU selection
 - [scripts/bench_diffusion_denoise.py](scripts/bench_diffusion_denoise.py) — end-to-end denoise benchmark preset runner via `sglang generate`; pins `--backend=sglang`, supports `--no-torch-compile`, and saves perf dumps by label for `compare_perf.py`
@@ -183,6 +183,10 @@ Nightly-aligned presets come first; skill-only presets stay available after them
 | `hunyuanvideo` | `hunyuanvideo-community/HunyuanVideo` | No | Skill-only extra preset |
 | `mova-720p` | `OpenMOSS-Team/MOVA-720p` | No | Skill-only extra preset |
 | `helios` | `BestWishYsh/Helios-Base` | No | Skill-only extra preset |
+| `joyai-edit` | `jdopensource/JoyAI-Image-Edit-Diffusers` | No | Skill-only JoyAI image-edit preset; uses the cat image, 1024x1024, 40 steps, guidance 4.0, 2-GPU CFG parallel |
+| `firered-edit-1.0` | `FireRedTeam/FireRed-Image-Edit-1.0` | No | Skill-only FireRed 1.0 image-edit preset; QwenImageEditPlus native path; uses 2-GPU CFG parallel |
+| `firered-edit-1.1` | `FireRedTeam/FireRed-Image-Edit-1.1` | No | Skill-only FireRed 1.1 image-edit preset; QwenImageEditPlus native path; uses 2-GPU CFG parallel |
+| `hunyuan3d-shape` | `tencent/Hunyuan3D-2` | No | Skill-only Hunyuan3D shape-generation preset; primary metric is `Hunyuan3DShapeDenoisingStage` |

 For Wan2.2 video models, remember the difference between **nightly alignment**
 and **best latency tuning**:
@@ -259,6 +263,65 @@ sglang generate \
 This matches the skill-only `ltx23-two-stage` preset. Use it as a
 high-resolution stress target, not as a nightly comparison case.

+### Manual command example: JoyAI Image Edit
+
+```bash
+sglang generate \
+  --backend=sglang \
+  --model-path=jdopensource/JoyAI-Image-Edit-Diffusers \
+  --prompt="Make the cat wear a red hat" \
+  --image-path="${ASSET_DIR}/cat.png" \
+  --width=1024 --height=1024 \
+  --num-inference-steps=40 --guidance-scale=4.0 \
+  --num-gpus=2 --enable-cfg-parallel --ulysses-degree=1 \
+  --dit-layerwise-offload false --dit-cpu-offload false \
+  --save-output --enable-torch-compile --warmup
+```
+
+### Manual command example: FireRed Image Edit
+
+```bash
+sglang generate \
+  --backend=sglang \
+  --model-path=FireRedTeam/FireRed-Image-Edit-1.1 \
+  --prompt="Make the cat wear a red hat" \
+  --image-path="${ASSET_DIR}/cat.png" \
+  --width=1024 --height=1024 \
+  --num-inference-steps=40 --guidance-scale=4.0 \
+  --num-gpus=2 --enable-cfg-parallel --ulysses-degree=1 \
+  --dit-layerwise-offload false --dit-cpu-offload false \
+  --save-output --enable-torch-compile --warmup
+```
+
+Use `FireRedTeam/FireRed-Image-Edit-1.0` in the same command when comparing the
+1.0 checkpoint. Both FireRed presets use the native `QwenImageEditPlusPipeline`
+path. On H100, 2-GPU CFG parallel reduced 40-step denoise latency versus the
+otherwise matching 2-GPU Ulysses command: FireRed 1.0 from 13419.15 ms to
+10955.90 ms, and FireRed 1.1 from 13414.72 ms to 10934.21 ms.
+
+### Manual command example: Hunyuan3D Shape
+
+```bash
+OUTPUT_DIR=$(python3 "$ENV_PY" print-output-dir --kind benchmarks --mkdir)
+CONFIG_DIR="${OUTPUT_DIR}/generated_configs"
+mkdir -p "${CONFIG_DIR}"
+printf '{"paint_enable": false}\n' > "${CONFIG_DIR}/hunyuan3d-shape.json"
+
+sglang generate \
+  --backend=sglang \
+  --model-path=tencent/Hunyuan3D-2 \
+  --prompt="generate 3d mesh" \
+  --image-path="${ASSET_DIR}/cat.png" \
+  --config="${CONFIG_DIR}/hunyuan3d-shape.json" \
+  --num-inference-steps=50 --guidance-scale=5.0 \
+  --dit-layerwise-offload false --dit-cpu-offload false \
+  --save-output --enable-torch-compile --warmup
+```
+
+For Hunyuan3D, compare the denoise stage separately from mesh export and paint
+stages. The benchmark helper reports `Hunyuan3DShapeDenoisingStage` as the
+primary denoise metric.
+
 ### Manual command example: Wan2.2-I2V-A14B 720P

 ```bash
@@ -12,7 +12,7 @@ Usage:
    # Tag the run for later compare_perf.py usage
    python3 python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py --model flux --label tuned

-    # All 15 preset models
+    # All 19 preset models
    python3 python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py --all

    # Show preset order, model path, and nightly mapping
@@ -42,7 +42,7 @@ SCRIPT_DIR = Path(__file__).resolve().parent
 if str(SCRIPT_DIR) not in sys.path:
    sys.path.insert(0, str(SCRIPT_DIR))

-from diffusion_skill_env import (
+from diffusion_skill_env import (  # noqa: E402
    ensure_dir,
    get_assets_dir,
    get_output_dir,
@@ -297,6 +297,84 @@ MODELS = {
            "false",
        ],
    },
+    # 16. Skill-only extra preset
+    # Requires: <repo>/inputs/diffusion_benchmark/figs/cat.png
+    "joyai-edit": {
+        "path": "jdopensource/JoyAI-Image-Edit-Diffusers",
+        "prompt": "Make the cat wear a red hat",
+        "image_path": str(ASSET_DIR / "cat.png"),
+        "extra_args": [
+            "--width=1024",
+            "--height=1024",
+            "--num-inference-steps=40",
+            "--guidance-scale=4.0",
+            "--dit-layerwise-offload",
+            "false",
+            "--dit-cpu-offload",
+            "false",
+            "--num-gpus=2",
+            "--enable-cfg-parallel",
+            "--ulysses-degree=1",
+        ],
+    },
+    # 17. Skill-only extra preset
+    # Requires: <repo>/inputs/diffusion_benchmark/figs/cat.png
+    "firered-edit-1.0": {
+        "path": "FireRedTeam/FireRed-Image-Edit-1.0",
+        "prompt": "Make the cat wear a red hat",
+        "image_path": str(ASSET_DIR / "cat.png"),
+        "extra_args": [
+            "--width=1024",
+            "--height=1024",
+            "--num-inference-steps=40",
+            "--guidance-scale=4.0",
+            "--dit-layerwise-offload",
+            "false",
+            "--dit-cpu-offload",
+            "false",
+            "--num-gpus=2",
+            "--enable-cfg-parallel",
+            "--ulysses-degree=1",
+        ],
+    },
+    # 18. Skill-only extra preset
+    # Requires: <repo>/inputs/diffusion_benchmark/figs/cat.png
+    "firered-edit-1.1": {
+        "path": "FireRedTeam/FireRed-Image-Edit-1.1",
+        "prompt": "Make the cat wear a red hat",
+        "image_path": str(ASSET_DIR / "cat.png"),
+        "extra_args": [
+            "--width=1024",
+            "--height=1024",
+            "--num-inference-steps=40",
+            "--guidance-scale=4.0",
+            "--dit-layerwise-offload",
+            "false",
+            "--dit-cpu-offload",
+            "false",
+            "--num-gpus=2",
+            "--enable-cfg-parallel",
+            "--ulysses-degree=1",
+        ],
+    },
+    # 19. Skill-only extra preset
+    # Requires: <repo>/inputs/diffusion_benchmark/figs/cat.png
+    "hunyuan3d-shape": {
+        "path": "tencent/Hunyuan3D-2",
+        "prompt": "generate 3d mesh",
+        "image_path": str(ASSET_DIR / "cat.png"),
+        "config_overrides": {
+            "paint_enable": False,
+        },
+        "extra_args": [
+            "--num-inference-steps=50",
+            "--guidance-scale=5.0",
+            "--dit-layerwise-offload",
+            "false",
+            "--dit-cpu-offload",
+            "false",
+        ],
+    },
 }


@@ -310,6 +388,9 @@ def required_gpus_for_model(model_key: str) -> int:
        "ltx23-ti2v-two-stage",
        "ltx23-one-stage",
        "ltx23-two-stage",
+        "joyai-edit",
+        "firered-edit-1.0",
+        "firered-edit-1.1",
    }:
        return 2
    return 1
@@ -370,6 +451,15 @@ def build_sglang_cmd(
    if "image_path" in cfg:
        cmd.append(f"--image-path={cfg['image_path']}")

+    if "config_overrides" in cfg:
+        config_dir = ensure_dir(
+            get_output_dir("benchmarks", REPO_ROOT) / "generated_configs"
+        )
+        config_path = config_dir / f"{model_key}.json"
+        with open(config_path, "w") as f:
+            json.dump(cfg["config_overrides"], f, indent=2, sort_keys=True)
+        cmd.append(f"--config={config_path}")
+
    cmd.extend(cfg["extra_args"])

    if save_output:
@@ -563,7 +653,7 @@ def main():
        choices=list(MODELS.keys()),
        help="Model to benchmark (default: flux)",
    )
-    parser.add_argument("--all", action="store_true", help="Benchmark all 15 models")
+    parser.add_argument("--all", action="store_true", help="Benchmark all 19 models")
    parser.add_argument(
        "--list-models",
        action="store_true",
@@ -148,10 +148,66 @@ Note: this is a high-resolution stress target for the native `LTX-2.3` two-stage
 sglang generate --model-path <IMAGE_MODEL> \
  --enable-torch-compile --warmup \
  --dit-layerwise-offload false \
+  --dit-cpu-offload false \
  --prompt "..." --save-output
 ```

-Note: for image models, per-layer compute is smaller, so layerwise offload may not fully hide H2D transfer. Disable it if VRAM allows.
+Note: for image models, per-layer compute is smaller, so layerwise offload may not fully hide H2D transfer. Disable DiT layerwise and CPU offload if VRAM allows; otherwise a large image DiT can stay resident on CPU and make the denoise loop H2D-bound.
+
+### Image-edit baselines: JoyAI and FireRed
+
+```bash
+sglang generate --backend=sglang \
+  --model-path jdopensource/JoyAI-Image-Edit-Diffusers \
+  --prompt "Make the cat wear a red hat" \
+  --image-path "${ASSET_DIR}/cat.png" \
+  --width 1024 --height 1024 \
+  --num-inference-steps 40 --guidance-scale 4.0 \
+  --num-gpus 2 --enable-cfg-parallel --ulysses-degree 1 \
+  --dit-layerwise-offload false --dit-cpu-offload false \
+  --enable-torch-compile --warmup --save-output
+```
+
+```bash
+sglang generate --backend=sglang \
+  --model-path FireRedTeam/FireRed-Image-Edit-1.1 \
+  --prompt "Make the cat wear a red hat" \
+  --image-path "${ASSET_DIR}/cat.png" \
+  --width 1024 --height 1024 \
+  --num-inference-steps 40 --guidance-scale 4.0 \
+  --num-gpus 2 --enable-cfg-parallel --ulysses-degree 1 \
+  --dit-layerwise-offload false --dit-cpu-offload false \
+  --enable-torch-compile --warmup --save-output
+```
+
+Use `FireRedTeam/FireRed-Image-Edit-1.0` in the same command when comparing
+FireRed 1.0. These are native image-edit paths; keep the reference image, prompt,
+seed, and output size fixed when comparing denoise numbers. On H100, 2-GPU CFG
+parallel was faster than the otherwise matching 2-GPU Ulysses command: FireRed
+1.0 improved from 13419.15 ms to 10955.90 ms, and FireRed 1.1 improved from
+13414.72 ms to 10934.21 ms.
+
+### Hunyuan3D shape baseline
+
+```bash
+OUTPUT_DIR=$(python3 "$ENV_PY" print-output-dir --kind benchmarks --mkdir)
+CONFIG_DIR="${OUTPUT_DIR}/generated_configs"
+mkdir -p "${CONFIG_DIR}"
+printf '{"paint_enable": false}\n' > "${CONFIG_DIR}/hunyuan3d-shape.json"
+
+sglang generate --backend=sglang \
+  --model-path tencent/Hunyuan3D-2 \
+  --prompt "generate 3d mesh" \
+  --image-path "${ASSET_DIR}/cat.png" \
+  --config "${CONFIG_DIR}/hunyuan3d-shape.json" \
+  --num-inference-steps 50 --guidance-scale 5.0 \
+  --dit-layerwise-offload false --dit-cpu-offload false \
+  --enable-torch-compile --warmup --save-output
+```
+
+For Hunyuan3D, treat `Hunyuan3DShapeDenoisingStage` as the primary latency
+metric. Mesh export and paint stages are useful end-to-end checks but should not
+drive DiT optimization decisions.

 ### Low VRAM, decent speed (single GPU)

@@ -202,7 +258,9 @@ Use these as first commands to benchmark, not as universal winners.
 | Wan2.2 TI2V 5B | 720p, 81 frames, 1 GPU | `--enable-torch-compile --warmup` | Keep the input image and motion prompt fixed when comparing sparse attention or Cache-DiT. |
 | LTX-2 / LTX-2.3 | 768x512, 121 frames, 2 GPUs | `--pipeline-class-name LTX2TwoStagePipeline --enable-torch-compile --warmup`; LTX-2 nightly also uses `--enable-cfg-parallel` | Use the benchmark/profile skill presets for exact nightly alignment. PRs #22441, #24025, and #23736 track additional LTX2 perf/parallel work. |
 | HunyuanVideo | 848x480 or 720p class video | `--text-encoder-cpu-offload --pin-cpu-memory --enable-torch-compile --warmup` | Check VAE decode separately. GroupNorm+SiLU is default-eligible in mainline when wrapper guards pass; use `bench_group_norm_silu.py` when VAE residual blocks are hot. |
-| JoyAI-Image-Edit | 1024-class TI2I, 40 steps, guidance 4.0 | Start from the CI/default TI2I path for `jdopensource/JoyAI-Image-Edit-Diffusers`; add `--warmup` and benchmark `--enable-torch-compile` separately | Newly supported image-edit path. Keep the input image, prompt, seed, and output size fixed; sequence shard is auto-enabled for Joy pipelines. |
+| JoyAI-Image-Edit | 1024-class TI2I, 40 steps, guidance 4.0 | `--backend=sglang --num-gpus 2 --enable-cfg-parallel --ulysses-degree 1 --enable-torch-compile --warmup --dit-layerwise-offload false --dit-cpu-offload false` | Newly supported image-edit path. Keep the input image, prompt, seed, and output size fixed; 2-GPU CFG parallel is the validated H100 starting point. |
+| FireRed-Image-Edit 1.0 / 1.1 | 1024x1024 image edit, 40 steps, guidance 4.0 | `--backend=sglang --num-gpus 2 --enable-cfg-parallel --ulysses-degree 1 --enable-torch-compile --warmup --dit-layerwise-offload false --dit-cpu-offload false` | Uses the native `QwenImageEditPlusPipeline` path. 2-GPU CFG parallel is the validated H100 starting point; benchmark 1.0 and 1.1 separately because checkpoint differences can change denoise latency. |
+| Hunyuan3D-2 shape | Shape generation, 50 steps, guidance 5.0 | `--backend=sglang --enable-torch-compile --warmup --dit-layerwise-offload false --dit-cpu-offload false` | Focus on `Hunyuan3DShapeDenoisingStage`; keep mesh export/paint timings separate from denoise. |
 | MOVA / Helios | Use the benchmark/profile presets first | `--enable-torch-compile --warmup`; pin offload flags explicitly | PR #20530 tracks MOVA fused RMSNorm+RoPE; PR #24059 tracks Helios fused norm modulation. |

 ## Open PR Watchlist