diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/SKILL.md b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/SKILL.md index e83b1a2fd..7a5e77813 100644 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/SKILL.md +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/SKILL.md @@ -42,7 +42,7 @@ If any benchmark, perf-dump, or `torch.profiler` command prints one of those sig ## Main Reference -- [benchmark-and-profile.md](benchmark-and-profile.md) — canonical denoise benchmark, perf dump, and `torch.profiler` workflow; uses checked-in nightly-aligned presets plus skill-only stress recipes such as `LTX-2.3` one-stage/two-stage, HunyuanVideo, MOVA, and Helios +- [benchmark-and-profile.md](benchmark-and-profile.md) — canonical denoise benchmark, perf dump, and `torch.profiler` workflow; uses checked-in nightly-aligned presets plus skill-only stress recipes such as `LTX-2.3` one-stage/two-stage, HunyuanVideo, MOVA, Helios, JoyAI/FireRed image edit, and Hunyuan3D shape - [existing-fast-paths.md](existing-fast-paths.md) — map bottlenecks to existing fused kernels, packed QKV paths, fused `QK norm + RoPE`, distributed overlap patterns, and open optimization PRs before proposing new code - [scripts/diffusion_skill_env.py](scripts/diffusion_skill_env.py) — preflight helper: repo root discovery via `sglang.__file__`, write-access probe, benchmark/profile output directories, idle GPU selection - [scripts/bench_diffusion_denoise.py](scripts/bench_diffusion_denoise.py) — end-to-end denoise benchmark preset runner via `sglang generate`; pins `--backend=sglang`, supports `--no-torch-compile`, and saves perf dumps by label for `compare_perf.py` diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/benchmark-and-profile.md b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/benchmark-and-profile.md index 9544901f0..20fc1ac35 100644 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/benchmark-and-profile.md +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/benchmark-and-profile.md @@ -183,6 +183,10 @@ Nightly-aligned presets come first; skill-only presets stay available after them | `hunyuanvideo` | `hunyuanvideo-community/HunyuanVideo` | No | Skill-only extra preset | | `mova-720p` | `OpenMOSS-Team/MOVA-720p` | No | Skill-only extra preset | | `helios` | `BestWishYsh/Helios-Base` | No | Skill-only extra preset | +| `joyai-edit` | `jdopensource/JoyAI-Image-Edit-Diffusers` | No | Skill-only JoyAI image-edit preset; uses the cat image, 1024x1024, 40 steps, guidance 4.0, 2-GPU CFG parallel | +| `firered-edit-1.0` | `FireRedTeam/FireRed-Image-Edit-1.0` | No | Skill-only FireRed 1.0 image-edit preset; QwenImageEditPlus native path; uses 2-GPU CFG parallel | +| `firered-edit-1.1` | `FireRedTeam/FireRed-Image-Edit-1.1` | No | Skill-only FireRed 1.1 image-edit preset; QwenImageEditPlus native path; uses 2-GPU CFG parallel | +| `hunyuan3d-shape` | `tencent/Hunyuan3D-2` | No | Skill-only Hunyuan3D shape-generation preset; primary metric is `Hunyuan3DShapeDenoisingStage` | For Wan2.2 video models, remember the difference between **nightly alignment** and **best latency tuning**: @@ -259,6 +263,65 @@ sglang generate \ This matches the skill-only `ltx23-two-stage` preset. Use it as a high-resolution stress target, not as a nightly comparison case. +### Manual command example: JoyAI Image Edit + +```bash +sglang generate \ + --backend=sglang \ + --model-path=jdopensource/JoyAI-Image-Edit-Diffusers \ + --prompt="Make the cat wear a red hat" \ + --image-path="${ASSET_DIR}/cat.png" \ + --width=1024 --height=1024 \ + --num-inference-steps=40 --guidance-scale=4.0 \ + --num-gpus=2 --enable-cfg-parallel --ulysses-degree=1 \ + --dit-layerwise-offload false --dit-cpu-offload false \ + --save-output --enable-torch-compile --warmup +``` + +### Manual command example: FireRed Image Edit + +```bash +sglang generate \ + --backend=sglang \ + --model-path=FireRedTeam/FireRed-Image-Edit-1.1 \ + --prompt="Make the cat wear a red hat" \ + --image-path="${ASSET_DIR}/cat.png" \ + --width=1024 --height=1024 \ + --num-inference-steps=40 --guidance-scale=4.0 \ + --num-gpus=2 --enable-cfg-parallel --ulysses-degree=1 \ + --dit-layerwise-offload false --dit-cpu-offload false \ + --save-output --enable-torch-compile --warmup +``` + +Use `FireRedTeam/FireRed-Image-Edit-1.0` in the same command when comparing the +1.0 checkpoint. Both FireRed presets use the native `QwenImageEditPlusPipeline` +path. On H100, 2-GPU CFG parallel reduced 40-step denoise latency versus the +otherwise matching 2-GPU Ulysses command: FireRed 1.0 from 13419.15 ms to +10955.90 ms, and FireRed 1.1 from 13414.72 ms to 10934.21 ms. + +### Manual command example: Hunyuan3D Shape + +```bash +OUTPUT_DIR=$(python3 "$ENV_PY" print-output-dir --kind benchmarks --mkdir) +CONFIG_DIR="${OUTPUT_DIR}/generated_configs" +mkdir -p "${CONFIG_DIR}" +printf '{"paint_enable": false}\n' > "${CONFIG_DIR}/hunyuan3d-shape.json" + +sglang generate \ + --backend=sglang \ + --model-path=tencent/Hunyuan3D-2 \ + --prompt="generate 3d mesh" \ + --image-path="${ASSET_DIR}/cat.png" \ + --config="${CONFIG_DIR}/hunyuan3d-shape.json" \ + --num-inference-steps=50 --guidance-scale=5.0 \ + --dit-layerwise-offload false --dit-cpu-offload false \ + --save-output --enable-torch-compile --warmup +``` + +For Hunyuan3D, compare the denoise stage separately from mesh export and paint +stages. The benchmark helper reports `Hunyuan3DShapeDenoisingStage` as the +primary denoise metric. + ### Manual command example: Wan2.2-I2V-A14B 720P ```bash diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py index e600187d6..2e93b2c8e 100755 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py @@ -12,7 +12,7 @@ Usage: # Tag the run for later compare_perf.py usage python3 python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py --model flux --label tuned - # All 15 preset models + # All 19 preset models python3 python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py --all # Show preset order, model path, and nightly mapping @@ -42,7 +42,7 @@ SCRIPT_DIR = Path(__file__).resolve().parent if str(SCRIPT_DIR) not in sys.path: sys.path.insert(0, str(SCRIPT_DIR)) -from diffusion_skill_env import ( +from diffusion_skill_env import ( # noqa: E402 ensure_dir, get_assets_dir, get_output_dir, @@ -297,6 +297,84 @@ MODELS = { "false", ], }, + # 16. Skill-only extra preset + # Requires: /inputs/diffusion_benchmark/figs/cat.png + "joyai-edit": { + "path": "jdopensource/JoyAI-Image-Edit-Diffusers", + "prompt": "Make the cat wear a red hat", + "image_path": str(ASSET_DIR / "cat.png"), + "extra_args": [ + "--width=1024", + "--height=1024", + "--num-inference-steps=40", + "--guidance-scale=4.0", + "--dit-layerwise-offload", + "false", + "--dit-cpu-offload", + "false", + "--num-gpus=2", + "--enable-cfg-parallel", + "--ulysses-degree=1", + ], + }, + # 17. Skill-only extra preset + # Requires: /inputs/diffusion_benchmark/figs/cat.png + "firered-edit-1.0": { + "path": "FireRedTeam/FireRed-Image-Edit-1.0", + "prompt": "Make the cat wear a red hat", + "image_path": str(ASSET_DIR / "cat.png"), + "extra_args": [ + "--width=1024", + "--height=1024", + "--num-inference-steps=40", + "--guidance-scale=4.0", + "--dit-layerwise-offload", + "false", + "--dit-cpu-offload", + "false", + "--num-gpus=2", + "--enable-cfg-parallel", + "--ulysses-degree=1", + ], + }, + # 18. Skill-only extra preset + # Requires: /inputs/diffusion_benchmark/figs/cat.png + "firered-edit-1.1": { + "path": "FireRedTeam/FireRed-Image-Edit-1.1", + "prompt": "Make the cat wear a red hat", + "image_path": str(ASSET_DIR / "cat.png"), + "extra_args": [ + "--width=1024", + "--height=1024", + "--num-inference-steps=40", + "--guidance-scale=4.0", + "--dit-layerwise-offload", + "false", + "--dit-cpu-offload", + "false", + "--num-gpus=2", + "--enable-cfg-parallel", + "--ulysses-degree=1", + ], + }, + # 19. Skill-only extra preset + # Requires: /inputs/diffusion_benchmark/figs/cat.png + "hunyuan3d-shape": { + "path": "tencent/Hunyuan3D-2", + "prompt": "generate 3d mesh", + "image_path": str(ASSET_DIR / "cat.png"), + "config_overrides": { + "paint_enable": False, + }, + "extra_args": [ + "--num-inference-steps=50", + "--guidance-scale=5.0", + "--dit-layerwise-offload", + "false", + "--dit-cpu-offload", + "false", + ], + }, } @@ -310,6 +388,9 @@ def required_gpus_for_model(model_key: str) -> int: "ltx23-ti2v-two-stage", "ltx23-one-stage", "ltx23-two-stage", + "joyai-edit", + "firered-edit-1.0", + "firered-edit-1.1", }: return 2 return 1 @@ -370,6 +451,15 @@ def build_sglang_cmd( if "image_path" in cfg: cmd.append(f"--image-path={cfg['image_path']}") + if "config_overrides" in cfg: + config_dir = ensure_dir( + get_output_dir("benchmarks", REPO_ROOT) / "generated_configs" + ) + config_path = config_dir / f"{model_key}.json" + with open(config_path, "w") as f: + json.dump(cfg["config_overrides"], f, indent=2, sort_keys=True) + cmd.append(f"--config={config_path}") + cmd.extend(cfg["extra_args"]) if save_output: @@ -563,7 +653,7 @@ def main(): choices=list(MODELS.keys()), help="Model to benchmark (default: flux)", ) - parser.add_argument("--all", action="store_true", help="Benchmark all 15 models") + parser.add_argument("--all", action="store_true", help="Benchmark all 19 models") parser.add_argument( "--list-models", action="store_true", diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-performance/SKILL.md b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-performance/SKILL.md index fd4934a4c..1bba482a4 100644 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-performance/SKILL.md +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-performance/SKILL.md @@ -148,10 +148,66 @@ Note: this is a high-resolution stress target for the native `LTX-2.3` two-stage sglang generate --model-path \ --enable-torch-compile --warmup \ --dit-layerwise-offload false \ + --dit-cpu-offload false \ --prompt "..." --save-output ``` -Note: for image models, per-layer compute is smaller, so layerwise offload may not fully hide H2D transfer. Disable it if VRAM allows. +Note: for image models, per-layer compute is smaller, so layerwise offload may not fully hide H2D transfer. Disable DiT layerwise and CPU offload if VRAM allows; otherwise a large image DiT can stay resident on CPU and make the denoise loop H2D-bound. + +### Image-edit baselines: JoyAI and FireRed + +```bash +sglang generate --backend=sglang \ + --model-path jdopensource/JoyAI-Image-Edit-Diffusers \ + --prompt "Make the cat wear a red hat" \ + --image-path "${ASSET_DIR}/cat.png" \ + --width 1024 --height 1024 \ + --num-inference-steps 40 --guidance-scale 4.0 \ + --num-gpus 2 --enable-cfg-parallel --ulysses-degree 1 \ + --dit-layerwise-offload false --dit-cpu-offload false \ + --enable-torch-compile --warmup --save-output +``` + +```bash +sglang generate --backend=sglang \ + --model-path FireRedTeam/FireRed-Image-Edit-1.1 \ + --prompt "Make the cat wear a red hat" \ + --image-path "${ASSET_DIR}/cat.png" \ + --width 1024 --height 1024 \ + --num-inference-steps 40 --guidance-scale 4.0 \ + --num-gpus 2 --enable-cfg-parallel --ulysses-degree 1 \ + --dit-layerwise-offload false --dit-cpu-offload false \ + --enable-torch-compile --warmup --save-output +``` + +Use `FireRedTeam/FireRed-Image-Edit-1.0` in the same command when comparing +FireRed 1.0. These are native image-edit paths; keep the reference image, prompt, +seed, and output size fixed when comparing denoise numbers. On H100, 2-GPU CFG +parallel was faster than the otherwise matching 2-GPU Ulysses command: FireRed +1.0 improved from 13419.15 ms to 10955.90 ms, and FireRed 1.1 improved from +13414.72 ms to 10934.21 ms. + +### Hunyuan3D shape baseline + +```bash +OUTPUT_DIR=$(python3 "$ENV_PY" print-output-dir --kind benchmarks --mkdir) +CONFIG_DIR="${OUTPUT_DIR}/generated_configs" +mkdir -p "${CONFIG_DIR}" +printf '{"paint_enable": false}\n' > "${CONFIG_DIR}/hunyuan3d-shape.json" + +sglang generate --backend=sglang \ + --model-path tencent/Hunyuan3D-2 \ + --prompt "generate 3d mesh" \ + --image-path "${ASSET_DIR}/cat.png" \ + --config "${CONFIG_DIR}/hunyuan3d-shape.json" \ + --num-inference-steps 50 --guidance-scale 5.0 \ + --dit-layerwise-offload false --dit-cpu-offload false \ + --enable-torch-compile --warmup --save-output +``` + +For Hunyuan3D, treat `Hunyuan3DShapeDenoisingStage` as the primary latency +metric. Mesh export and paint stages are useful end-to-end checks but should not +drive DiT optimization decisions. ### Low VRAM, decent speed (single GPU) @@ -202,7 +258,9 @@ Use these as first commands to benchmark, not as universal winners. | Wan2.2 TI2V 5B | 720p, 81 frames, 1 GPU | `--enable-torch-compile --warmup` | Keep the input image and motion prompt fixed when comparing sparse attention or Cache-DiT. | | LTX-2 / LTX-2.3 | 768x512, 121 frames, 2 GPUs | `--pipeline-class-name LTX2TwoStagePipeline --enable-torch-compile --warmup`; LTX-2 nightly also uses `--enable-cfg-parallel` | Use the benchmark/profile skill presets for exact nightly alignment. PRs #22441, #24025, and #23736 track additional LTX2 perf/parallel work. | | HunyuanVideo | 848x480 or 720p class video | `--text-encoder-cpu-offload --pin-cpu-memory --enable-torch-compile --warmup` | Check VAE decode separately. GroupNorm+SiLU is default-eligible in mainline when wrapper guards pass; use `bench_group_norm_silu.py` when VAE residual blocks are hot. | -| JoyAI-Image-Edit | 1024-class TI2I, 40 steps, guidance 4.0 | Start from the CI/default TI2I path for `jdopensource/JoyAI-Image-Edit-Diffusers`; add `--warmup` and benchmark `--enable-torch-compile` separately | Newly supported image-edit path. Keep the input image, prompt, seed, and output size fixed; sequence shard is auto-enabled for Joy pipelines. | +| JoyAI-Image-Edit | 1024-class TI2I, 40 steps, guidance 4.0 | `--backend=sglang --num-gpus 2 --enable-cfg-parallel --ulysses-degree 1 --enable-torch-compile --warmup --dit-layerwise-offload false --dit-cpu-offload false` | Newly supported image-edit path. Keep the input image, prompt, seed, and output size fixed; 2-GPU CFG parallel is the validated H100 starting point. | +| FireRed-Image-Edit 1.0 / 1.1 | 1024x1024 image edit, 40 steps, guidance 4.0 | `--backend=sglang --num-gpus 2 --enable-cfg-parallel --ulysses-degree 1 --enable-torch-compile --warmup --dit-layerwise-offload false --dit-cpu-offload false` | Uses the native `QwenImageEditPlusPipeline` path. 2-GPU CFG parallel is the validated H100 starting point; benchmark 1.0 and 1.1 separately because checkpoint differences can change denoise latency. | +| Hunyuan3D-2 shape | Shape generation, 50 steps, guidance 5.0 | `--backend=sglang --enable-torch-compile --warmup --dit-layerwise-offload false --dit-cpu-offload false` | Focus on `Hunyuan3DShapeDenoisingStage`; keep mesh export/paint timings separate from denoise. | | MOVA / Helios | Use the benchmark/profile presets first | `--enable-torch-compile --warmup`; pin offload flags explicitly | PR #20530 tracks MOVA fused RMSNorm+RoPE; PR #24059 tracks Helios fused norm modulation. | ## Open PR Watchlist