minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[Diffusion] Align diffusion benchmark skill presets with nightly comparison cases (#21616)

Browse Source 
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
This commit is contained in:
Xiaoyu Zhang
2026-03-29 12:12:17 +08:00
committed by GitHub
co-authored by gemini-code-assist[bot]
parent 343a7ac652
commit 516cff97a3
2 changed files with 207 additions and 138 deletions
Download Patch File Download Diff File Expand all files Collapse all files
python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/benchmark-and-profile.md
+91 -53
View File
@@ -63,8 +63,6 @@ Download input images required by some models:
```bash
wget -O "${ASSET_DIR}/cat.png" \
https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png
wget -O "${ASSET_DIR}/astronaut.jpg" \
https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg
wget -O "${ASSET_DIR}/mova_single_person.jpg" \
https://github.com/OpenMOSS/MOVA/raw/main/assets/single_person.jpg
```
@@ -75,7 +73,27 @@ wget -O "${ASSET_DIR}/mova_single_person.jpg" \
All commands include `--warmup` and `--enable-torch-compile` for real production performance. Add `--perf-dump-path <file>.json` for machine-readable output.
If you want a checked-in preset runner instead of copying commands manually, use `scripts/bench_diffusion_denoise.py --model <preset> --label <name>`. It writes the same perf dump JSONs used by `compare_perf.py`.
Nightly diffusion comparison is server/API based (`sglang serve` + OpenAI-compatible requests). The commands below stay on `sglang generate` for local profiling, but the first 8 presets are aligned to nightly on model, prompt, reference image, steps, guidance scale, GPU count, and parallelism flags.
If you want a checked-in preset runner instead of copying commands manually, use `scripts/bench_diffusion_denoise.py --model <preset> --label <name>` or `--list-models`. It writes the same perf dump JSONs used by `compare_perf.py`.
### Preset Catalog
Nightly-aligned presets come first; skill-only presets stay available after them.
| Preset | Model | Nightly | Notes |
| --- | --- | --- | --- |
| `flux` | `black-forest-labs/FLUX.1-dev` | Yes: `flux1_dev_t2i_1024` | Aligned to nightly prompt + `--dit-layerwise-offload false` |
| `flux2` | `black-forest-labs/FLUX.2-dev` | Yes: `flux2_dev_t2i_1024` | Aligned to nightly prompt, 50 steps, guidance 4.0 |
| `qwen` | `Qwen/Qwen-Image-2512` | Yes: `qwen_image_2512_t2i_1024` | Aligned to nightly prompt/steps; no extra offload overrides |
| `qwen-edit` | `Qwen/Qwen-Image-Edit-2511` | Yes: `qwen_image_edit_2511` | Uses nightly cat image + edit prompt |
| `zimage` | `Tongyi-MAI/Z-Image-Turbo` | Yes: `zimage_turbo_t2i_1024` | Aligned to nightly prompt + guidance 4.0 |
| `wan-t2v` | `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | Yes: `wan22_t2v_a14b_720p` | Aligned to nightly CFG-parallel 4-GPU launch |
| `wan-ti2v` | `Wan-AI/Wan2.2-TI2V-5B-Diffusers` | Yes: `wan22_ti2v_5b_720p` | Uses nightly cat image + motion prompt |
| `wan-i2v` | `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | Yes: `wan22_i2v_a14b_720p` | Added to match nightly; aligned to CFG-parallel 4-GPU launch |
| `hunyuanvideo` | `hunyuanvideo-community/HunyuanVideo` | No | Skill-only extra preset |
| `mova-720p` | `OpenMOSS-Team/MOVA-720p` | No | Skill-only extra preset |
| `helios` | `BestWishYsh/Helios-Base` | No | Skill-only extra preset |
### Perf dump & before/after compare
@@ -98,86 +116,95 @@ python3 python/sglang/multimodal_gen/benchmarks/compare_perf.py \
"${BENCH_DIR}/baseline.json" "${BENCH_DIR}/new.json"
```
### Qwen-Image-2512 (1024×1024, 50 steps)
```bash
sglang generate \
--model-path=Qwen/Qwen-Image-2512 \
--prompt="A futuristic cyberpunk city at night, neon lights reflecting on wet streets, highly detailed, 8k" \
'--negative-prompt= ' \
--width=1024 --height=1024 --num-inference-steps=50 --guidance-scale=4.0 \
--seed=42 --save-output --enable-torch-compile --warmup \
--dit-cpu-offload false --text-encoder-cpu-offload false
```
### Qwen-Image-Edit-2511 (image editing, 1024×1024, 50 steps)
```bash
sglang generate \
--model-path=Qwen/Qwen-Image-Edit-2511 \
'--prompt=Transform into anime style' '--negative-prompt= ' \
--image-path="${ASSET_DIR}/cat.png" \
--width=1024 --height=1024 --num-inference-steps=50 --guidance-scale=4.0 \
--seed=42 --save-output --enable-torch-compile --warmup \
--dit-cpu-offload false --text-encoder-cpu-offload false
```
### FLUX.1-dev (1024×1024, 50 steps)
### FLUX.1-dev (Nightly: `flux1_dev_t2i_1024`)
```bash
sglang generate \
--model-path=black-forest-labs/FLUX.1-dev \
--prompt="A futuristic cyberpunk city at night, neon lights reflecting on wet streets, highly detailed, 8k" \
--prompt="A futuristic cyberpunk city at night, neon lights reflecting on wet streets" \
--width=1024 --height=1024 --num-inference-steps=50 --guidance-scale=4.0 \
--seed=42 --save-output --enable-torch-compile --warmup \
--dit-layerwise-offload false
```
### FLUX.2-dev (Nightly: `flux2_dev_t2i_1024`)
```bash
sglang generate \
--model-path=black-forest-labs/FLUX.2-dev \
--prompt="A futuristic cyberpunk city at night, neon lights reflecting on wet streets" \
--width=1024 --height=1024 --num-inference-steps=50 --guidance-scale=4.0 \
--seed=42 --save-output --enable-torch-compile --warmup \
--dit-layerwise-offload false
```
### Qwen-Image-2512 (Nightly: `qwen_image_2512_t2i_1024`)
```bash
sglang generate \
--model-path=Qwen/Qwen-Image-2512 \
--prompt="A futuristic cyberpunk city at night, neon lights reflecting on wet streets" \
--width=1024 --height=1024 --num-inference-steps=50 --guidance-scale=4.0 \
--seed=42 --save-output --enable-torch-compile --warmup
```
### FLUX.2-dev (1024×1024)
### Qwen-Image-Edit-2511 (Nightly: `qwen_image_edit_2511`)
```bash
sglang generate \
--model-path black-forest-labs/FLUX.2-dev \
--prompt "A Logo With Bold Large Text: SGL Diffusion" \
--width=1024 --height=1024 \
--dit-layerwise-offload false --enable-torch-compile --warmup \
--dit-cpu-offload false --text-encoder-cpu-offload true --vae-cpu-offload false
--model-path=Qwen/Qwen-Image-Edit-2511 \
--prompt="Make the cat wear a red hat" \
--image-path="${ASSET_DIR}/cat.png" \
--width=1024 --height=1024 --num-inference-steps=50 --guidance-scale=4.0 \
--seed=42 --save-output --enable-torch-compile --warmup
```
### Z-Image-Turbo (1024×1024, 9 steps)
### Z-Image-Turbo (Nightly: `zimage_turbo_t2i_1024`)
```bash
sglang generate \
--model-path=Tongyi-MAI/Z-Image-Turbo \
--prompt='A fantasy landscape with mountains and a river, detailed, vibrant colors' \
--width=1024 --height=1024 --num-inference-steps=9 --guidance-scale=0.0 \
--seed=42 --save-output --enable-torch-compile --warmup \
--dit-cpu-offload false --text-encoder-cpu-offload false
--prompt="A futuristic cyberpunk city at night, neon lights reflecting on wet streets" \
--width=1024 --height=1024 --num-inference-steps=9 --guidance-scale=4.0 \
--seed=42 --save-output --enable-torch-compile --warmup
```
### Wan2.2-T2V-A14B 720P (4 GPUs, 81 frames, 2 steps)
### Wan2.2-T2V-A14B 720P (Nightly: `wan22_t2v_a14b_720p`)
```bash
# Select four idle GPUs first:
# export CUDA_VISIBLE_DEVICES=$(python3 "$ENV_PY" print-idle-gpus --count 4)
sglang generate \
--model-path=Wan-AI/Wan2.2-T2V-A14B-Diffusers \
--prompt="A cat and a dog baking a cake together in a kitchen. The cat is carefully measuring flour, while the dog is stirring the batter with a wooden spoon." \
--negative-prompt=" " --720p --num-inference-steps=2 --num-frames=81 \
--prompt="A cat and a dog baking a cake together in a kitchen." \
--720p --num-inference-steps=2 --num-frames=81 \
--guidance-scale=5.0 --seed=42 --save-output \
--num-gpus=4 --ulysses-degree=4 \
--num-gpus=4 --enable-cfg-parallel --ulysses-degree=2 \
--text-encoder-cpu-offload --pin-cpu-memory \
--warmup --enable-torch-compile
```
### Wan2.2-TI2V-5B 720P (single GPU, 81 frames, 50 steps)
### Wan2.2-TI2V-5B 720P (Nightly: `wan22_ti2v_5b_720p`)
```bash
sglang generate \
--model-path Wan-AI/Wan2.2-TI2V-5B-Diffusers \
--prompt "An astronaut hatching from an egg, on the surface of the moon..." \
--negative-prompt "Bright tones, overexposed, static, blurred details..." \
--image-path="${ASSET_DIR}/astronaut.jpg" \
--num-frames 81 --720p --num-inference-steps 50 --guidance-scale 5.0 \
--seed 42 --save-output \
--dit-layerwise-offload false --dit-cpu-offload false \
--vae-cpu-offload false --text-encoder-cpu-offload false \
--model-path=Wan-AI/Wan2.2-TI2V-5B-Diffusers \
--prompt="The cat starts walking slowly towards the camera." \
--image-path="${ASSET_DIR}/cat.png" \
--num-frames=81 --720p --num-inference-steps=50 --guidance-scale=5.0 \
--seed=42 --save-output \
--enable-torch-compile --warmup
```
### HunyuanVideo (848×480, 65 frames, 30 steps)
### Wan2.2-I2V-A14B 720P (Nightly: `wan22_i2v_a14b_720p`)
```bash
# Select four idle GPUs first:
# export CUDA_VISIBLE_DEVICES=$(python3 "$ENV_PY" print-idle-gpus --count 4)
sglang generate \
--model-path=Wan-AI/Wan2.2-I2V-A14B-Diffusers \
--prompt="The cat starts walking slowly towards the camera." \
--image-path="${ASSET_DIR}/cat.png" \
--720p --num-inference-steps=2 --num-frames=81 \
--guidance-scale=5.0 --seed=42 --save-output \
--num-gpus=4 --enable-cfg-parallel --ulysses-degree=2 \
--text-encoder-cpu-offload --pin-cpu-memory \
--warmup --enable-torch-compile
```
### HunyuanVideo (Skill-only, not nightly)
```bash
sglang generate \
--model-path=hunyuanvideo-community/HunyuanVideo \
@@ -188,7 +215,7 @@ sglang generate \
--warmup --enable-torch-compile
```
### MOVA-720p (4 GPUs, 193 frames, 2 steps)
### MOVA-720p (Skill-only, not nightly)
```bash
# Select four idle GPUs first:
# export CUDA_VISIBLE_DEVICES=$(python3 "$ENV_PY" print-idle-gpus --count 4)
@@ -203,6 +230,17 @@ sglang generate \
--enable-torch-compile --save-output --warmup
```
### Helios-Base (Skill-only, not nightly)
```bash
sglang generate \
--model-path=BestWishYsh/Helios-Base \
--prompt="A curious raccoon" \
--width=640 --height=384 --num-frames=33 \
--dit-layerwise-offload false --dit-cpu-offload false \
--text-encoder-cpu-offload false --vae-cpu-offload false \
--seed=42 --save-output --enable-torch-compile --warmup
```
**Key metrics** (all models): denoise latency ★, end-to-end latency, peak GPU memory.
---
python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py
+116 -85
View File
@@ -12,9 +12,12 @@ Usage:
# Tag the run for later compare_perf.py usage
python3 python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py --model flux --label tuned
# All 10 preset models
# All 11 preset models
python3 python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py --all
# Show preset order, model path, and nightly mapping
python3 python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py --list-models
For gated Hugging Face repos such as FLUX, export HF_TOKEN first:
export HF_TOKEN=<your_hf_token>
@@ -22,8 +25,6 @@ Input images required for image-guided models:
ASSET_DIR=$(python3 python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/diffusion_skill_env.py print-assets-dir --mkdir)
wget -O "${ASSET_DIR}/cat.png" \
https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png
wget -O "${ASSET_DIR}/astronaut.jpg" \
https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg
wget -O "${ASSET_DIR}/mova_single_person.jpg" \
https://github.com/OpenMOSS/MOVA/raw/main/assets/single_person.jpg
"""
@@ -55,128 +56,127 @@ GATED_MODELS = {"flux", "flux2"}
# ---------------------------------------------------------------------------
# Model configs — kept in exact sync with benchmark-and-profile.md
# Nightly-aligned presets come first, followed by skill-only extras.
# Each entry produces the same `sglang generate` command as shown in that doc.
# ---------------------------------------------------------------------------
MODELS = {
# 1. Qwen/Qwen-Image-2512 — Text-to-Image, 1024×1024, 50 steps
"qwen": {
"path": "Qwen/Qwen-Image-2512",
"prompt": "A futuristic cyberpunk city at night, neon lights reflecting on wet streets, highly detailed, 8k",
"negative_prompt": " ",
# 1. Nightly: flux1_dev_t2i_1024
"flux": {
"nightly_case_id": "flux1_dev_t2i_1024",
"path": "black-forest-labs/FLUX.1-dev",
"prompt": "A futuristic cyberpunk city at night, neon lights reflecting on wet streets",
"extra_args": [
"--width=1024",
"--height=1024",
"--num-inference-steps=50",
"--guidance-scale=4.0",
"--dit-cpu-offload",
"false",
"--text-encoder-cpu-offload",
"--dit-layerwise-offload",
"false",
],
},
# 2. Qwen/Qwen-Image-Edit-2511 — Image Editing, 1024×1024, 50 steps
# 2. Nightly: flux2_dev_t2i_1024
"flux2": {
"nightly_case_id": "flux2_dev_t2i_1024",
"path": "black-forest-labs/FLUX.2-dev",
"prompt": "A futuristic cyberpunk city at night, neon lights reflecting on wet streets",
"extra_args": [
"--width=1024",
"--height=1024",
"--num-inference-steps=50",
"--guidance-scale=4.0",
"--dit-layerwise-offload",
"false",
],
},
# 3. Nightly: qwen_image_2512_t2i_1024
"qwen": {
"nightly_case_id": "qwen_image_2512_t2i_1024",
"path": "Qwen/Qwen-Image-2512",
"prompt": "A futuristic cyberpunk city at night, neon lights reflecting on wet streets",
"extra_args": [
"--width=1024",
"--height=1024",
"--num-inference-steps=50",
"--guidance-scale=4.0",
],
},
# 4. Nightly: qwen_image_edit_2511
# Requires: <repo>/inputs/diffusion_benchmark/figs/cat.png
"qwen-edit": {
"nightly_case_id": "qwen_image_edit_2511",
"path": "Qwen/Qwen-Image-Edit-2511",
"prompt": "Transform into anime style",
"negative_prompt": " ",
"prompt": "Make the cat wear a red hat",
"image_path": str(ASSET_DIR / "cat.png"),
"extra_args": [
"--width=1024",
"--height=1024",
"--num-inference-steps=50",
"--guidance-scale=4.0",
"--dit-cpu-offload",
"false",
"--text-encoder-cpu-offload",
"false",
],
},
# 3. black-forest-labs/FLUX.1-dev — Text-to-Image, 1024×1024, 50 steps
"flux": {
"path": "black-forest-labs/FLUX.1-dev",
"prompt": "A futuristic cyberpunk city at night, neon lights reflecting on wet streets, highly detailed, 8k",
"extra_args": [
"--width=1024",
"--height=1024",
"--num-inference-steps=50",
"--guidance-scale=4.0",
],
},
# 4. black-forest-labs/FLUX.2-dev — Text-to-Image, 1024×1024
"flux2": {
"path": "black-forest-labs/FLUX.2-dev",
"prompt": "A Logo With Bold Large Text: SGL Diffusion",
"extra_args": [
"--width=1024",
"--height=1024",
"--dit-layerwise-offload",
"false",
"--dit-cpu-offload",
"false",
"--text-encoder-cpu-offload",
"true",
"--vae-cpu-offload",
"false",
],
},
# 5. Tongyi-MAI/Z-Image-Turbo — Turbo Text-to-Image, 1024×1024, 9 steps
# 5. Nightly: zimage_turbo_t2i_1024
"zimage": {
"nightly_case_id": "zimage_turbo_t2i_1024",
"path": "Tongyi-MAI/Z-Image-Turbo",
"prompt": "A fantasy landscape with mountains and a river, detailed, vibrant colors",
"prompt": "A futuristic cyberpunk city at night, neon lights reflecting on wet streets",
"extra_args": [
"--width=1024",
"--height=1024",
"--num-inference-steps=9",
"--guidance-scale=0.0",
"--dit-cpu-offload",
"false",
"--text-encoder-cpu-offload",
"false",
"--guidance-scale=4.0",
],
},
# 6. Wan-AI/Wan2.2-T2V-A14B-Diffusers — Text-to-Video, 720P, 4 GPUs, 81 frames, 2 steps
# 6. Nightly: wan22_t2v_a14b_720p
"wan-t2v": {
"nightly_case_id": "wan22_t2v_a14b_720p",
"path": "Wan-AI/Wan2.2-T2V-A14B-Diffusers",
"prompt": "A cat and a dog baking a cake together in a kitchen. The cat is carefully measuring flour, while the dog is stirring the batter with a wooden spoon.",
"negative_prompt": " ",
"prompt": "A cat and a dog baking a cake together in a kitchen.",
"extra_args": [
"--720p",
"--num-inference-steps=2",
"--num-frames=81",
"--guidance-scale=5.0",
"--num-gpus=4",
"--ulysses-degree=4",
"--enable-cfg-parallel",
"--ulysses-degree=2",
"--text-encoder-cpu-offload",
"--pin-cpu-memory",
],
},
# 7. Wan-AI/Wan2.2-TI2V-5B-Diffusers — Text-Image-to-Video, 720P, 1 GPU, 81 frames, 50 steps
# Requires: <repo>/inputs/diffusion_benchmark/figs/astronaut.jpg
# 7. Nightly: wan22_ti2v_5b_720p
# Requires: <repo>/inputs/diffusion_benchmark/figs/cat.png
"wan-ti2v": {
"nightly_case_id": "wan22_ti2v_5b_720p",
"path": "Wan-AI/Wan2.2-TI2V-5B-Diffusers",
"prompt": "An astronaut hatching from an egg, on the surface of the moon, the darkness and depth of space realised in the background. High quality, ultrarealistic detail and breath-taking movie-like camera shot.",
"negative_prompt": "Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality, low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured, misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards",
"image_path": str(ASSET_DIR / "astronaut.jpg"),
"prompt": "The cat starts walking slowly towards the camera.",
"image_path": str(ASSET_DIR / "cat.png"),
"extra_args": [
"--num-frames",
"81",
"--720p",
"--num-inference-steps",
"50",
"--guidance-scale",
"5.0",
"--dit-layerwise-offload",
"false",
"--dit-cpu-offload",
"false",
"--vae-cpu-offload",
"false",
"--text-encoder-cpu-offload",
"false",
"--num-frames=81",
"--num-inference-steps=50",
"--guidance-scale=5.0",
],
},
# 8. hunyuanvideo-community/HunyuanVideo — Text-to-Video, 848×480, 65 frames, 30 steps
# 8. Nightly: wan22_i2v_a14b_720p
# Requires: <repo>/inputs/diffusion_benchmark/figs/cat.png
"wan-i2v": {
"nightly_case_id": "wan22_i2v_a14b_720p",
"path": "Wan-AI/Wan2.2-I2V-A14B-Diffusers",
"prompt": "The cat starts walking slowly towards the camera.",
"image_path": str(ASSET_DIR / "cat.png"),
"extra_args": [
"--720p",
"--num-inference-steps=2",
"--num-frames=81",
"--guidance-scale=5.0",
"--num-gpus=4",
"--enable-cfg-parallel",
"--ulysses-degree=2",
"--text-encoder-cpu-offload",
"--pin-cpu-memory",
],
},
# 9. Skill-only extra preset
"hunyuanvideo": {
"path": "hunyuanvideo-community/HunyuanVideo",
"prompt": "A cat and a dog baking a cake together in a kitchen. The cat is carefully measuring flour, while the dog is stirring the batter with a wooden spoon. The kitchen is cozy, with sunlight streaming through the window.",
@@ -189,7 +189,7 @@ MODELS = {
"--num-inference-steps=30",
],
},
# 9. OpenMOSS-Team/MOVA-720p — Image-to-Video, 4 GPUs, 193 frames, 2 steps
# 10. Skill-only extra preset
# Requires: <repo>/inputs/diffusion_benchmark/figs/mova_single_person.jpg
"mova-720p": {
"path": "OpenMOSS-Team/MOVA-720p",
@@ -205,7 +205,7 @@ MODELS = {
"--num-inference-steps=2",
],
},
# 10. BestWishYsh/Helios-Base — Text-to-Video, 640×384, 33 frames
# 11. Skill-only extra preset
"helios": {
"path": "BestWishYsh/Helios-Base",
"prompt": "A curious raccoon",
@@ -227,13 +227,35 @@ MODELS = {
def required_gpus_for_model(model_key: str) -> int:
if model_key == "wan-t2v":
if model_key in {"wan-t2v", "wan-i2v"}:
return 4
if model_key == "mova-720p":
return 4
return 1
def model_nightly_case_id(model_key: str) -> str:
return MODELS[model_key].get("nightly_case_id", "-")
def print_model_catalog():
"""Print preset order, model path, and whether each preset maps to nightly."""
print()
print("=" * 95)
print("MODEL PRESETS — Nightly-aligned first, skill-only extras after")
print("=" * 95)
print(f"{'Preset':<14} {'Nightly':<28} {'Model Path':<46} {'GPUs':>4}")
print("-" * 95)
for model_key, cfg in MODELS.items():
print(
f"{model_key:<14} {model_nightly_case_id(model_key):<28} {cfg['path']:<46} {required_gpus_for_model(model_key):>4}"
)
print("-" * 112)
print(
"Nightly column shows the comparison_configs.json case id; '-' means skill-only."
)
def build_sglang_cmd(
model_key: str,
perf_dump_path: Optional[str] = None,
@@ -393,9 +415,9 @@ def print_results_table(results: list[dict]):
print("=" * 80)
print(
f"{'Model':<16} {'Label':<12} {'Denoise(s)':>12} {'E2E(s)':>10} {'Peak Mem(GB)':>14}"
f"{'Model':<14} {'Nightly':<24} {'Label':<12} {'Denoise(s)':>12} {'E2E(s)':>10} {'Peak Mem(GB)':>14}"
)
print("-" * 64)
print("-" * 92)
for result in results:
denoise_s = result.get("denoise_latency_s")
@@ -405,10 +427,10 @@ def print_results_table(results: list[dict]):
e2e_text = f"{e2e_s:.2f}" if isinstance(e2e_s, float) else "n/a"
mem_text = f"{peak_mem:.1f}" if isinstance(peak_mem, float) else "n/a"
print(
f"{result['model']:<16} {result['label']:<12} {denoise_text:>12} {e2e_text:>10} {mem_text:>14}"
f"{result['model']:<14} {model_nightly_case_id(result['model']):<24} {result['label']:<12} {denoise_text:>12} {e2e_text:>10} {mem_text:>14}"
)
print("-" * 64)
print("-" * 92)
print()
print("★ Denoise latency = total DiT forward pass time across all inference steps.")
print(
@@ -425,7 +447,12 @@ def main():
choices=list(MODELS.keys()),
help="Model to benchmark (default: flux)",
)
parser.add_argument("--all", action="store_true", help="Benchmark all 10 models")
parser.add_argument("--all", action="store_true", help="Benchmark all 11 models")
parser.add_argument(
"--list-models",
action="store_true",
help="List preset order, nightly mapping, and exit",
)
parser.add_argument(
"--label",
type=str,
@@ -442,6 +469,10 @@ def main():
args = parser.parse_args()
if args.list_models:
print_model_catalog()
return
output_dir = Path(args.output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
warmup = not args.no_warmup