minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[diffusion] make benchmark caches seedable and cover missing native families (#36463)

Browse Source
This commit is contained in:
Xiaoyu Zhang
2026-08-26 19:04:46 +08:00
committed by GitHub
parent 170da72c13
commit 702de26310
4 changed files with 290 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/SKILL.md
+8 -1
View File
@@ -50,7 +50,7 @@ If any benchmark, perf-dump, or `torch.profiler` command prints one of those sig
## Main Reference
- [benchmark-and-profile.md](benchmark-and-profile.md) — canonical denoise benchmark, perf dump, and `torch.profiler` workflow; uses checked-in nightly-aligned presets plus current-source extras such as LongCat-Image, SANA-Video/SANA-WM, LingBot Video/World, Cosmos3 Edge/Super I2V/distilled and the explicit Super TP2 x CFG2 comparator, LTX-2.5 and its diffusion decoder, MiniMax-H3, FLUX.2 Klein, Ideogram4, ERNIE/GLM/SANA image models, FastWan2.1/2.2, the Blackwell-only Wan2.2 NVFP4 comparator, `LTX-2.3`, HunyuanVideo, MOVA, Helios, image edit, and Hunyuan3D shape
- [benchmark-and-profile.md](benchmark-and-profile.md) — canonical denoise benchmark, perf dump, and `torch.profiler` workflow; uses checked-in nightly-aligned presets plus current-source extras such as LongCat image/edit, Qwen base edit/layered, SD3.5, SANA-Video/SANA-WM, LingBot Video/World, Cosmos3 Edge/Super I2V/distilled and the explicit Super TP2 x CFG2 comparator, LTX-2.5 and its diffusion decoder, MiniMax-H3, FLUX.2 Klein, Ideogram4, ERNIE/GLM/SANA image models, FastWan2.1/2.2, the Blackwell-only Wan2.2 NVFP4 comparator, `LTX-2.3`, HunyuanVideo, MOVA, Helios, image edit, Hunyuan3D shape, and a separate Pi0.5 action-policy lane
- [existing-fast-paths.md](existing-fast-paths.md) — map bottlenecks to existing fused kernels, packed QKV paths, fused `QK norm + RoPE`, distributed overlap patterns, and open optimization PRs before proposing new code
- [scripts/diffusion_skill_env.py](scripts/diffusion_skill_env.py) — preflight helper: repo root discovery from the skill's owning checkout before falling back to `sglang.__file__`, write-access probe, benchmark/profile output directories, idle GPU selection
- [scripts/bench_diffusion_denoise.py](scripts/bench_diffusion_denoise.py) — end-to-end denoise benchmark preset runner via `sglang generate`; defaults to eager/lossless, supports explicit quality and BCG comparators plus a same-GPU applicability matrix, rejects invalid BCG capture/fallback logs and late high-quality DiT fusion mounts, forces H3 to its eager consistency mode, enables synchronized stage attribution, validates nightly preset drift, and can clean one isolated model cache after the full matrix in a `finally` block with a JSONL ledger
@@ -110,6 +110,13 @@ A zero process exit is not sufficient evidence: every accepted row must also
contain its requested perf dump and a generated image, video, or audio file.
The helper gives every cell a unique output name and rejects missing artifacts.
On machines with a read-only Hugging Face cache, combine
`--model-cache-root <task-owned-dir>` with one or more
`--seed-model-cache-root <read-only-HF-home-or-hub>` options. The helper exposes
cached repos through a task-owned copy-on-write directory overlay, downloads
misses only into the isolated cache, and removes links plus new downloads in
its normal cleanup finally block without modifying the seed cache.
Keep prompt, negative prompt, seed, shape, steps, guidance, dtype, topology,
and residency fixed. Lossless comparisons require byte-identical artifacts.
For `quality=high`, report aggregate and worst-frame SSIM/PSNR; the repository
python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/benchmark-and-profile.md
+45
View File
@@ -213,6 +213,28 @@ The helper refuses to reuse an existing per-run cache directory and never
redirects `SGLANG_CACHE_DIR`, so compiled kernel caches remain separate. Never
point this option at a shared Hugging Face or ModelScope cache.
When a machine already exposes a read-only Hugging Face cache, seed the
task-owned cache with a copy-on-write directory overlay instead of copying its
checkpoints. Immutable blobs and snapshot payloads remain symlinks, while
metadata directories stay writable so a partial seed can download missing
files into the task cache. The option may be repeated. Cleanup removes only the
task-owned overlay and new downloads; it never follows links or modifies the
seed cache:
```bash
PYTHONPATH=python python3 "$BENCH_PY" \
--model longcat-image \
--quality-bcg-matrix \
--label h100 \
--output-dir "${BENCH_DIR}" \
--model-cache-root "${MODEL_CACHE_ROOT}" \
--seed-model-cache-root /path/to/read-only/huggingface \
--cleanup-model-cache
```
Each seed path must be either a Hugging Face home containing `hub/` or the
`hub` directory itself. Do not seed from a task cache that is being cleaned.
Run the `LTX-2.3` one-stage skill preset:
```bash
@@ -293,6 +315,11 @@ Use the preset categories this way:
| `wan-i2v` | `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | Yes: `wan22_i2v_a14b_720p` | Nightly cat image and motion prompt, 1280x720, 81 frames, 4 GPUs, CFG parallel, Ulysses degree 2, text encoder CPU offload and pinned CPU memory |
| `minimax-h3-t2va` | `MiniMaxAI/MiniMax-H3` | Yes: `minimax_h3_t2va_5s` | H3 FL2VA-partition T2VA baseline: 1344x768 resolved canvas, 5 seconds / 124 frames at 24 fps, 50 joint video-audio steps, 4 GPUs, TP2 + Ulysses2, eager BF16/FP32. The helper writes H3's request contract to a generated config. |
| `longcat-image` | `meituan-longcat/LongCat-Image` | No | Eager DiT baseline at 1024x1024, 50 steps, guidance 4.5; prompt rewrite is disabled so Qwen2.5-VL does not contaminate the DiT A/B. |
| `longcat-image-edit` | `meituan-longcat/LongCat-Image-Edit` | No | Native edit baseline using the public SGLang edit fixture. Its 1536x1024 source resolves to 1264x848 under the checkpoint's roughly-one-megapixel aspect-ratio rule, and the BCG comparator captures that exact serving canvas; prompt rewrite is disabled to isolate the DiT. |
| `longcat-image-edit-turbo` | `meituan-longcat/LongCat-Image-Edit-Turbo` | No | Matching distilled edit baseline using the same public fixture, prompt, and 1264x848 BCG canvas. Its registered sampling class owns the eight-step, guidance-1 schedule. |
| `qwen-edit-base` | `Qwen/Qwen-Image-Edit` | No | Covers the original native `QwenImageEditPipelineConfig`, which is distinct from the 2509/2511 edit-plus paths; public SGLang edit fixture, 1024x1024. |
| `qwen-image-layered` | `Qwen/Qwen-Image-Layered` | No | Native layered-image path using the same public reference image and four-frame request as the GPU server case, at the registered 640x640 canvas. |
| `stable-diffusion-3.5-medium` | `stabilityai/stable-diffusion-3.5-medium-diffusers` | No | Representative native `StableDiffusion3PipelineConfig` path at 1024x1024. The repository is gated, so export `HF_TOKEN`; an unauthenticated run is a recorded access blocker, not model evidence. |
| `sana-video` | `Efficient-Large-Model/SANA-Video_2B_480p_diffusers` | No | CI-sized eager T2V baseline: 832x480, 17 frames, 8 steps, guidance 6.0. Compare `quality=lossless` and `quality=high`; high enables the BF16-input first linear-attention GEMM while retaining FP32 output and the FP32 second GEMM. |
| `sana-wm-bidirectional` | `Efficient-Large-Model/SANA-WM_bidirectional` | No | Dense two-stage TI2V baseline at the native 1280x704 shape, 49 frames, 16 fps, 20 steps, guidance 4.5, and a 48-frame forward/left action program. Uses the shared cat fixture. |
| `sana-wm-streaming` | `Efficient-Large-Model/SANA-WM_streaming` | No | Matching offline chunk-causal two-stage baseline with the streaming DiT and chunked refiner enabled; uses the same shape, fixture, seed, and camera action for comparison. |
@@ -352,6 +379,24 @@ Use the preset categories this way:
| `firered-edit-1.1` | `FireRedTeam/FireRed-Image-Edit-1.1` | No | Skill-only FireRed 1.1 image-edit preset; QwenImageEditPlus native path; uses 2-GPU CFG parallel |
| `hunyuan3d-shape` | `tencent/Hunyuan3D-2` | No | Skill-only Hunyuan3D shape-generation preset; primary metric is `Hunyuan3DShapeDenoisingStage` |
Pi0.5 is registered as an action-policy pipeline, not an image/video `sglang
generate` pipeline, so it must not be inserted into this preset table or timed
with visual-output hashes. Use its checked-in real-model lane instead:
```bash
SGLANG_RUN_PI05_E2E=1 \
SGLANG_PI05_E2E_NUM_GPUS=1 \
SGLANG_PI05_E2E_PERF_DUMP=/path/to/pi05-perf.json \
PYTHONPATH=python python3 -m pytest -s \
python/sglang/multimodal_gen/test/single_test_file/test_pi05_e2e.py
```
The action lane uses three deterministic 224x224 camera inputs, deterministic
noise, two denoise steps by default, repeatability/prefix-cache checks, and a
three-request median. Treat `action_denoise_ms` as its primary metric. Isolate
and clean its model cache with the same task-owned-cache discipline as visual
models; BCG/quality comparisons are not applicable to this API.
For Wan2.2 video models, remember the difference between **nightly alignment**
and **best latency tuning**:
- the nightly-aligned 4-GPU commands intentionally keep `--enable-cfg-parallel --ulysses-degree=2` so CFG and ring behavior stay covered
python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/bench_diffusion_denoise.py
+151 -14
View File
@@ -73,6 +73,7 @@ GATED_MODELS = {
"flux2",
"flux2-klein",
"flux2-klein-base",
"stable-diffusion-3.5-medium",
}
DIFFUSERS_FALLBACK_SIGNALS = (
"falling back to diffusers backend",
@@ -423,6 +424,55 @@ MODELS = {
"--performance-mode=manual",
],
},
"longcat-image-edit": {
"path": "meituan-longcat/LongCat-Image-Edit",
"prompt": "Make the cat wear a red hat.",
"image_path": "https://github.com/lm-sys/lm-sys.github.io/releases/download/test/TI2I_Qwen_Image_Edit_Input.jpg",
"bcg_warmup_resolutions": ["1264x848"],
"extra_args": [
"--enable-prompt-rewrite=false",
"--performance-mode=manual",
],
},
"longcat-image-edit-turbo": {
"path": "meituan-longcat/LongCat-Image-Edit-Turbo",
"prompt": "Make the cat wear a red hat.",
"image_path": "https://github.com/lm-sys/lm-sys.github.io/releases/download/test/TI2I_Qwen_Image_Edit_Input.jpg",
"bcg_warmup_resolutions": ["1264x848"],
"extra_args": [
"--enable-prompt-rewrite=false",
"--performance-mode=manual",
],
},
# The original Qwen edit checkpoint has a separate pipeline config from
# the 2509/2511 multi-image checkpoints, so keep an explicit preset.
"qwen-edit-base": {
"path": "Qwen/Qwen-Image-Edit",
"prompt": "Make the cat wear a red hat.",
"image_path": "https://github.com/lm-sys/lm-sys.github.io/releases/download/test/TI2I_Qwen_Image_Edit_Input.jpg",
"extra_args": [
"--width=1024",
"--height=1024",
],
},
"qwen-image-layered": {
"path": "Qwen/Qwen-Image-Layered",
"prompt": "a high quality, cute halloween themed illustration, consistent style and lighting",
"image_path": "https://raw.githubusercontent.com/QwenLM/Qwen-Image-Layered/main/assets/test_images/4.png",
"extra_args": [
"--num-frames=4",
"--width=640",
"--height=640",
],
},
"stable-diffusion-3.5-medium": {
"path": "stabilityai/stable-diffusion-3.5-medium-diffusers",
"prompt": "A red panda reading a book beside a sunlit window.",
"extra_args": [
"--width=1024",
"--height=1024",
],
},
"sana-video": {
"path": "Efficient-Large-Model/SANA-Video_2B_480p_diffusers",
"prompt": "A curious raccoon walks through a sunlit forest. motion score: 30.",
@@ -1261,7 +1311,70 @@ def _safe_cache_component(value: str) -> str:
return component
def _prepare_model_cache(cache_root: Path, model_key: str, label: str) -> Path:
def _resolve_seed_hub_cache(seed_root: Path) -> Path:
seed_root = seed_root.expanduser().resolve()
hub_root = seed_root / "hub"
if hub_root.is_dir():
return hub_root
if seed_root.name == "hub" and seed_root.is_dir():
return seed_root
raise FileNotFoundError(
"A seed model cache must be either a Hugging Face home containing "
f"hub/ or the hub directory itself: {seed_root}"
)
def _seed_hub_entry(source_entry: Path, target_entry: Path) -> None:
"""Build a writable cache overlay without copying immutable payloads."""
if not source_entry.is_dir():
if not target_entry.exists() and not target_entry.is_symlink():
target_entry.symlink_to(source_entry.resolve())
return
target_entry.mkdir(exist_ok=True)
for source_path in sorted(source_entry.rglob("*")):
relative_path = source_path.relative_to(source_entry)
target_path = target_entry / relative_path
if target_path.exists() or target_path.is_symlink():
continue
target_path.parent.mkdir(parents=True, exist_ok=True)
if source_path.is_symlink():
target_path.symlink_to(
os.readlink(source_path), target_is_directory=source_path.is_dir()
)
elif source_path.is_dir():
target_path.mkdir()
elif relative_path.parts[0] in {"refs", "trees"}:
shutil.copy2(source_path, target_path)
else:
target_path.symlink_to(source_path.resolve())
def _seed_model_cache(cache_dir: Path, seed_roots: list[Path]) -> None:
"""Expose read-only Hugging Face caches through copy-on-write overlays."""
target_hub = cache_dir / "huggingface" / "hub"
target_hub.mkdir(parents=True, exist_ok=True)
(target_hub / ".locks").mkdir()
for seed_root in seed_roots:
source_hub = _resolve_seed_hub_cache(seed_root)
if source_hub == target_hub or target_hub in source_hub.parents:
raise ValueError(
f"Refusing to seed the isolated cache from itself: {source_hub}"
)
for source_entry in sorted(source_hub.iterdir()):
if source_entry.name == ".locks":
continue
target_entry = target_hub / source_entry.name
_seed_hub_entry(source_entry, target_entry)
def _prepare_model_cache(
cache_root: Path,
model_key: str,
label: str,
seed_model_cache_roots: list[Path] | None = None,
) -> Path:
cache_root = cache_root.expanduser().resolve()
unsafe_roots = {Path("/"), Path.home().resolve(), REPO_ROOT.resolve()}
if cache_root in unsafe_roots:
@@ -1288,6 +1401,8 @@ def _prepare_model_cache(cache_root: Path, model_key: str, label: str) -> Path:
f"delete it without inspection: {cache_dir}"
)
cache_dir.mkdir()
if seed_model_cache_roots:
_seed_model_cache(cache_dir, seed_model_cache_roots)
return cache_dir
@@ -1600,17 +1715,15 @@ def build_sglang_cmd(
if breakable_cuda_graph:
cmd.append("--enable-breakable-cuda-graph")
parsed_args = _parse_cli_args(cmd)
if (
"warmup-resolutions" not in parsed_args
and "width" in parsed_args
and "height" in parsed_args
):
cmd.extend(
[
"--warmup-resolutions",
f"{parsed_args['width']}x{parsed_args['height']}",
]
)
if "warmup-resolutions" not in parsed_args:
warmup_resolutions = cfg.get("bcg_warmup_resolutions")
if warmup_resolutions is None and all(
name in parsed_args for name in ("width", "height")
):
warmup_resolutions = [f"{parsed_args['width']}x{parsed_args['height']}"]
if warmup_resolutions:
cmd.append("--warmup-resolutions")
cmd.extend(warmup_resolutions)
if bcg_text_buckets is not None:
cmd.append("--bcg-text-buckets")
cmd.extend(str(bucket) for bucket in bcg_text_buckets)
@@ -1910,6 +2023,7 @@ def run_benchmark_once(
breakable_cuda_graph: bool = False,
bcg_text_buckets: list[int] | None = None,
model_cache_root: Path | None = None,
seed_model_cache_roots: list[Path] | None = None,
cleanup_model_cache: bool = False,
cleanup_ledger_path: Path | None = None,
) -> dict:
@@ -1917,7 +2031,12 @@ def run_benchmark_once(
cache_dir = None
exit_reason = "error"
if model_cache_root is not None:
cache_dir = _prepare_model_cache(model_cache_root, model_key, label)
cache_dir = _prepare_model_cache(
model_cache_root,
model_key,
label,
seed_model_cache_roots=seed_model_cache_roots,
)
try:
result = _run_benchmark_once_impl(
@@ -1964,6 +2083,7 @@ def run_quality_bcg_matrix(
warmup: bool = True,
bcg_text_buckets: list[int] | None = None,
model_cache_root: Path | None = None,
seed_model_cache_roots: list[Path] | None = None,
cleanup_model_cache: bool = False,
cleanup_ledger_path: Path | None = None,
) -> list[dict]:
@@ -1976,7 +2096,10 @@ def run_quality_bcg_matrix(
exit_reason = "error"
if model_cache_root is not None:
cache_dir = _prepare_model_cache(
model_cache_root, model_key, f"{label}-quality-bcg-matrix"
model_cache_root,
model_key,
f"{label}-quality-bcg-matrix",
seed_model_cache_roots=seed_model_cache_roots,
)
cuda_visible_devices = os.environ.get("CUDA_VISIBLE_DEVICES")
@@ -2153,6 +2276,15 @@ def main():
"a cleanup ledger record. Requires --model-cache-root."
),
)
parser.add_argument(
"--seed-model-cache-root",
action="append",
default=[],
help=(
"Seed each isolated cache with a copy-on-write overlay from this "
"read-only Hugging Face home or hub directory. May be repeated."
),
)
parser.add_argument(
"--cleanup-ledger",
type=str,
@@ -2185,9 +2317,12 @@ def main():
)
if args.cleanup_model_cache and not args.model_cache_root:
parser.error("--cleanup-model-cache requires --model-cache-root")
if args.seed_model_cache_root and not args.model_cache_root:
parser.error("--seed-model-cache-root requires --model-cache-root")
model_cache_root = (
Path(args.model_cache_root) if args.model_cache_root is not None else None
)
seed_model_cache_roots = [Path(path) for path in args.seed_model_cache_root]
cleanup_ledger_path = (
Path(args.cleanup_ledger) if args.cleanup_ledger is not None else None
)
@@ -2205,6 +2340,7 @@ def main():
warmup=warmup,
bcg_text_buckets=args.bcg_text_buckets,
model_cache_root=model_cache_root,
seed_model_cache_roots=seed_model_cache_roots,
cleanup_model_cache=args.cleanup_model_cache,
cleanup_ledger_path=cleanup_ledger_path,
)
@@ -2221,6 +2357,7 @@ def main():
breakable_cuda_graph=args.breakable_cuda_graph,
bcg_text_buckets=args.bcg_text_buckets,
model_cache_root=model_cache_root,
seed_model_cache_roots=seed_model_cache_roots,
cleanup_model_cache=args.cleanup_model_cache,
cleanup_ledger_path=cleanup_ledger_path,
)
python/sglang/multimodal_gen/test/unit/test_diffusion_benchmark_skill.py
+86
View File
@@ -87,6 +87,11 @@ class TestDiffusionBenchmarkSkill(unittest.TestCase):
expected = {
"longcat-image",
"longcat-image-edit",
"longcat-image-edit-turbo",
"qwen-edit-base",
"qwen-image-layered",
"stable-diffusion-3.5-medium",
"sana-video",
"sana-wm-bidirectional",
"sana-wm-streaming",
@@ -123,6 +128,39 @@ class TestDiffusionBenchmarkSkill(unittest.TestCase):
compiled_cmd = module.build_sglang_cmd("longcat-image", torch_compile=True)
self.assertIn("--enable-torch-compile", compiled_cmd)
longcat_edit_cmd = module.build_sglang_cmd("longcat-image-edit")
self.assertIn(
"--model-path=meituan-longcat/LongCat-Image-Edit",
longcat_edit_cmd,
)
self.assertTrue(
any(arg.startswith("--image-path=") for arg in longcat_edit_cmd)
)
self.assertIn("--enable-prompt-rewrite=false", longcat_edit_cmd)
longcat_edit_bcg_cmd = module.build_sglang_cmd(
"longcat-image-edit", breakable_cuda_graph=True
)
resolution_index = longcat_edit_bcg_cmd.index("--warmup-resolutions")
self.assertEqual(longcat_edit_bcg_cmd[resolution_index + 1], "1264x848")
longcat_edit_turbo_cmd = module.build_sglang_cmd("longcat-image-edit-turbo")
self.assertIn(
"--model-path=meituan-longcat/LongCat-Image-Edit-Turbo",
longcat_edit_turbo_cmd,
)
layered_cmd = module.build_sglang_cmd("qwen-image-layered")
self.assertIn("--model-path=Qwen/Qwen-Image-Layered", layered_cmd)
self.assertIn("--num-frames=4", layered_cmd)
sd35_cmd = module.build_sglang_cmd("stable-diffusion-3.5-medium")
self.assertIn(
"--model-path=stabilityai/stable-diffusion-3.5-medium-diffusers",
sd35_cmd,
)
self.assertIn("stable-diffusion-3.5-medium", module.GATED_MODELS)
h3_cmd = module.build_sglang_cmd("minimax-h3-t2va", torch_compile=True)
self.assertNotIn("--enable-torch-compile", h3_cmd)
@@ -295,6 +333,54 @@ class TestDiffusionBenchmarkSkill(unittest.TestCase):
with self.assertRaises(FileExistsError):
module._prepare_model_cache(cache_root, "sana-video", "baseline")
def test_isolated_cache_seeds_read_only_hf_cache_with_writable_overlay(self):
with tempfile.TemporaryDirectory() as tmpdir:
temp_root = Path(tmpdir)
module = _load_benchmark_module(temp_root)
seed_root = temp_root / "shared-hf"
source_model = seed_root / "hub" / "models--org--model"
source_weight = source_model / "snapshots" / "abc" / "model.safetensors"
source_weight.parent.mkdir(parents=True)
source_weight.write_bytes(b"shared weights")
source_ref = source_model / "refs" / "main"
source_ref.parent.mkdir()
source_ref.write_text("abc")
cache_root = temp_root / "model-caches"
cache_dir = module._prepare_model_cache(
cache_root,
"sana-video",
"baseline",
seed_model_cache_roots=[seed_root],
)
seeded_model = cache_dir / "huggingface" / "hub" / "models--org--model"
self.assertTrue(seeded_model.is_dir())
self.assertFalse(seeded_model.is_symlink())
seeded_weight = seeded_model / "snapshots" / "abc" / "model.safetensors"
self.assertTrue(seeded_weight.is_symlink())
self.assertEqual(
seeded_weight.read_bytes(),
b"shared weights",
)
new_blob = seeded_model / "blobs" / "downloaded"
new_blob.parent.mkdir(exist_ok=True)
new_blob.write_bytes(b"new download")
(seeded_model / "refs" / "main").write_text("new-revision")
self.assertEqual(new_blob.read_bytes(), b"new download")
self.assertEqual(source_ref.read_text(), "abc")
module._cleanup_model_cache(
cache_root,
cache_dir,
temp_root / "cleanup.jsonl",
"sana-video",
"baseline",
"success",
)
self.assertFalse(cache_dir.exists())
self.assertEqual(source_weight.read_bytes(), b"shared weights")
def test_interrupted_run_cleans_isolated_cache_in_finally(self):
with tempfile.TemporaryDirectory() as tmpdir:
temp_root = Path(tmpdir)