diff --git a/.claude/rules/cookbook-first.md b/.claude/rules/cookbook-first.md new file mode 100644 index 0000000000..92f2e94a05 --- /dev/null +++ b/.claude/rules/cookbook-first.md @@ -0,0 +1,21 @@ +# Serving a model? Read its cookbook page first + +Before running, deploying, benchmarking, or reproducing a specific model, read +its page under `docs/cookbook/`: + +- `docs/cookbook/autoregressive/` — LLM and VLM serving +- `docs/cookbook/diffusion/` — image, video, 3D, robotics +- `docs/cookbook/omni/`, `docs/cookbook/vla/`, `docs/cookbook/specbundle/`, + `docs/cookbook/base/` + +That page carries the deployment we recommend to users — GPU count, parallelism +degrees, quantization, and the flags that matter for that model — so it is the +answer to "how should this be served", and the baseline any tuning starts from. + +Configs under `test/` are a different thing: pinned, reproducible CI setups for +correctness and regression checks, not deployment advice. Reach for them when +you need an exactly reproducible run, and say which cookbook flags you diverged +from — a measurement taken on a configuration nobody deploys describes nothing. + +Keep it true in both directions: when a change alters a model's recommended +deployment, update that model's cookbook page in the same PR. diff --git a/python/sglang/multimodal_gen/.claude/CLAUDE.md b/python/sglang/multimodal_gen/.claude/CLAUDE.md index a8b4f5bb5b..e952cb2ac3 100644 --- a/python/sglang/multimodal_gen/.claude/CLAUDE.md +++ b/python/sglang/multimodal_gen/.claude/CLAUDE.md @@ -4,6 +4,20 @@ SGLang's diffusion/multimodal generation subsystem. Separate from the LLM runtime (`srt`). Supports 20+ image/video diffusion models (Wan, FLUX, HunyuanVideo, LTX, Qwen-Image, etc.) with distributed inference, LoRA, and multiple attention backends. +## Serving a specific model — read its cookbook page first + +**Before running or deploying any model, read `docs/cookbook/diffusion//`.** +That page carries the deployment command we recommend to users — GPU count, +parallelism degrees, pipeline class, offload flags — and it is the answer to +"how should this model be served". Families: Cosmos, Ernie-Image, FLUX, +Ideogram, JoyEcho, Krea, LTX, LingBot-Video, LingBot-World, LongLive, MOVA, +MiniMax, Qwen-Image, SANA-Video, SANA-WM, Wan, Z-Image. + +The configs under `test/` are a different thing: pinned, reproducible CI setups +for correctness and regression checks, not deployment advice. Reach for them +when you need an exact reproducible configuration, and say which cookbook flags +you diverged from. + ## Quick Start ```bash diff --git a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-performance/SKILL.md b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-performance/SKILL.md index a6d1ea657f..459ed2ea3c 100644 --- a/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-performance/SKILL.md +++ b/python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-performance/SKILL.md @@ -7,6 +7,22 @@ description: Use when choosing the fastest SGLang Diffusion flags for a model, G Use this skill when the user wants the fastest command line, lower VRAM, or the right performance flags for a specific model and GPU setup. +## Start from the model's cookbook page + +Tune from the deployment we already recommend, not from scratch: read +`docs/cookbook/diffusion//` for the model first, take its command as +the baseline, then apply the options below. The cookbook page is what users +follow, so a recipe that silently disagrees with it on GPU count, parallelism +degrees, or offload flags is a recipe for a configuration nobody runs. + +Families with a cookbook page: Cosmos, Ernie-Image, FLUX, Ideogram, JoyEcho, +Krea, LTX, LingBot-Video, LingBot-World, LongLive, MOVA, MiniMax, Qwen-Image, +SANA-Video, SANA-WM, Wan, Z-Image. + +`test/` configs are pinned CI setups for correctness and regression checks, not +deployment advice. Use them when you need an exactly reproducible run, and note +which cookbook flags you diverged from. + Before running any `sglang generate` command below inside the diffusion container: - use `python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/diffusion_skill_env.py` to derive the repo root, verify write access, and choose idle GPU(s) - export `HF_TOKEN` first when the selected model lives in a gated Hugging Face repo such as `black-forest-labs/FLUX.*` @@ -508,7 +524,7 @@ Use these as first commands to benchmark, not as universal winners. | FireRed-Image-Edit 1.0 / 1.1 | 1024x1024 image edit, 40 steps, guidance 4.0 | `--backend=sglang --num-gpus 2 --enable-cfg-parallel --ulysses-degree 1 --enable-torch-compile --warmup-mode request --dit-layerwise-offload false --dit-cpu-offload false` | Uses the native `QwenImageEditPlusPipeline` path. 2-GPU CFG parallel is the validated H100 starting point; benchmark 1.0 and 1.1 separately because checkpoint differences can change denoise latency. | | Hunyuan3D-2 shape | Shape generation, 50 steps, guidance 5.0 | `--backend=sglang --enable-torch-compile --warmup-mode request --dit-layerwise-offload false --dit-cpu-offload false` | Focus on `Hunyuan3DShapeDenoisingStage`; keep mesh export/paint timings separate from denoise. | | LingBot Video MoE 30B | 384x640, 17 frames, 12 steps for the current GPU case | `--model-path robbyant/lingbot-video-moe-30b-a3b --text-encoder-cpu-offload` | Native T2V path. Prompts are structured JSON captions, not raw free text; keep that contract when comparing latency or quality. Current main can mount the fused Triton RMSNorm path at `quality=extra-high` or `quality=high`; keep `lossless` as the reference. `--text-encoder-cpu-offload` targets memory-bound multi-GPU or small-VRAM cards; on a single large-VRAM GPU (e.g. 275 GB B300) the whole model stays resident (~73 GB peak), so dropping the flag removes H2D/D2H traffic and was 8% faster end to end (3.80s -> 3.49s, bit-identical). | -| MOVA / Helios / LingBot World | Use the benchmark/profile presets or server test cases first | `--enable-torch-compile --warmup-mode request`; pin offload and topology flags explicitly | These video/realtime families have model-specific stages and condition handling. For LingBot World causal serving, keep `--kv-cache-quant off` as the exact cache baseline before testing INT4/INT2. | +| MOVA / Helios / LingBot World | Start from the family's cookbook page — it pins the GPU count and parallelism these realtime paths need (e.g. LingBot World is served with `--num-gpus 4 --ulysses-degree 4`, LingBot World 2.0 with 8) | `--enable-torch-compile --warmup-mode request`; pin offload and topology flags explicitly | These video/realtime families have model-specific stages and condition handling. Realtime keeps up with playback only at the documented GPU count, so do not tune down from it by accident. For LingBot World causal serving, keep `--kv-cache-quant off` as the exact cache baseline before testing INT4/INT2. | ## Historical PR Watchlist