[chore] point agents at the cookbook before test configs (#40818)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Mick
2026-09-23 10:54:15 +08:00
committed by GitHub
co-authored by Mick Qian Claude Opus 5
parent 2bc435d32e
commit c2bc456b64
3 changed files with 52 additions and 1 deletions
+21
View File
@@ -0,0 +1,21 @@
# Serving a model? Read its cookbook page first
Before running, deploying, benchmarking, or reproducing a specific model, read
its page under `docs/cookbook/`:
- `docs/cookbook/autoregressive/` — LLM and VLM serving
- `docs/cookbook/diffusion/` — image, video, 3D, robotics
- `docs/cookbook/omni/`, `docs/cookbook/vla/`, `docs/cookbook/specbundle/`,
`docs/cookbook/base/`
That page carries the deployment we recommend to users — GPU count, parallelism
degrees, quantization, and the flags that matter for that model — so it is the
answer to "how should this be served", and the baseline any tuning starts from.
Configs under `test/` are a different thing: pinned, reproducible CI setups for
correctness and regression checks, not deployment advice. Reach for them when
you need an exactly reproducible run, and say which cookbook flags you diverged
from — a measurement taken on a configuration nobody deploys describes nothing.
Keep it true in both directions: when a change alters a model's recommended
deployment, update that model's cookbook page in the same PR.
@@ -4,6 +4,20 @@
SGLang's diffusion/multimodal generation subsystem. Separate from the LLM runtime (`srt`). Supports 20+ image/video diffusion models (Wan, FLUX, HunyuanVideo, LTX, Qwen-Image, etc.) with distributed inference, LoRA, and multiple attention backends.
## Serving a specific model — read its cookbook page first
**Before running or deploying any model, read `docs/cookbook/diffusion/<Family>/`.**
That page carries the deployment command we recommend to users — GPU count,
parallelism degrees, pipeline class, offload flags — and it is the answer to
"how should this model be served". Families: Cosmos, Ernie-Image, FLUX,
Ideogram, JoyEcho, Krea, LTX, LingBot-Video, LingBot-World, LongLive, MOVA,
MiniMax, Qwen-Image, SANA-Video, SANA-WM, Wan, Z-Image.
The configs under `test/` are a different thing: pinned, reproducible CI setups
for correctness and regression checks, not deployment advice. Reach for them
when you need an exact reproducible configuration, and say which cookbook flags
you diverged from.
## Quick Start
```bash
@@ -7,6 +7,22 @@ description: Use when choosing the fastest SGLang Diffusion flags for a model, G
Use this skill when the user wants the fastest command line, lower VRAM, or the right performance flags for a specific model and GPU setup.
## Start from the model's cookbook page
Tune from the deployment we already recommend, not from scratch: read
`docs/cookbook/diffusion/<Family>/` for the model first, take its command as
the baseline, then apply the options below. The cookbook page is what users
follow, so a recipe that silently disagrees with it on GPU count, parallelism
degrees, or offload flags is a recipe for a configuration nobody runs.
Families with a cookbook page: Cosmos, Ernie-Image, FLUX, Ideogram, JoyEcho,
Krea, LTX, LingBot-Video, LingBot-World, LongLive, MOVA, MiniMax, Qwen-Image,
SANA-Video, SANA-WM, Wan, Z-Image.
`test/` configs are pinned CI setups for correctness and regression checks, not
deployment advice. Use them when you need an exactly reproducible run, and note
which cookbook flags you diverged from.
Before running any `sglang generate` command below inside the diffusion container:
- use `python/sglang/multimodal_gen/.claude/skills/sglang-diffusion-benchmark-profile/scripts/diffusion_skill_env.py` to derive the repo root, verify write access, and choose idle GPU(s)
- export `HF_TOKEN` first when the selected model lives in a gated Hugging Face repo such as `black-forest-labs/FLUX.*`
@@ -508,7 +524,7 @@ Use these as first commands to benchmark, not as universal winners.
| FireRed-Image-Edit 1.0 / 1.1 | 1024x1024 image edit, 40 steps, guidance 4.0 | `--backend=sglang --num-gpus 2 --enable-cfg-parallel --ulysses-degree 1 --enable-torch-compile --warmup-mode request --dit-layerwise-offload false --dit-cpu-offload false` | Uses the native `QwenImageEditPlusPipeline` path. 2-GPU CFG parallel is the validated H100 starting point; benchmark 1.0 and 1.1 separately because checkpoint differences can change denoise latency. |
| Hunyuan3D-2 shape | Shape generation, 50 steps, guidance 5.0 | `--backend=sglang --enable-torch-compile --warmup-mode request --dit-layerwise-offload false --dit-cpu-offload false` | Focus on `Hunyuan3DShapeDenoisingStage`; keep mesh export/paint timings separate from denoise. |
| LingBot Video MoE 30B | 384x640, 17 frames, 12 steps for the current GPU case | `--model-path robbyant/lingbot-video-moe-30b-a3b --text-encoder-cpu-offload` | Native T2V path. Prompts are structured JSON captions, not raw free text; keep that contract when comparing latency or quality. Current main can mount the fused Triton RMSNorm path at `quality=extra-high` or `quality=high`; keep `lossless` as the reference. `--text-encoder-cpu-offload` targets memory-bound multi-GPU or small-VRAM cards; on a single large-VRAM GPU (e.g. 275 GB B300) the whole model stays resident (~73 GB peak), so dropping the flag removes H2D/D2H traffic and was 8% faster end to end (3.80s -> 3.49s, bit-identical). |
| MOVA / Helios / LingBot World | Use the benchmark/profile presets or server test cases first | `--enable-torch-compile --warmup-mode request`; pin offload and topology flags explicitly | These video/realtime families have model-specific stages and condition handling. For LingBot World causal serving, keep `--kv-cache-quant off` as the exact cache baseline before testing INT4/INT2. |
| MOVA / Helios / LingBot World | Start from the family's cookbook page — it pins the GPU count and parallelism these realtime paths need (e.g. LingBot World is served with `--num-gpus 4 --ulysses-degree 4`, LingBot World 2.0 with 8) | `--enable-torch-compile --warmup-mode request`; pin offload and topology flags explicitly | These video/realtime families have model-specific stages and condition handling. Realtime keeps up with playback only at the documented GPU count, so do not tune down from it by accident. For LingBot World causal serving, keep `--kv-cache-quant off` as the exact cache baseline before testing INT4/INT2. |
## Historical PR Watchlist