[diffusion] feat: generalize layerwise offload residency mixin to all components (#24593)

This commit is contained in:
Mick
2026-05-16 11:44:46 +08:00
committed by GitHub
parent a741d0cc56
commit 416fdbbb3d
66 changed files with 1569 additions and 316 deletions
+15
View File
@@ -160,6 +160,21 @@ HTTP server-only arguments are ignored by `sglang generate`.
For diffusers pipelines, Cache-DiT can be enabled with `SGLANG_CACHE_DIT_ENABLED=true` or `--cache-dit-config`. See [Cache-DiT](../performance/cache/cache_dit.md).
### Layerwise Offload
Use layerwise offload when a large component does not fit comfortably in GPU memory. By default, `--dit-layerwise-offload` only applies to legacy DiT components. Use `--layerwise-offload-components` to select pipeline component names explicitly (`--layerwise-offload-modules` is accepted as an alias):
```bash
sglang generate \
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
--dit-layerwise-offload \
--layerwise-offload-components transformer text_encoder \
--dit-offload-prefetch-size 0 \
--prompt "A quiet city street after rain"
```
The values must match keys in the selected pipeline's `pipeline.modules`, such as `transformer`, `text_encoder`, `image_encoder`, `vae`, `condition_image_encoder`, `spatial_upsampler`, or `vocoder`. Use `all` to select every layerwise-offloadable component. Prefer the smallest component set that solves the memory issue because layerwise offload can increase latency.
## Serve
`sglang serve` starts the HTTP server and keeps the model loaded for repeated requests.