[diffusion] feat: let every layerwise component be configurable (#35688)

This commit is contained in:
Mick
2026-08-20 22:38:05 +08:00
committed by GitHub
parent 04444ee352
commit 7f8f030000
4 changed files with 190 additions and 15 deletions
+16
View File
@@ -246,6 +246,22 @@ Values passed to the compatibility option `--layerwise-offload-components` must
Layerwise tuning options such as `--dit-offload-prefetch-size`, `--dit-layerwise-resident-layers`, and `--dit-layerwise-residency-policy` continue to control the streamed layer working set. Prefer the smallest component set that solves the memory issue because layerwise offload can increase latency.
Those three set the default for every streamed component. To give one component its own values, use the `component=value` forms, which also accept JSON:
```bash
sglang serve --model-path <MODEL> \
--layerwise-offload-components dit,text_encoder \
--layerwise-prefetch-size text_encoder=2 \
--layerwise-resident-layers text_encoder=4 \
--layerwise-residency-policy text_encoder=strided
```
- `--layerwise-prefetch-size`: how many layers to fetch ahead. Fractional values are a share of the stack, `>= 1` an absolute count. Deeper prefetch overlaps more of the transfer with compute, at the cost of staging buffers.
- `--layerwise-resident-layers`: how many layers stay on the GPU instead of being streamed. Resident layers are transferred once at startup, so they are removed from every pass. Fractional values are a share of the stack.
- `--layerwise-residency-policy`: `leading` keeps the first layers, `strided` spreads them across the stack so the transfers do not arrive as one burst.
A component without an entry keeps the group default, so adding these options changes nothing until one is set. Both knobs trade VRAM for transfer, and the return differs by component: a DiT earns it back once per denoising step, a text encoder or VAE once per request. Measure before raising either.
## Serve
`sglang serve` starts the HTTP server and keeps the model loaded for repeated requests.