[diffusion] fix: fix image encoder parallel folding proposal (#36863)

This commit is contained in:
Mick
2026-08-29 14:34:36 +08:00
committed by GitHub
parent f5319af1c9
commit fa474b0441
4 changed files with 40 additions and 12 deletions
+1 -1
View File
@@ -93,7 +93,7 @@ Use `sglang generate --help` and `sglang serve --help` for the full argument lis
- `--ulysses-degree {N}` and `--ring-degree {N}`: USP parallelism controls
- `--kv-gather-degree {N}`: sequence-parallel degree that splits rows inside attention and exchanges with one K/V all-gather (queries stay local) instead of Ulysses all-to-all. Non-causal attention only; does not compose with `--ulysses-degree`/`--ring-degree` yet. When no SP degree is set explicitly, `sp_degree=2` defaults to `kv_gather_degree=2` (its measured-win zone) and higher degrees default to Ulysses; under that auto assignment, attention calls the gather path cannot take fall back to the Ulysses exchange, while an explicit degree fails instead of degrading.
- `--enable-cfg-parallel {true|false}`: enable or explicitly disable CFG parallelism
- `--encoder-parallel {auto|fold|dp|replicate}`: how text/image encoders use the GPUs in each DiT replica. `auto` TP-folds an encoder wide enough to benefit, selects batch DP when it can engage, and otherwise keeps the existing encoder TP layout; `fold` shards across the full replica whenever dimensions allow; `dp` splits a batched encode across encoder copies and composes with encoder TP; `replicate` disables folding and batch DP. Encoder collectives never cross `--dp-size` replicas. See [Encoder Parallelism](/docs/sglang-diffusion/encoder_parallel).
- `--encoder-parallel {auto|fold|dp|replicate}`: how native encoders use the GPUs in each DiT replica. `auto` TP-folds a native text/image encoder wide enough to benefit, selects batch DP for an explicitly supported native text encoder when it can engage, and otherwise keeps the existing encoder TP layout; `fold` shards native text/image encoders across the full replica whenever dimensions allow; `dp` splits a batched encode across supported native text encoder copies and composes with encoder TP; `replicate` disables folding and batch DP. Encoder collectives never cross `--dp-size` replicas. See [Encoder Parallelism](/docs/sglang-diffusion/encoder_parallel).
- `--warmup-mode {off|request|server}`: control startup warmup for `sglang serve`; `off` skips warmup, `request` primes the request path, and `server` runs a full synthetic server warmup before serving traffic
- `--enable-torch-compile {true|false}`: compile native diffusion hot paths. When no warmup mode is configured, this also enables server warmup so first real requests do not pay compile latency.
- `--offload-during-compile {true|false}`: when compile warmup is active, temporarily layerwise-offload DiT weights and move resident non-DiT components off-device so `max-autotune` fits on tighter-memory GPUs; the configured serving residency is restored before real traffic. Skipped under existing layerwise offload, Cache-DiT, or FSDP.
@@ -6,8 +6,10 @@ metatags:
---
While the DiT denoises, the text and image encoders are idle — and while they
encode, the whole DiT replica is idle. `--encoder-parallel` decides how to use
those otherwise-unused GPUs for the encoding stage.
encode, the whole DiT replica is idle. `--encoder-parallel` decides how native
encoders use those otherwise-unused GPUs for the encoding stage. Folding applies
to native text and image encoders; within-replica batch DP currently requires an
explicitly supported native text encoder.
```bash
--encoder-parallel {auto,fold,dp,replicate}
@@ -15,9 +17,9 @@ those otherwise-unused GPUs for the encoding stage.
| Mode | What it does | Use when |
| --- | --- | --- |
| `auto` | Picks `fold`, `dp`, or the existing encoder layout from its width and the request's batch width | Default; you want the decision made per encoder |
| `auto` | Picks folding or the existing layout for native text/image encoders, and batch DP for supported native text encoders | Default; you want the decision made per encoder |
| `fold` | TP-shards the encoder weights across the idle DiT replica | One wide encoder dominates a single-request encode |
| `dp` | Encoder copies split the prompt batch, then all-gather their outputs inside the replica | Throughput serving with `--batching-max-size > 1` |
| `dp` | Supported native text encoder copies split the prompt batch, then all-gather their outputs inside the replica | Throughput serving with `--batching-max-size > 1` |
| `replicate` | Keeps the encoder on its DiT TP group and encodes redundantly across the other replica ranks | You want to disable folding and batch DP |
The two accelerated modes are mutually exclusive per encoder: folding shards the