[diffusion] fix: fix image encoder parallel folding proposal (#36863)
This commit is contained in:
@@ -93,7 +93,7 @@ Use `sglang generate --help` and `sglang serve --help` for the full argument lis
|
||||
- `--ulysses-degree {N}` and `--ring-degree {N}`: USP parallelism controls
|
||||
- `--kv-gather-degree {N}`: sequence-parallel degree that splits rows inside attention and exchanges with one K/V all-gather (queries stay local) instead of Ulysses all-to-all. Non-causal attention only; does not compose with `--ulysses-degree`/`--ring-degree` yet. When no SP degree is set explicitly, `sp_degree=2` defaults to `kv_gather_degree=2` (its measured-win zone) and higher degrees default to Ulysses; under that auto assignment, attention calls the gather path cannot take fall back to the Ulysses exchange, while an explicit degree fails instead of degrading.
|
||||
- `--enable-cfg-parallel {true|false}`: enable or explicitly disable CFG parallelism
|
||||
- `--encoder-parallel {auto|fold|dp|replicate}`: how text/image encoders use the GPUs in each DiT replica. `auto` TP-folds an encoder wide enough to benefit, selects batch DP when it can engage, and otherwise keeps the existing encoder TP layout; `fold` shards across the full replica whenever dimensions allow; `dp` splits a batched encode across encoder copies and composes with encoder TP; `replicate` disables folding and batch DP. Encoder collectives never cross `--dp-size` replicas. See [Encoder Parallelism](/docs/sglang-diffusion/encoder_parallel).
|
||||
- `--encoder-parallel {auto|fold|dp|replicate}`: how native encoders use the GPUs in each DiT replica. `auto` TP-folds a native text/image encoder wide enough to benefit, selects batch DP for an explicitly supported native text encoder when it can engage, and otherwise keeps the existing encoder TP layout; `fold` shards native text/image encoders across the full replica whenever dimensions allow; `dp` splits a batched encode across supported native text encoder copies and composes with encoder TP; `replicate` disables folding and batch DP. Encoder collectives never cross `--dp-size` replicas. See [Encoder Parallelism](/docs/sglang-diffusion/encoder_parallel).
|
||||
- `--warmup-mode {off|request|server}`: control startup warmup for `sglang serve`; `off` skips warmup, `request` primes the request path, and `server` runs a full synthetic server warmup before serving traffic
|
||||
- `--enable-torch-compile {true|false}`: compile native diffusion hot paths. When no warmup mode is configured, this also enables server warmup so first real requests do not pay compile latency.
|
||||
- `--offload-during-compile {true|false}`: when compile warmup is active, temporarily layerwise-offload DiT weights and move resident non-DiT components off-device so `max-autotune` fits on tighter-memory GPUs; the configured serving residency is restored before real traffic. Skipped under existing layerwise offload, Cache-DiT, or FSDP.
|
||||
|
||||
@@ -6,8 +6,10 @@ metatags:
|
||||
---
|
||||
|
||||
While the DiT denoises, the text and image encoders are idle — and while they
|
||||
encode, the whole DiT replica is idle. `--encoder-parallel` decides how to use
|
||||
those otherwise-unused GPUs for the encoding stage.
|
||||
encode, the whole DiT replica is idle. `--encoder-parallel` decides how native
|
||||
encoders use those otherwise-unused GPUs for the encoding stage. Folding applies
|
||||
to native text and image encoders; within-replica batch DP currently requires an
|
||||
explicitly supported native text encoder.
|
||||
|
||||
```bash
|
||||
--encoder-parallel {auto,fold,dp,replicate}
|
||||
@@ -15,9 +17,9 @@ those otherwise-unused GPUs for the encoding stage.
|
||||
|
||||
| Mode | What it does | Use when |
|
||||
| --- | --- | --- |
|
||||
| `auto` | Picks `fold`, `dp`, or the existing encoder layout from its width and the request's batch width | Default; you want the decision made per encoder |
|
||||
| `auto` | Picks folding or the existing layout for native text/image encoders, and batch DP for supported native text encoders | Default; you want the decision made per encoder |
|
||||
| `fold` | TP-shards the encoder weights across the idle DiT replica | One wide encoder dominates a single-request encode |
|
||||
| `dp` | Encoder copies split the prompt batch, then all-gather their outputs inside the replica | Throughput serving with `--batching-max-size > 1` |
|
||||
| `dp` | Supported native text encoder copies split the prompt batch, then all-gather their outputs inside the replica | Throughput serving with `--batching-max-size > 1` |
|
||||
| `replicate` | Keeps the encoder on its DiT TP group and encodes redundantly across the other replica ranks | You want to disable folding and batch DP |
|
||||
|
||||
The two accelerated modes are mutually exclusive per encoder: folding shards the
|
||||
|
||||
Reference in New Issue
Block a user