107 lines
4.3 KiB
Plaintext
107 lines
4.3 KiB
Plaintext
---
|
||
title: "Encoder Parallelism"
|
||
tag: "preserve"
|
||
metatags:
|
||
description: "Configure how SGLang Diffusion spreads text and image encoding across GPUs: parallel folding, batch data-parallel encoding, or replication."
|
||
---
|
||
|
||
While the DiT denoises, the text and image encoders are idle — and while they
|
||
encode, the whole DiT replica is idle. `--encoder-parallel` decides how to use
|
||
those otherwise-unused GPUs for the encoding stage.
|
||
|
||
```bash
|
||
--encoder-parallel {auto,fold,dp,replicate}
|
||
```
|
||
|
||
| Mode | What it does | Use when |
|
||
| --- | --- | --- |
|
||
| `auto` | Picks `fold`, `dp`, or `replicate` per encoder from its width and the request's batch width | Default for `generate`; you want the decision made per encoder |
|
||
| `fold` | TP-shards the encoder weights across the idle DiT replica | One wide encoder dominates a single-request encode |
|
||
| `dp` | Each rank encodes its slice of the prompt batch, then the outputs are all-gathered | Default for `serve`; needs `--batching-max-size > 1` to engage |
|
||
| `replicate` | Every rank encodes the whole batch redundantly | You want the encoding stage to match single-GPU numerics exactly |
|
||
|
||
The two accelerated modes are mutually exclusive per encoder: folding shards the
|
||
weights for the lifetime of the loaded model, so a folded encoder cannot also be
|
||
data-parallel.
|
||
|
||
## Which Mode Wins
|
||
|
||
Measured on H100 across T5 (hidden 4096), Qwen3 (2560), and CLIP-L (768) at
|
||
batch 1–8 and replica sizes 2 and 4:
|
||
|
||
- **Folding** pays when the encoder is wide enough that sharding its GEMMs beats
|
||
the per-layer all-reduce it adds. T5 gains; Qwen3 (+35%) and CLIP-L (+50%) get
|
||
slower, so folding is gated at hidden ≥ 4096. Its benefit also saturates as
|
||
the replica grows, since each rank's slice keeps shrinking.
|
||
- **Data-parallel** pays only when the encode is compute-bound, which needs a
|
||
wide encoder (hidden ≥ 1024 — CLIP-L is slower at every batch and replica
|
||
measured) and more than one prompt in a single encode call.
|
||
- **Replication** is the right answer whenever neither condition holds, which is
|
||
most single-request latency work.
|
||
|
||
`auto` encodes exactly these rules, so prefer it unless you are pinning a
|
||
configuration you measured yourself.
|
||
|
||
## Numerics
|
||
|
||
`fold` and `replicate` are bitwise-identical to single-GPU encoding: folding
|
||
shards a GEMM and reduces it, which is the same arithmetic the unsharded kernel
|
||
performs.
|
||
|
||
`dp` is **not** bitwise-identical. Each rank runs the full unsharded encoder on
|
||
a smaller batch, so the GEMM tiling and reduction order differ from the batched
|
||
reference — the same floating-point reordering class as choosing a different
|
||
attention backend or parallelism strategy, not a precision loss. The gathered
|
||
result is mathematically equivalent, and per-request results stay deterministic
|
||
for a fixed batch shape, but embeddings will not match a `replicate` run
|
||
bit-for-bit, and long video sampling can amplify the difference into visible
|
||
frame differences. Use `replicate` (or `fold`) when you need bit-exact
|
||
reproducibility against a single-GPU reference, e.g. when refreshing consistency
|
||
baselines.
|
||
|
||
## Recommended Commands
|
||
|
||
Throughput serving. `serve` already defaults to `dp`, but a single encode call
|
||
must carry more than one prompt for it to engage, so raise the batching ceiling
|
||
too — an encoder flag deliberately does not change DiT batching for you:
|
||
|
||
```bash
|
||
sglang serve \
|
||
--model-path Qwen/Qwen-Image-2512 \
|
||
--model-type diffusion \
|
||
--num-gpus 2 \
|
||
--encoder-parallel dp \
|
||
--batching-max-size 2
|
||
```
|
||
|
||
Single-request latency with one wide text encoder:
|
||
|
||
```bash
|
||
sglang serve \
|
||
--model-path Wan-AI/Wan2.2-TI2V-5B-Diffusers \
|
||
--model-type diffusion \
|
||
--num-gpus 4 \
|
||
--ulysses-degree 4 \
|
||
--encoder-parallel fold
|
||
```
|
||
|
||
Bit-exact reproducibility against a single-GPU reference:
|
||
|
||
```bash
|
||
sglang serve \
|
||
--model-path Qwen/Qwen-Image-2512 \
|
||
--model-type diffusion \
|
||
--num-gpus 2 \
|
||
--encoder-parallel replicate
|
||
```
|
||
|
||
## Interaction With Other Flags
|
||
|
||
- **Tensor / data parallel**: `dp` requires a replicated encoder, so it is
|
||
skipped when `--tp-size > 1` or `--dp-size > 1`.
|
||
- **Dynamic batching**: `dp` only pays with a wide batch, so selecting it raises
|
||
the default batching ceiling. See [Inference Batching](./dynamic_batching).
|
||
- **Sequence parallelism**: independent — SP splits the DiT's latent sequence,
|
||
encoder parallelism splits the encoding stage. See
|
||
[Sequence Parallelism](./ring_sp_performance).
|