[diffusion] fix: decouple encoder parallelism from the dit parallel layout (#34713)

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx

@@ -842,7 +842,7 @@ the strict `quality="high"` deployment contract.
 - The released visual VAE quality recipe uses overlapping tiled decode. SGLang keeps that recipe by default and distributes complete tiles across the decode group; this changes scheduling, not the computation inside each tile.
 - H3 rejects `--vae-config.parallel-decode-mode spatial` and `spatial_shard`: validation found output mismatches. Use the default released tiled recipe.
 - Keep the default `--encoder-parallel auto`. With the server’s default `batching_max_size` of 1, single-node H100/H200/B200/B300 recipes with peer-to-peer access fold the Qwen text encoder over otherwise idle Ulysses ranks. This is separate from DiT tensor parallelism. A pure-TP recipe already shards the encoder over its TP group and does not add a world fold.
-- For throughput-oriented serving, select **DP (batched throughput)**. The picker pairs `--encoder-parallel dp` with an editable `--batching-max-size` greater than 1; compatible requests are distributed across ranks, while every rank keeps a full encoder replica. Encoder DP requires TP1 and DiT DP1, so it is disabled for the H100 TP2 + Ulysses2 and RTX 5090 TP2 recipes. It provides no benefit for a batch of one and is not bitwise-identical to the folded deployment.
+- For throughput-oriented serving, select **DP (batched throughput)**. The picker pairs `--encoder-parallel dp` with an editable `--batching-max-size` greater than 1. Encoder DP stays inside each DiT replica and composes with encoder TP: the H100 TP2 + Ulysses2 recipe has two TP-sharded encoder copies that can split a batch, while the RTX 5090 pure-TP2 recipe has one encoder copy and therefore no additional batch-DP degree. It provides no benefit for a batch of one and is not bitwise-identical to the unsplit deployment.
 - Use explicit **Fold** to prioritize single-request latency and encoder memory on a measured high-bandwidth single-node topology. Use **Replicate** as the compatibility path when folding or encoder DP is unsuitable.
 - `--use-fsdp-inference true` shards only the DiT. MiniMax-H3 preserves the original FP32 dtype of its patch, time, and output projections during FSDP all-gather, so this path does not trade numerical correctness for memory. On 4×H100, prefer TP2 + Ulysses2 for speed; use FSDP as an explicit capacity policy rather than assuming it is faster.
 - `speed` keeps model components resident, while `auto` applies the model-aware 120 GiB residency threshold. `memory` prioritizes avoiding OOM and includes the executable VAE decoder in its default layerwise set. A measured recipe with sufficient headroom can opt into `--component-residency vae=resident`; the 2×H100 CI recipe does this because the VAE's 4.8 GiB/GPU cost avoids repeated decoder transfers during tiled decode. DiT residency and prefetch knobs remain scoped to the DiT. Use `speed` only after confirming that the complete target workload fits.

docs/docs/sglang-diffusion/api/cli.mdx

@@ -92,7 +92,7 @@ Use `sglang generate --help` and `sglang serve --help` for the full argument lis
 - `--ulysses-degree {N}` and `--ring-degree {N}`: USP parallelism controls
 - `--kv-gather-degree {N}`: sequence-parallel degree that splits rows inside attention and exchanges with one K/V all-gather (queries stay local) instead of Ulysses all-to-all. Non-causal attention only; does not compose with `--ulysses-degree`/`--ring-degree` yet. When no SP degree is set explicitly, `sp_degree=2` defaults to `kv_gather_degree=2` (its measured-win zone) and higher degrees default to Ulysses; under that auto assignment, attention calls the gather path cannot take fall back to the Ulysses exchange, while an explicit degree fails instead of degrading.
 - `--enable-cfg-parallel {true|false}`: enable or explicitly disable CFG parallelism
-- `--encoder-parallel {auto|fold|dp|replicate}`: how the text/image encoders use the GPUs the DiT replica leaves idle during encoding. `auto` (the default for both `generate` and `serve`) TP-folds an encoder wide enough to pay for the per-layer all-reduce, selects DP for a server batch when it can engage, and otherwise replicates; `fold` forces the shard whenever the dims allow it; `dp` splits a batched encode across ranks and needs `--batching-max-size > 1` to engage; `replicate` encodes redundantly on every rank. `fold` and `replicate` are bitwise-identical to single-GPU encoding. See [Encoder Parallelism](/docs/sglang-diffusion/encoder_parallel).
+- `--encoder-parallel {auto|fold|dp|replicate}`: how text/image encoders use the GPUs in each DiT replica. `auto` TP-folds an encoder wide enough to benefit, selects batch DP when it can engage, and otherwise keeps the existing encoder TP layout; `fold` shards across the full replica whenever dimensions allow; `dp` splits a batched encode across encoder copies and composes with encoder TP; `replicate` disables folding and batch DP. Encoder collectives never cross `--dp-size` replicas. See [Encoder Parallelism](/docs/sglang-diffusion/encoder_parallel).
 - `--warmup-mode {off|request|server}`: control startup warmup for `sglang serve`; `off` skips warmup, `request` primes the request path, and `server` runs a full synthetic server warmup before serving traffic
 - `--enable-torch-compile {true|false}`: compile native diffusion hot paths. When no warmup mode is configured, this also enables server warmup so first real requests do not pay compile latency.
 - `--offload-during-compile {true|false}`: when compile warmup is active, temporarily layerwise-offload DiT weights and move resident non-DiT components off-device so `max-autotune` fits on tighter-memory GPUs; the configured serving residency is restored before real traffic. Skipped under existing layerwise offload, Cache-DiT, or FSDP.

docs/docs/sglang-diffusion/encoder_parallel.mdx

@@ -15,10 +15,10 @@ those otherwise-unused GPUs for the encoding stage.
 
 | Mode | What it does | Use when |
 | --- | --- | --- |
-| `auto` | Picks `fold`, `dp`, or `replicate` per encoder from its width and the request's batch width | Default for `generate`; you want the decision made per encoder |
+| `auto` | Picks `fold`, `dp`, or the existing encoder layout from its width and the request's batch width | Default; you want the decision made per encoder |
 | `fold` | TP-shards the encoder weights across the idle DiT replica | One wide encoder dominates a single-request encode |
-| `dp` | Each rank encodes its slice of the prompt batch, then the outputs are all-gathered | Default for `serve`; needs `--batching-max-size > 1` to engage |
-| `replicate` | Every rank encodes the whole batch redundantly | You want the encoding stage to match single-GPU numerics exactly |
+| `dp` | Encoder copies split the prompt batch, then all-gather their outputs inside the replica | Throughput serving with `--batching-max-size > 1` |
+| `replicate` | Keeps the encoder on its DiT TP group and encodes redundantly across the other replica ranks | You want to disable folding and batch DP |
 
 The two accelerated modes are mutually exclusive per encoder: folding shards the
 weights for the lifetime of the loaded model, so a folded encoder cannot also be
@@ -44,32 +44,31 @@ configuration you measured yourself.
 
 ## Numerics
 
-`fold` and `replicate` are bitwise-identical to single-GPU encoding: folding
-shards a GEMM and reduces it, which is the same arithmetic the unsharded kernel
-performs.
+`replicate` matches single-GPU encoding bit-for-bit only when the encoder TP
+degree is one. Both the existing DiT TP layout and `fold` can reorder parallel
+reductions; they are mathematically equivalent but are not generally bitwise
+identical to a single-GPU kernel.
 
-`dp` is **not** bitwise-identical. Each rank runs the full unsharded encoder on
-a smaller batch, so the GEMM tiling and reduction order differ from the batched
-reference — the same floating-point reordering class as choosing a different
-attention backend or parallelism strategy, not a precision loss. The gathered
-result is mathematically equivalent, and per-request results stay deterministic
-for a fixed batch shape, but embeddings will not match a `replicate` run
-bit-for-bit, and long video sampling can amplify the difference into visible
-frame differences. Use `replicate` (or `fold`) when you need bit-exact
-reproducibility against a single-GPU reference, e.g. when refreshing consistency
-baselines.
+`dp` is also not bitwise-identical: each encoder copy sees a smaller batch, so
+GEMM tiling can differ from the unsplit reference. It may compose with encoder
+TP: all ranks in one TP group receive the same batch slice, and corresponding TP
+ranks gather outputs across the orthogonal encoder-DP group. The result remains
+mathematically equivalent and deterministic for a fixed topology and batch
+shape, but long video sampling can amplify small floating-point differences.
 
 ## Recommended Commands
 
-Throughput serving. `serve` already defaults to `dp`, but a single encode call
-must carry more than one prompt for it to engage, so raise the batching ceiling
-too — an encoder flag deliberately does not change DiT batching for you:
+Throughput serving. A single encode call must carry more than one prompt for DP
+to engage, so raise the batching ceiling too; an encoder flag deliberately does
+not change DiT batching for you:
 
 ```bash
 sglang serve \
-  --model-path Qwen/Qwen-Image-2512 \
+  --model-path Wan-AI/Wan2.2-TI2V-5B-Diffusers \
   --model-type diffusion \
   --num-gpus 2 \
+  --tp-size 1 \
+  --ulysses-degree 2 \
   --encoder-parallel dp \
   --batching-max-size 2
 ```
@@ -97,10 +96,16 @@ sglang serve \
 
 ## Interaction With Other Flags
 
-- **Tensor / data parallel**: `dp` requires a replicated encoder, so it is
-  skipped when `--tp-size > 1` or `--dp-size > 1`.
-- **Dynamic batching**: `dp` only pays with a wide batch, so selecting it raises
-  the default batching ceiling. See [Inference Batching](./dynamic_batching).
-- **Sequence parallelism**: independent — SP splits the DiT's latent sequence,
-  encoder parallelism splits the encoding stage. See
+- **Tensor parallel**: encoder DP composes with TP. A TP group jointly encodes
+  one batch slice; the orthogonal ranks inside the same pipeline replica split
+  and gather the batch. A pure-TP replica has one encoder copy, so there is no
+  additional batch-DP degree.
+- **Data parallel**: encoder collectives never cross pipeline replicas. With
+  `--dp-size > 1`, each replica independently uses its own TP/SP/CFG ranks.
+- **Dynamic batching**: `dp` only pays with a wide batch. Selecting it does not
+  change `--batching-max-size`; configure that separately. See
+  [Inference Batching](./dynamic_batching).
+- **Sequence parallelism**: SP splits the DiT latent sequence. During encoding,
+  those ranks either hold encoder copies for batch DP or join a folded encoder.
+  See
   [Sequence Parallelism](./ring_sp_performance).