[diffusion] feat: cross-node sequence parallelism (Ulysses x Ring) (#33327)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx

@@ -65,7 +65,8 @@ behavior such as Cache-DiT is documented separately below.
 
 **Deployment Profile** exposes resident and FSDP placement on B200, B300,
 H200, and H100. Resident is the latency-oriented default; FSDP reduces DiT
-weight residency at the cost of per-block parameter collectives. **Online
+weight residency at the cost of per-block parameter collectives. On H200 it
+also selects the verified 2-node cross-node topology. **Online
 Quantization** appears only on B200 and B300. AMD keeps its resident AITER
 recipe, while RTX 5090 uses its dedicated layerwise-offload profile.
 
@@ -564,7 +565,8 @@ listed hardware and topology; it is not inherited by a similar GPU family.
 
 | Feature | Validation status | Notes |
 | --- | --- | --- |
-| Ulysses sequence parallelism | Verified: 8× B200, 4× H200, 4× H100, and Ulysses1/2/4/8 on MI300X and MI355X | Use `--ulysses-degree`; Ring is not compatible with H3's packed multi-segment attention. |
+| Ulysses sequence parallelism | Verified: 8× B200, 4× H200, 4× H100, and Ulysses1/2/4/8 on MI300X and MI355X | Use `--ulysses-degree`. Combine with Ring for cross-node scaling; see the next row. |
+| Ring sequence parallelism (cross-node) | Verified: 2 nodes of 8× H200 each (Ulysses8 × Ring2) | Use `--ring-degree` together with `--nnodes`/`--node-rank`/`--dist-init-addr`. Ring shards the sequence across nodes while Ulysses shards heads within a node; H3's packed multi-segment attention only supports Ring across the node boundary, not within a single node's Ulysses group. Requires `--encoder-parallel replicate` — `auto`'s fold decision is not node-boundary aware. See the benchmark section below. |
 | SageAttention | Supported | Use `--attention-backend sage_attn` to select the native packed varlen path; install the SageAttention dependency first. |
 | Tensor parallelism | Verified: B200 TP2 + Ulysses4; H100 TP2 + Ulysses2 and TP4 + Ulysses1 | `--tp-size` may be combined with Ulysses when the TP-local head count remains divisible by the Ulysses degree. On 4×H100, TP2 + Ulysses2 is the measured speed default. |
 | FSDP inference | Verified: 4× B200 and 4× H100 + Ulysses4 | Preserves H3's mixed BF16/FP32 parameter policy. B200 completed the exact eager comparison; H100 completed consecutive real requests at about 57 GB peak memory per GPU. |
@@ -673,7 +675,7 @@ the configurations with collected measurements:
 | --- | --- | --- |
 | B300 | 8× Ulysses8 resident | 8× FSDP + Ulysses8; the 8-GPU sweep is not a minimum-GPU claim. |
 | B200 | 8× Ulysses8 resident | 4× FSDP + Ulysses4 |
-| H200 | 4× Ulysses4 resident | 4× FSDP + Ulysses4; 4× TP2 + Ulysses2 |
+| H200 | 4× Ulysses4 resident | 4× FSDP + Ulysses4; 4× TP2 + Ulysses2; 2 nodes × 8× Ulysses8×Ring2 cross-node |
 | H100 | 4× TP2 + Ulysses2 resident | 4× TP4 + Ulysses1; 4× FSDP + Ulysses4 |
 | MI300X / MI355X | 8× Ulysses8 resident | 1×, 2×, and 4× scaling runs |
 | RTX 5090 | 2× TP2 + layerwise offload | — |
@@ -779,6 +781,72 @@ GPU lower, which is why it remains the 80 GB H100 recipe. Matching the warmup
 request to the served resolution removes the cold first-request cost on both
 topologies (about 10 s end-to-end on this workload).
 
+### H200 cross-node scaling
+
+Long references and long durations grow the packed sequence length, and
+Ulysses alone cannot scale sequence parallelism past the GPU count of one
+node without either violating head-count divisibility or exposing
+all-to-all traffic across the slower inter-node link. H3 combines
+node-local Ulysses with cross-node Ring: Ring's point-to-point KV rotation
+is designed to overlap with attention compute, which fits a slower
+cross-node link better than an all-to-all does.
+
+**Hardware.** 2 nodes × 8× NVIDIA H200 SXM, same cluster, InfiniBand
+between nodes.
+
+**Serve command.** The cross-node cell the picker emits for H200, run
+identically on both nodes with `--node-rank` set to 0 and 1:
+
+```bash Command
+sglang serve \
+  --model-path MiniMaxAI/MiniMax-H3 \
+  --model-variant ref2va \
+  --num-gpus 16 \
+  --nnodes 2 \
+  --node-rank {{NODE_RANK}} \
+  --dist-init-addr {{NODE0_IP}}:20000 \
+  --sp-degree 16 \
+  --ulysses-degree 8 \
+  --ring-degree 2 \
+  --encoder-parallel replicate \
+  --performance-mode speed \
+  --host 0.0.0.0 \
+  --port 30010
+```
+
+**What was measured.** A controlled denoise-stage comparison on identical
+hardware: 8× H200 single-node (Ulysses8, no Ring) versus the same 16-GPU
+cross-node command above (Ulysses8 × Ring2), holding prompt, seed, and
+step count fixed:
+
+| Task | Single-node (Ulysses8) | Cross-node (Ulysses8 × Ring2) | Change |
+| --- | ---: | ---: | ---: |
+| T2VA denoise/step | 0.749 s | 0.477 s | −36.3% |
+| Ref2VA/V2V denoise/step | 2.572 s | 1.494 s | −41.9% |
+
+The gain grows with sequence length because Ring's per-hop communication
+cost stays roughly constant while attention compute grows quadratically
+with sequence length, so V2V's longer packed sequence benefits more than
+T2VA's shorter one. With the point-to-point KV rotation pipelined against
+attention compute, one V2V request's full denoise stage completed in
+68.1–68.3 seconds versus 128.6 seconds on the single-node 8-GPU baseline
+(−47.0%), with byte-identical output to the unpipelined cross-node path.
+
+Cross-node determinism was confirmed separately: the same request run
+twice against the same cross-node deployment produced byte-identical
+output. A cross-node run's output is not expected to bit-match a
+single-node run of the same prompt and seed — Ring's online-softmax merge
+across hops accumulates floating-point operations in a different order
+than single-node attention, which is an expected source of bit-level
+difference, not a correctness regression.
+
+<Warning>
+`--encoder-parallel auto`'s fold decision is not yet node-boundary aware
+and attempts to fold the text encoder across nodes, which crashes the
+Ref2VA reference-conditioned encoder. Always pass
+`--encoder-parallel replicate` explicitly for cross-node H3 deployments.
+</Warning>
+
 ### H100 topology comparison
 
 The same four-card H100 host completed three lossless placements. TP2 with

docs/docs/sglang-diffusion/ring_sp_performance.mdx

@@ -245,6 +245,82 @@ sglang serve \
   </tbody>
 </table>
 
+## Cross-Node Sequence Parallelism
+
+Ulysses alone cannot scale sequence parallelism past the GPU count of one
+node: going wider either violates head-count divisibility or exposes an
+all-to-all across the slower inter-node link. Ring's point-to-point KV
+rotation is designed to overlap with attention compute, which tolerates a
+slower cross-node link far better than an all-to-all does — so the pattern
+for scaling SP across nodes is **node-local Ulysses × cross-node Ring**, not
+Ulysses alone.
+
+Cross-node launches add three flags on top of the usual SP degrees:
+
+- `--nnodes`: number of nodes. `--num-gpus` stays the *total* GPU count
+  across every node; each node runs `num_gpus // nnodes` local workers.
+- `--node-rank`: this node's rank, `0` on the head node (which keeps the
+  HTTP/TokenizerManager surface) and `1..nnodes-1` on the others (worker-only).
+- `--dist-init-addr`: a `host:port` rendezvous address reachable from every
+  node — typically the head node's address.
+
+Run the same command on every node, changing only `--node-rank`:
+
+```bash
+# node 0 (head)
+sglang serve \
+  --model-path MiniMaxAI/MiniMax-H3 \
+  --model-variant ref2va \
+  --num-gpus 16 \
+  --nnodes 2 \
+  --node-rank 0 \
+  --dist-init-addr <node0-ip>:23456 \
+  --sp-degree 16 \
+  --ulysses-degree 8 \
+  --ring-degree 2 \
+  --encoder-parallel replicate \
+  --port 30010
+
+# node 1 (worker)
+sglang serve \
+  --model-path MiniMaxAI/MiniMax-H3 \
+  --model-variant ref2va \
+  --num-gpus 16 \
+  --nnodes 2 \
+  --node-rank 1 \
+  --dist-init-addr <node0-ip>:23456 \
+  --sp-degree 16 \
+  --ulysses-degree 8 \
+  --ring-degree 2 \
+  --encoder-parallel replicate \
+  --port 30010
+```
+
+`--encoder-parallel replicate` is required for cross-node deployments today:
+the `auto` fold decision is not yet node-boundary aware and will try to fold
+the text encoder across nodes, which crashes reference-conditioned encoders.
+See [Encoder Parallelism](/docs/sglang-diffusion/encoder_parallel).
+
+<Warning>
+Cross-node ring support is model-specific, not a property of the launch
+flags alone — see [Supported Models and Optimization Compatibility](/docs/sglang-diffusion/compatibility_matrix)
+for which models have it. Passing `--ring-degree > 1` for a model that only
+has single-node Ulysses may either raise or, in some cases, silently
+compute incorrect output; check the model's cookbook page before assuming
+cross-node scaling is supported.
+</Warning>
+
+### Numerics across node boundaries
+
+Ring's online-softmax merge across P2P hops accumulates floating-point
+operations in a different order than single-node attention, so a cross-node
+run is **not** expected to bit-match a single-node run of the same prompt and
+seed — this is the same class of difference as choosing a different
+attention backend, not a correctness regression. What *is* expected: the same
+request run twice against the same cross-node deployment must produce
+byte-identical output. Use that repeat-request check, not a cross-topology
+comparison, to validate a cross-node deployment's determinism.
+
 ## Benchmarking Guidance
 
 When benchmarking SP, compare the same model, precision, resolution, frame count, step count, scheduler settings, prompt type, and output path. Report both stage latency and peak GPU memory; SP can reduce per-GPU memory while adding communication overhead.
@@ -375,3 +451,56 @@ The following numbers are a reference measurement for one setup. They are not a
 </table>
 
 In this setup, end-to-end latency improved from `90.63s` to `63.74s` (`1.42x`) and peak GPU memory dropped by `7.33GB`. The overhead ratio increased, so future tuning should still check communication and runtime overhead on the target hardware.
+
+## Cross-Node Reference Benchmark
+
+The following numbers are a reference measurement for MiniMax-H3's cross-node
+Ulysses × Ring deployment. They are not a general promise for every
+model or topology — see each model's cookbook page for its own verified
+cross-node status.
+
+- Model: `MiniMaxAI/MiniMax-H3`
+- Hardware: 2 nodes × 8× NVIDIA H200 SXM, same cluster, InfiniBand between nodes
+- Cross-node config: `--num-gpus 16 --sp-degree 16 --ulysses-degree 8 --ring-degree 2`
+- Single-node baseline: `--num-gpus 8 --sp-degree 8 --ulysses-degree 8 --ring-degree 1`
+
+Denoise-stage-only comparison, holding prompt, seed, and step count fixed:
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "22%"}} />
+    <col style={{width: "22%"}} />
+    <col style={{width: "22%"}} />
+  </colgroup>
+  <thead>
+    <tr>
+      <th>Task</th>
+      <th>Single-node (s/step)</th>
+      <th>Cross-node (s/step)</th>
+      <th>Change</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>T2VA denoise</td>
+      <td>0.749</td>
+      <td>0.477</td>
+      <td>-36.3%</td>
+    </tr>
+    <tr>
+      <td>Ref2VA / V2V denoise</td>
+      <td>2.572</td>
+      <td>1.494</td>
+      <td>-41.9%</td>
+    </tr>
+  </tbody>
+</table>
+
+The gain grows with sequence length: ring's per-hop communication cost stays
+roughly constant while attention compute grows quadratically with sequence
+length, so V2V's longer packed sequence benefits more than T2VA's shorter
+one. With the point-to-point KV rotation pipelined against attention compute
+(rather than a blocking `all_gather`), one V2V request's full denoise stage
+completed in 68.1-68.3s versus 128.6s on the single-node 8-GPU baseline
+(-47.0%), with byte-identical output to the unpipelined cross-node path.

docs/src/snippets/configs/MiniMaxAI/minimax-h3.jsx

@@ -44,6 +44,11 @@ export const config = {
           label: "Layerwise offload",
           showWhen: (s) => s.hw === "rtx5090",
         },
+        {
+          id: "cross_node",
+          label: "Cross-node (2 nodes)",
+          showWhen: (s) => s.hw === "h200",
+        },
       ],
     },
   ],
@@ -512,6 +517,24 @@ export const config = {
       warn:
         "FSDP reduces resident DiT memory but adds per-block parameter collectives. Prefer Resident when the full pipeline fits.",
     },
+    {
+      match: { hw: "h200", profile: "cross_node" },
+      nnodes: 2,
+      verified: true,
+      flags: [
+        "--model-path {{MODEL_NAME}}",
+        "--num-gpus 16",
+        "--sp-degree 16",
+        "--ulysses-degree 8",
+        "--ring-degree 2",
+        "--encoder-parallel replicate",
+        "--performance-mode speed",
+        "--host {{HOST_IP}}",
+        "--port {{PORT}}",
+      ],
+      warn:
+        "Verified on 2 nodes of 8× H200 each (Ulysses8 within a node, Ring2 across nodes). Requires --encoder-parallel replicate: --encoder-parallel auto's fold decision is not yet node-boundary aware and will crash across nodes.",
+    },
     {
       match: { hw: "b200", profile: "fsdp" },
       nnodes: 1,