diff --git a/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx b/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx index a1c653b897..80516c27c6 100644 --- a/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx +++ b/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx @@ -1479,7 +1479,7 @@ fold decision is not node-boundary aware: - The released visual VAE quality recipe uses overlapping tiled decode. SGLang keeps that recipe by default and distributes complete tiles across the decode group; this changes scheduling, not the computation inside each tile. - H3 rejects `--vae-config.parallel-decode-mode spatial` and `spatial_shard`: validation found output mismatches. Use the default released tiled recipe. - Keep the default `--encoder-parallel auto`. With the server’s default `batching_max_size` of 1, single-node H100/H200/B200/B300 recipes with peer-to-peer access fold the Qwen text encoder over otherwise idle Ulysses ranks. This is separate from DiT tensor parallelism. A pure-TP recipe already shards the encoder over its TP group and does not add a world fold. -- Single-node Ulysses recipes pipeline each block's all-to-all against dense attention (FlashAttention on Hopper, cuDNN SDPA on Blackwell) over the copy engine: one head group's q/k/v moves while the previous group attends, every copy (the merge into the output included) runs on the copy engine, and the output is bit-identical to the sequential exchange. It needs peer-to-peer access and stream memory operations on every rank and about 10 × (rows per rank) × heads × head_dim extra activations per rank (3.6 GB for Ref2VA at Ulysses 2); when that does not fit on some rank, every rank keeps the sequential exchange. `SGLANG_DIFFUSION_ULYSSES_PIPELINE_GROUPS=0` restores the sequential exchange. +- Single-node Ulysses recipes pipeline each block's all-to-all against dense attention (FlashAttention on Hopper, cuDNN SDPA on Blackwell) over the copy engine: one head group's q/k/v moves while the previous group attends, every copy (the merge into the output included) runs on the copy engine, and the output is bit-identical to the sequential exchange. It needs peer-to-peer access and stream memory operations on every rank and about 6 × (rows per rank) × heads × head_dim extra activations per rank (2.1 GB for Ref2VA at Ulysses 2), one set shared by every resolution; when that does not fit on some rank, every rank keeps the sequential exchange. `SGLANG_DIFFUSION_ULYSSES_PIPELINE_GROUPS=0` restores the sequential exchange. - For throughput-oriented serving, select **DP (batched throughput)**. The picker pairs `--encoder-parallel dp` with an editable `--batching-max-size` greater than 1. Encoder DP stays inside each DiT replica and composes with encoder TP: the H100 TP2 + Ulysses2 recipe has two TP-sharded encoder copies that can split a batch, while the RTX 5090 pure-TP2 recipe has one encoder copy and therefore no additional batch-DP degree. It provides no benefit for a batch of one and is not bitwise-identical to the unsplit deployment. - Use explicit **Fold** to prioritize single-request latency and encoder memory on a measured high-bandwidth single-node topology. Use **Replicate** as the compatibility path when folding or encoder DP is unsuitable. - `--use-fsdp-inference true` shards only the DiT. MiniMax-H3 preserves the original FP32 dtype of its patch, time, and output projections during FSDP all-gather, so this path does not trade numerical correctness for memory. On 4×H100, prefer TP2 + Ulysses2 for speed; use FSDP as an explicit capacity policy rather than assuming it is faster. diff --git a/docs/docs/sglang-diffusion/environment_variables.mdx b/docs/docs/sglang-diffusion/environment_variables.mdx index 67699723de..337ced9459 100644 --- a/docs/docs/sglang-diffusion/environment_variables.mdx +++ b/docs/docs/sglang-diffusion/environment_variables.mdx @@ -93,6 +93,11 @@ description: "Configure SGLang diffusion behavior with environment variables." 16 Maximum cached IPC staging-buffer shape pairs. Cap this for multi-resolution serving to bound permanent staging memory. + + SGLANG_DIFFUSION_ULYSSES_PIPELINE_GROUPS + -1 + Head groups for pipelining the Ulysses exchange against dense attention over the copy engine. -1 picks the largest count that divides the heads per rank while each group still fills the GPU; 0 keeps the sequential exchange; N forces N groups. Results are bit-identical either way. + SGLANG_USE_RUNAI_MODEL_STREAMER true diff --git a/docs/docs/sglang-diffusion/parallelism.mdx b/docs/docs/sglang-diffusion/parallelism.mdx index d08dc55dba..ad7587fd6f 100644 --- a/docs/docs/sglang-diffusion/parallelism.mdx +++ b/docs/docs/sglang-diffusion/parallelism.mdx @@ -112,6 +112,28 @@ identical to the NCCL path. | `SGLANG_DIFFUSION_IPC_A2A` | `1` | set `0` to force NCCL | | `SGLANG_DIFFUSION_IPC_A2A_TIMEOUT_MS` | `10000` | deadlock backstop for the peer wait; on expiry the transport retires on every rank and the request fails rather than returning incomplete data | | `SGLANG_DIFFUSION_IPC_A2A_MAX_BUFFERS` | `16` | staging pairs kept alive; raise for many-resolution serving | +| `SGLANG_DIFFUSION_ULYSSES_PIPELINE_GROUPS` | `-1` | head groups for the pipelined exchange below; `0` keeps the sequential exchange, `N >= 2` forces `N` groups | + +On a single host with peer-to-peer access, Ulysses attention also pipelines +the exchange against attention, one head group at a time: while group g +attends, group g + 1's q/k/v moves on the copy engine, and finished groups send +their output back while later groups still attend. Every Ulysses path of +`USPAttention` goes through it under FlashAttention or cuDNN SDPA, at any batch +size and with GQA: dense attention, key masks and tail padding, and text tokens +replicated on every rank (a prefix, a suffix, or a K/V-only prefix), which stay +out of the exchange. MiniMax-H3's attention uses the same transport. Ring +attention, uneven sequence shards (`seq_lens`) and the K/V-gather mode keep the +sequential exchange. The automatic count is the largest of 7, 5, 4, 3 and 2 +that divides the heads per rank while each group's call still fills four waves +of 128-row tiles, so short sequences such as a 1024² image keep the sequential +exchange. Its buffers are one pool per dtype that every shape shares, grown to +the largest (about 4 × rows per rank × heads × head_dim); a shape that would +not fit on every rank keeps the sequential exchange. The first call of every +shape runs both exchanges and keeps the pipeline only if they give the same +bytes on every rank, so results never change; that check runs the sequential +exchange on per-call buffers, so no staging for it stays allocated. A custom +attention module reuses it through +`ulysses_pipelined_attention(q, k, v, attend, groups, replicated=..., sequential=...)`. Independently of the transport, the default path already packs the three Q/K/V input exchanges into one destination-major collective. A handful of diff --git a/python/sglang/multimodal_gen/envs.py b/python/sglang/multimodal_gen/envs.py index 5d2089017f..d0eeaaccd2 100644 --- a/python/sglang/multimodal_gen/envs.py +++ b/python/sglang/multimodal_gen/envs.py @@ -68,10 +68,11 @@ if TYPE_CHECKING: # copy-engine all-to-all for Ulysses groups of any size on one host; off by # default while it is validated. Falls back to NCCL when unavailable. SGLANG_DIFFUSION_IPC_A2A_MULTI: bool = False - # head groups for pipelining MiniMax-H3's Ulysses exchange against dense - # attention over the copy-engine transport: -1 picks a count that divides - # the heads per rank and steps aside when its buffers do not fit, 0 or 1 - # keeps the sequential exchange, N >= 2 forces N groups + # head groups for pipelining the Ulysses exchange against dense attention + # over the copy-engine transport (MiniMax-H3, and USPAttention's plain path): + # -1 picks the most groups that divide the heads per rank while each group's + # call still fills the GPU, and steps aside when the buffers do not fit; 0 or + # 1 keeps the sequential exchange; N >= 2 forces N groups SGLANG_DIFFUSION_ULYSSES_PIPELINE_GROUPS: int = -1 # a deadlock backstop, not a per-step budget: a rank can legitimately stall # for seconds (layerwise offload, wan2.2 expert-tower swaps), and expiry now diff --git a/python/sglang/multimodal_gen/runtime/distributed/device_communicators/ipc_a2a_multi.py b/python/sglang/multimodal_gen/runtime/distributed/device_communicators/ipc_a2a_multi.py index d12d0f9f62..dd6c6d74fd 100644 --- a/python/sglang/multimodal_gen/runtime/distributed/device_communicators/ipc_a2a_multi.py +++ b/python/sglang/multimodal_gen/runtime/distributed/device_communicators/ipc_a2a_multi.py @@ -9,15 +9,19 @@ NCCL's all-to-all is an SM kernel, and next to a full-occupancy attention it slows that attention by more than the exchange costs. Per peer, a sequence counter published after the copy orders the data before -the flag, and the receiver spins on that peer's entry. Double-buffered slots -alternate per call; a slot is only rewritten two calls later, after a spin on -every peer, and each peer publishes that call only after it has consumed the -slot in stream order. +the flag, and the receiver spins on that peer's entry. The plain exchange's +double-buffered slots alternate per call; a slot is only rewritten two calls +later, after a spin on every peer, and each peer publishes that call only after +it has consumed the slot in stream order. The pipelined attention needs one set +of slots (see ``_PipelinePool``). """ +import dataclasses +import functools import logging import socket from collections import OrderedDict +from dataclasses import dataclass import torch import torch.distributed as dist @@ -32,8 +36,86 @@ logger = logging.getLogger(__name__) # Group counts tried, in order, when the caller leaves the choice to the # transport: MiniMax-H3's 56 heads give 28, 14 and 7 per rank at Ulysses 2, 4 -# and 8, and 7 groups divide all three. -_AUTO_PIPELINE_GROUPS = (7, 4, 2) +# and 8, and 7 groups divide all three; Wan's 40 and 12 heads need 5 and 3. +_AUTO_PIPELINE_GROUPS = (7, 5, 4, 3, 2) +# Each group's attention call must fill this many waves of 128-row tiles: below +# it the call's tail costs more than the overlap saves (cuDNN SDPA on B300 runs +# Wan 1.3B at Ulysses 4 in three one-head groups 44% slower than sequentially). +_MIN_GROUP_WAVES = 4 + + +@functools.lru_cache(maxsize=None) +def _sm_count(device: int) -> int: + return torch.cuda.get_device_properties(device).multi_processor_count + + +@dataclass(frozen=True) +class _PipelinePlan: + """Rows and heads of one pipelined attention call. + + Each part (q, k, v) is a ``[batch, s_local, heads, head_dim]`` sequence + shard plus ``replicated`` rows that every rank holds and that stay out of + the exchange (text tokens), attended before the gathered sequence when + ``replicated_first`` and after it otherwise. A receive slot holds one part + of one head group as attention reads it: its replicated rows and every + rank's shard in rank order. The output keeps q's rows in this rank's + layout: its replicated rows and this rank's shard. + """ + + batch: int + s_local: int + heads: tuple[int, int, int] + head_dim: int + replicated: tuple[int, int, int] + replicated_first: bool + world: int + groups: int + + def group_heads(self, part: int) -> int: + return self.heads[part] // (self.world * self.groups) + + def rows(self, part: int) -> int: + return self.replicated[part] + self.world * self.s_local + + def shard_row(self, part: int, rank: int) -> int: + return (self.replicated[part] if self.replicated_first else 0) + ( + rank * self.s_local + ) + + def replicated_row(self, part: int) -> int: + return 0 if self.replicated_first else self.world * self.s_local + + @property + def out_rows(self) -> int: + return self.replicated[0] + self.s_local + + @property + def out_shard_row(self) -> int: + return self.replicated[0] if self.replicated_first else 0 + + @property + def out_replicated_row(self) -> int: + return 0 if self.replicated_first else self.s_local + + def slot_numel(self, part: int) -> int: + return self.batch * self.rows(part) * self.group_heads(part) * self.head_dim + + def out_block_numel(self) -> int: + return self.batch * self.out_rows * self.group_heads(0) * self.head_dim + + +def _auto_pipeline_groups(plan: _PipelinePlan) -> int: + """The most head groups whose calls still fill the GPU, or 0 for none.""" + tiles = plan.batch * -(-plan.rows(0) // 128) + floor = _MIN_GROUP_WAVES * _sm_count(torch.cuda.current_device()) + for n in _AUTO_PIPELINE_GROUPS: + split = plan.world * n + if ( + all(h % split == 0 for h in plan.heads) + and plan.heads[0] // split * tiles >= floor + ): + return n + return 0 def _member_devices(group, device: int) -> list[int]: @@ -114,15 +196,17 @@ class _StreamMemOps: raise _Unsupported(f"stream memory operations unavailable ({e})") from e stream.synchronize() - def write(self, stream, flag: torch.Tensor, value: int) -> None: + def write(self, stream, flag: torch.Tensor | int, value: int) -> None: # default flags fence the stream's earlier writes (the copies) first - rc = self._write(stream.cuda_stream, flag.data_ptr(), value & 0xFFFFFFFF, 0) + ptr = flag if isinstance(flag, int) else flag.data_ptr() + rc = self._write(stream.cuda_stream, ptr, value & 0xFFFFFFFF, 0) if rc: raise RuntimeError(f"cuStreamWriteValue32 failed with CUresult {rc}") - def wait(self, stream, flag: torch.Tensor, value: int) -> None: + def wait(self, stream, flag: torch.Tensor | int, value: int) -> None: # CU_STREAM_WAIT_VALUE_GEQ compares cyclically, so counters may wrap - rc = self._wait(stream.cuda_stream, flag.data_ptr(), value & 0xFFFFFFFF, 0) + ptr = flag if isinstance(flag, int) else flag.data_ptr() + rc = self._wait(stream.cuda_stream, ptr, value & 0xFFFFFFFF, 0) if rc: raise RuntimeError(f"cuStreamWaitValue32 failed with CUresult {rc}") @@ -153,41 +237,104 @@ class _StreamMemOps: raise RuntimeError(f"cuMemcpy2DAsync failed with CUresult {rc}") -class _PipelineBuffers: - """Receive slots and flags for one pipelined-attention shape. +class _PipelinePool: + """Receive slots, output blocks and flags that every pipelined call of one + dtype shares, grown to the largest call seen rather than kept per shape. - Input slots are [2, groups, 3, world, chunk]: group g's q (k, v) rows from - every rank form one contiguous [S, group_heads, head_dim] tensor for - attention. Output slots are source-major ([2, world, groups, chunk]), each - block a peer's rows for one head group, copied into the merged output as it - arrives. + One set serves every call, with no slot per call parity. Rank r sends a + group's output blocks only after attending over its slot, and peer p + returns from a call only after merging all of r's blocks; p writes r's + slots for the next call only after that, so after r has read them. And p + writes its next block into r's output block only after attending over r's + rows of that call, which r sends only after its own previous call + returned, so after r merged p's previous block. """ - def __init__(self, state, s_local, heads, head_dim, groups, dtype): + def __init__(self, state, dtype, in_numel: int, out_numel: int, groups: int): world = state.world - self.group_heads = heads // world // groups - rows = s_local * self.group_heads - zeros = lambda *shape, dt=dtype: torch.zeros(*shape, dtype=dt, device="cuda") - self.inb = state._share( - zeros(2, groups, 3, world, rows * head_dim), state.group - ) - self.outb = state._share(zeros(2, world, groups, rows * head_dim), state.group) + self.sizes = (in_numel, out_numel, groups) + zeros = lambda n, dt=dtype: torch.zeros(n, dtype=dt, device="cuda") + self.inb = state._share(zeros(in_numel), state.group) + self.outb = state._share(zeros(out_numel), state.group) # fin[p]: groups peer p has delivered; fout[g, p]: calls whose group g # peer p has delivered. Groups finish on separate streams, so a single # output counter could be lowered by a signal that lands late. self.fin = state._share(zeros(world, dt=torch.int32), state.group) - self.fout = state._share(zeros(groups, world, dt=torch.int32), state.group) - # the caller's q/k for each peer block; v moves straight from its source - self.send = torch.empty( - world * groups, - 2, - s_local, - self.group_heads, - head_dim, - dtype=dtype, - device="cuda", - ) + self.fout = state._share(zeros(groups * world, dt=torch.int32), state.group) + self.inb_ptr = [t.data_ptr() for t in self.inb] + self.outb_ptr = [t.data_ptr() for t in self.outb] + self.fin_ptr = [t.data_ptr() for t in self.fin] + self.fout_ptr = [t.data_ptr() for t in self.fout] self.calls = 0 + self.groups_sent = 0 + # the caller's q/k for each peer block when a fill writes them; v moves + # straight from its source + self.send = None + + def nbytes(self, esz: int) -> int: + return (self.sizes[0] + self.sizes[1]) * esz + + +class _PipelineSlots: + """Where one plan's receive slots and output blocks sit in the pool. + + Receive slots, per head group and part: one contiguous ``[batch, rows, + group_heads, head_dim]`` tensor laid out as attention reads it (see + ``_PipelinePlan``). Output blocks are source-major: block ``(source, + group)`` is that rank's output rows for this rank, one ``[batch, out_rows, + group_heads, head_dim]`` block in the output's row order, copied into the + merged output as it arrives. + """ + + def __init__(self, plan: _PipelinePlan, dtype): + self.plan = plan + self.dtype = dtype + numel = [plan.slot_numel(part) for part in range(3)] + self.part_numel = numel + self.part_offset = (0, numel[0], numel[0] + numel[1]) + self.group_numel = sum(numel) + self.out_numel = plan.out_block_numel() + self.in_total = plan.groups * self.group_numel + self.out_total = plan.world * plan.groups * self.out_numel + # raw addresses: the copies are issued per call from plain integers, + # since a tensor view per copy costs more host time than the copy + self.esz = torch.empty((), dtype=dtype).element_size() + self.rows = [plan.rows(part) for part in range(3)] + self.row_bytes = [ + plan.group_heads(part) * plan.head_dim * self.esz for part in range(3) + ] + self.send_numel = 2 * plan.s_local * plan.group_heads(0) * plan.head_dim + + def slot_ptr(self, pool, member, group, part, batch_row, row) -> int: + elems = group * self.group_numel + self.part_offset[part] + return ( + pool.inb_ptr[member] + + elems * self.esz + + (batch_row * self.rows[part] + row) * self.row_bytes[part] + ) + + def out_block_ptr(self, pool, member, source, group) -> int: + block = (source * self.plan.groups + group) * self.out_numel + return pool.outb_ptr[member] + block * self.esz + + def slot(self, pool, member: int, group: int, part: int) -> torch.Tensor: + """``member``'s receive slot, mapped in this rank's context.""" + plan = self.plan + start = group * self.group_numel + self.part_offset[part] + return ( + pool.inb[member] + .narrow(0, start, self.part_numel[part]) + .view(plan.batch, plan.rows(part), plan.group_heads(part), plan.head_dim) + ) + + def send_block(self, pool, peer: int, group: int) -> torch.Tensor: + plan = self.plan + need = plan.world * plan.groups * self.send_numel + if pool.send is None or pool.send.numel() < need: + pool.send = torch.empty(need, dtype=self.dtype, device="cuda") + return pool.send.narrow( + 0, (peer * plan.groups + group) * self.send_numel, self.send_numel + ).view(2, plan.s_local, plan.group_heads(0), plan.head_dim) class IpcA2AMultiState: @@ -213,17 +360,29 @@ class IpcA2AMultiState: self.pipe_group_streams = [] # pipeline shapes whose buffers did not fit on some rank; agreed by all self.declined = set() + # (shape, caller signature) checked against the sequential exchange, + # by outcome; agreed by all + self.verified = set() + self.mismatched = set() + # call shapes -> the plan they resolve to, None when they cannot + # pipeline; repeated calls skip straight to the answer + self.plans = {} + # dtype -> the _PipelinePool every pipelined call of that dtype uses + self.pools = {} + # ("pipeline", plan, dtype) -> its _PipelineSlots + self.slots = {} def reset(self) -> None: """Drop mappings that belong to a model-parallel group being replaced.""" self.__init__() def drop_staging(self) -> None: - if not self.staging: + if not self.staging and not self.pools: return if torch.cuda.is_available(): torch.cuda.synchronize() self.staging.clear() + self.pools.clear() def _share(self, t: torch.Tensor, group) -> list[torch.Tensor]: """Every member's `t`, peers' re-opened in the LOCAL device context @@ -314,53 +473,112 @@ class IpcA2AMultiState: self.staging.popitem(last=False) self.staging[key] = value - def pipelined_attention(self, q, k, v, attend, groups: int, fill=None): + def pipelined_attention( + self, + q, + k, + v, + attend, + groups: int, + fill=None, + sequential=None, + signature=None, + replicated=(None, None, None), + replicated_first: bool = True, + ): """Ulysses attention with the head exchange pipelined against it. - q/k/v are [s_local, heads, head_dim], sequence-sharded with every head. - ``attend(q, k, v)`` runs attention on [S, heads / (world * groups), - head_dim] tensors of one head group and must treat heads independently, - which keeps the result bit-identical to the sequential exchange. - Group g's copies run on the copy engine while group g-1 attends, and - each group attends on its own stream and sends its output from there, - so no stream waits on a kernel still running on another. Every copy, - the merge into the output included, is a pitched copy-engine copy, so - the exchange takes no SMs. Returns [s_local, heads, head_dim], or None - when this call cannot pipeline. ``groups < 0`` picks the count and - steps aside, on every rank, when the buffers would not fit. + q/k/v are ``[batch, s_local, heads, head_dim]`` sequence shards with + every head (``[s_local, heads, head_dim]`` reads as batch 1 and returns + that shape); k and v may have fewer heads than q (GQA). + ``replicated`` optionally holds, per part, ``[batch, n, heads, + head_dim]`` rows every rank holds (text tokens); they stay out of the + exchange, attend before the gathered sequence when + ``replicated_first`` and after it otherwise, and their output, all + heads, leads (or trails) the returned rows. - ``fill(head_start, head_count, q_dst, k_dst)``, when given, writes those - heads' q and k into two [s_local, head_count, head_dim] blocks in place - of a copy (the caller's QK-norm writes its output there); blocks for - this rank go straight into its own receive slot, and v always moves - straight from ``v``. It is only called once the call is known to - pipeline. + ``attend(q, k, v)`` runs attention on one head group, ``[batch, rows, + heads / (world * groups), head_dim]`` per part in the sequence order + above, and must treat heads independently, which keeps the result + bit-identical to the sequential exchange. Group g's copies run on the + copy engine while group g-1 attends, and each group attends on its own + stream and sends its output from there, so no stream waits on a kernel + still running on another. Every copy, the merge into the output + included, is a pitched copy-engine copy, so the exchange takes no SMs. + Returns the output rows of this rank, or None when this call cannot + pipeline. ``groups < 0`` picks the count and steps aside, on every + rank, when the buffers would not fit. + + ``fill(head_start, head_count, q_dst, k_dst)``, when given (batch 1, + no replicated rows, as many k as q heads), writes those heads' q and k + into two [s_local, head_count, head_dim] blocks in place of a copy + (the caller's QK-norm writes its output there); blocks for this rank + go straight into its own receive slot, and v always moves straight + from ``v``. It is only called once the call is known to pipeline. + + ``sequential()``, when given, returns what the sequential exchange + gives for this call. The first call per plan and ``signature`` (the + caller's attention setup) runs both and keeps the pipeline for them + only if every rank sees the same bytes, so a backend that does not + treat a head group as it treats those heads in the full call never + changes a result. It runs the sequential exchange first, with the + pool released, so that call never holds both at once. """ world, r = self.world, self.rank - s_local, heads, head_dim = q.shape + squeeze = q.dim() == 3 + if squeeze: + q, k, v = q[None], k[None], v[None] + parts = (q, k, v) + replicated = tuple(replicated) + batch, s_local, _, head_dim = q.shape auto = groups < 0 - if auto: - groups = next( - (n for n in _AUTO_PIPELINE_GROUPS if heads % (world * n) == 0), 0 + shapes = ( + q.shape, + k.shape, + v.shape, + tuple(None if t is None else t.shape for t in replicated), + replicated_first, + groups, + fill is not None, + ) + if shapes not in self.plans: + self.plans[shapes] = self._plan( + q, k, v, replicated, replicated_first, groups, fill is not None ) + plan = self.plans[shapes] + if plan is None: + return None + groups = plan.groups + # each token's heads of one group must be one contiguous run to move # with a pitched copy - copied = (v,) if fill is not None else (q, k, v) - if ( - not groups - or heads % (world * groups) - or not (q.shape == k.shape == v.shape) - or any(t.stride(2) != 1 or t.stride(1) != head_dim for t in copied) - ): + copied = [v] if fill is not None else [q, k, v] + copied += [t for t in replicated if t is not None] + if not all(t.stride(-1) == 1 and t.stride(-2) == head_dim for t in copied): return None - key = ("pipeline", s_local, heads, head_dim, groups, q.dtype) - bufs = self.staging.get(key) - if bufs is None: + key = ("pipeline", plan, q.dtype) + check = None if sequential is None else (key, signature) + if check in self.mismatched: + return None + if check in self.verified: + check = None + slots = self.slots.get(key) + if slots is None: + slots = self.slots[key] = _PipelineSlots(plan, q.dtype) + pool = self.pools.get(q.dtype) + sizes = (slots.in_total, slots.out_total, groups) + if pool is not None: + sizes = tuple(map(max, sizes, pool.sizes)) + reference = None + if pool is None or pool.sizes != sizes or check is not None: if torch.cuda.is_current_stream_capturing() or key in self.declined: return None - # send + both receive and output slots: 10 * s_local * heads * head_dim - need = 10 * s_local * heads * head_dim * q.element_size() - if auto and not self._fits_on_every_rank(need): + esz = q.element_size() + held = 0 if pool is None else pool.nbytes(esz) + need = (sizes[0] + sizes[1]) * esz + if fill is not None: + need += plan.world * groups * slots.send_numel * esz + if auto and not self._fits_on_every_rank(need, reclaimable=held): logger.info( "Ulysses pipeline buffers (%.1f GiB) do not fit; using the " "sequential exchange for this shape", @@ -368,82 +586,137 @@ class IpcA2AMultiState: ) self.declined.add(key) return None - bufs = _PipelineBuffers(self, s_local, heads, head_dim, groups, q.dtype) - self._insert(key, bufs) + if pool is not None: + # queued copies may still target the old mappings + torch.cuda.synchronize() + del self.pools[q.dtype] + pool = None + if check is not None: + reference = sequential() + pool = self.pools[q.dtype] = _PipelinePool(self, q.dtype, *sizes) if self.pipe_in is None: self.pipe_in = torch.cuda.Stream() while len(self.pipe_group_streams) < groups: self.pipe_group_streams.append(torch.cuda.Stream()) - mem, cin, hg = self.memops, self.pipe_in, bufs.group_heads + mem, cin = self.memops, self.pipe_in main = torch.cuda.current_stream() - slot = bufs.calls % 2 - base = bufs.calls * groups - bufs.calls += 1 - call = bufs.calls + base = pool.groups_sent + pool.groups_sent += groups + pool.calls += 1 + call = pool.calls peers = [(r + step) % world for step in range(1, world)] esz = q.element_size() - width = hg * head_dim * esz # one token's heads of one group - chunk = s_local * width # one rank's rows of one group, one of q/k/v - start = lambda p, g: (p * groups + g) * hg # first head of block (p, g) - slot_of = lambda p, g, part: bufs.inb[p][slot, g, part, r] # my rows there + head_bytes = head_dim * esz + row_bytes = slots.row_bytes + # first head of block (p, g) of a part + start = lambda p, g, part: (p * groups + g) * plan.group_heads(part) + # (address, batch pitch, token pitch) of each part's shard and replicated rows + src = [(t.data_ptr(), t.stride(0) * esz, t.stride(1) * esz) for t in parts] + rep_src = [ + None if t is None else (t.data_ptr(), t.stride(0) * esz, t.stride(1) * esz) + for t in replicated + ] + my_shard_row = [plan.shard_row(part, r) for part in range(3)] - def send_rows(stream, t, p, g, part): - src = t[:, start(p, g)] - mem.copy2d( - stream, - slot_of(p, g, part).data_ptr(), - width, - src.data_ptr(), - t.stride(0) * esz, - width, - s_local, - ) + def send_rows(stream, part, p, g): + # my shard of one part and head group into rank p's receive slot + ptr, batch_pitch, token_pitch = src[part] + ptr += start(p, g, part) * head_bytes + for b in range(batch): + mem.copy2d( + stream, + slots.slot_ptr(pool, p, g, part, b, my_shard_row[part]), + row_bytes[part], + ptr + b * batch_pitch, + token_pitch, + row_bytes[part], + s_local, + ) + + def place_replicated(stream, part, g): + # rows every rank holds go straight into my own slot + n = plan.replicated[part] + if not n: + return + ptr, batch_pitch, token_pitch = rep_src[part] + ptr += start(r, g, part) * head_bytes + row = plan.replicated_row(part) + for b in range(batch): + mem.copy2d( + stream, + slots.slot_ptr(pool, r, g, part, b, row), + row_bytes[part], + ptr + b * batch_pitch, + token_pitch, + row_bytes[part], + n, + ) ready = main.record_event() own, sent = [], [] if fill is not None: # group by group: the peers' blocks first, so their copies start # while this rank fills its own block straight into its slot - block = lambda t: t.view(s_local, hg, head_dim) + own_rows = lambda part, g: slots.slot(pool, r, g, part)[ + 0, my_shard_row[part] : my_shard_row[part] + s_local + ] + hg = plan.group_heads(0) for g in range(groups): for p in peers: - qk = bufs.send[p * groups + g] - fill(start(p, g), hg, qk[0], qk[1]) + qk = slots.send_block(pool, p, g) + fill(start(p, g, 0), hg, qk[0], qk[1]) sent.append(main.record_event()) - fill(start(r, g), hg, block(slot_of(r, g, 0)), block(slot_of(r, g, 1))) + fill(start(r, g, 0), hg, own_rows(0, g), own_rows(1, g)) own.append(main.record_event()) own_cin = [] cin.wait_event(ready) with torch.cuda.stream(cin): for g in range(groups): for p in (r, *peers): - send_rows(cin, v, p, g, 2) + send_rows(cin, 2, p, g) if fill is None: - send_rows(cin, q, p, g, 0) - send_rows(cin, k, p, g, 1) + send_rows(cin, 0, p, g) + send_rows(cin, 1, p, g) if p == r: + for part in range(3): + place_replicated(cin, part, g) own_cin.append(cin.record_event()) if fill is not None: cin.wait_event(sent[g]) + chunk = s_local * row_bytes[0] for p in peers: + q_slot = slots.slot_ptr(pool, p, g, 0, 0, my_shard_row[0]) + k_slot = slots.slot_ptr(pool, p, g, 1, 0, my_shard_row[1]) # q then k: two rows of `chunk`, one part apart in the slot mem.copy2d( cin, - slot_of(p, g, 0).data_ptr(), - slot_of(p, g, 1).data_ptr() - slot_of(p, g, 0).data_ptr(), - bufs.send[p * groups + g].data_ptr(), + q_slot, + k_slot - q_slot, + slots.send_block(pool, p, g).data_ptr(), chunk, chunk, 2, ) for p in peers: - mem.write(cin, bufs.fin[p].narrow(0, r, 1), base + g + 1) + mem.write(cin, pool.fin_ptr[p] + 4 * r, base + g + 1) # Head block b = p * groups + g of the output is group g of rank p's # heads. Each group stream writes its blocks as soon as they exist, so # the merge overlaps the groups still attending instead of trailing them. - merged = torch.empty(s_local, heads, head_dim, dtype=q.dtype, device=q.device) - merged_blocks = merged.view(s_local, world * groups, hg, head_dim) - merged_pitch = heads * head_dim * esz + hq = plan.group_heads(0) + merged = torch.empty( + batch, + plan.out_rows, + plan.heads[0], + head_dim, + dtype=q.dtype, + device=q.device, + ) + merged_ptr = merged.data_ptr() + merged_row = plan.heads[0] * head_bytes + out_row = row_bytes[0] + block_batch = plan.out_rows * out_row + shard_out, rep_out = plan.out_shard_row, plan.out_replicated_row + n_rep, rep_row = plan.replicated[0], plan.replicated_row(0) done = [] for g in range(groups): stream = self.pipe_group_streams[g] @@ -452,37 +725,60 @@ class IpcA2AMultiState: stream.wait_event(own_cin[g]) with torch.cuda.stream(stream): for p in peers: - mem.wait(stream, bufs.fin[r].narrow(0, p, 1), base + g + 1) - part = lambda i: bufs.inb[r][slot, g, i].view( - world * s_local, hg, head_dim - ) - out = attend(part(0), part(1), part(2)).contiguous() - rows = lambda p: out[p * s_local : (p + 1) * s_local] + mem.wait(stream, pool.fin_ptr[r] + 4 * p, base + g + 1) + views = [slots.slot(pool, r, g, part) for part in range(3)] + if squeeze: + out = attend(*(t[0] for t in views)).contiguous()[None] + else: + out = attend(*views).contiguous() + out_ptr = out.data_ptr() + out_batch = out.stride(0) * esz for p in peers: - bufs.outb[p][slot, r, g].copy_( - rows(p).reshape(-1), non_blocking=True - ) + # this peer's rows and the replicated rows every rank needs, + # one contiguous run per batch row + block = slots.out_block_ptr(pool, p, r, g) + runs = [(shard_out, plan.shard_row(0, p), s_local)] + if n_rep: + runs.append((rep_out, rep_row, n_rep)) + for dst_row, src_row, n in runs: + mem.copy2d( + stream, + block + dst_row * out_row, + block_batch, + out_ptr + src_row * out_row, + out_batch, + n * out_row, + batch, + ) for p in peers: - mem.write(stream, bufs.fout[p][g].narrow(0, r, 1), call) - mem.copy2d( - stream, - merged_blocks[:, r * groups + g].data_ptr(), - merged_pitch, - rows(r).data_ptr(), - width, - width, - s_local, - ) + mem.write(stream, pool.fout_ptr[p] + 4 * (g * world + r), call) + own_head = merged_ptr + start(r, g, 0) * head_bytes + for b in range(batch): + runs = [(shard_out, my_shard_row[0], s_local)] + if n_rep: + runs.append((rep_out, rep_row, n_rep)) + for dst_row, src_row, n in runs: + mem.copy2d( + stream, + own_head + (b * plan.out_rows + dst_row) * merged_row, + merged_row, + out_ptr + b * out_batch + src_row * out_row, + out_row, + out_row, + n, + ) for p in peers: - mem.wait(stream, bufs.fout[r][g].narrow(0, p, 1), call) + mem.wait(stream, pool.fout_ptr[r] + 4 * (g * world + p), call) + # the block and the merged output both have one row pitch + # across the batch, so a single copy covers every batch row mem.copy2d( stream, - merged_blocks[:, p * groups + g].data_ptr(), - merged_pitch, - bufs.outb[r][slot, p, g].data_ptr(), - width, - width, - s_local, + merged_ptr + start(p, g, 0) * head_bytes, + merged_row, + slots.out_block_ptr(pool, r, p, g), + out_row, + out_row, + batch * plan.out_rows, ) done.append(stream.record_event()) for event in done: @@ -490,15 +786,78 @@ class IpcA2AMultiState: # the next call's fill rewrites `send` and the caller may free `v`, both # read by the copy stream main.wait_stream(cin) + if squeeze: + merged = merged[0] + if check is not None: + return self._first_sight(check, merged, reference) return merged - def _fits_on_every_rank(self, nbytes: int) -> bool: + def _plan(self, q, k, v, replicated, replicated_first, groups, filled): + """The plan these shapes pipeline with, or None; rank-independent.""" + world = self.world + batch, s_local, _, head_dim = q.shape + parts = (q, k, v) + rows = tuple(0 if t is None else t.shape[1] for t in replicated) + if ( + k.shape != v.shape + or any( + t.shape[:2] != (batch, s_local) or t.shape[3] != head_dim for t in parts + ) + or rows[1] != rows[2] + or any( + t is not None and t.shape != (batch, n, parts[i].shape[2], head_dim) + for i, (t, n) in enumerate(zip(replicated, rows)) + ) + or (filled and (batch != 1 or any(rows) or q.shape != k.shape)) + ): + return None + plan = _PipelinePlan( + batch=batch, + s_local=s_local, + heads=tuple(t.shape[2] for t in parts), + head_dim=head_dim, + replicated=rows, + replicated_first=replicated_first, + world=world, + groups=groups, + ) + if groups < 0: + plan = dataclasses.replace(plan, groups=_auto_pipeline_groups(plan)) + if not plan.groups or any(h % (world * plan.groups) for h in plan.heads): + return None + return plan + + def _first_sight(self, check, merged, reference): + """Keep the pipeline for `check` only if it matches the sequential + exchange's `reference` byte for byte on every rank; a group collective.""" + same = torch.tensor( + [int(torch.equal(merged, reference))], dtype=torch.int32, device="cuda" + ) + dist.all_reduce(same, op=dist.ReduceOp.MIN, group=self.group) + if same.item(): + self.verified.add(check) + return merged + self.mismatched.add(check) + plan = check[0][1] + logger.info( + "Ulysses pipeline over %d head groups differs from the sequential " + "exchange for %s under %s; keeping the sequential exchange", + plan.groups, + plan, + check[1], + ) + return reference + + def _fits_on_every_rank(self, nbytes: int, reclaimable: int = 0) -> bool: """Whether every rank can spare twice `nbytes`, counting the allocator's - cached blocks; a group collective, so all ranks reach the same answer.""" + cached blocks and `reclaimable` bytes it is about to free; a group + collective, so all ranks reach the same answer.""" free, _ = torch.cuda.mem_get_info() cached = torch.cuda.memory_reserved() - torch.cuda.memory_allocated() fits = torch.tensor( - [int(free + cached >= 2 * nbytes)], dtype=torch.int32, device="cuda" + [int(free + cached + reclaimable >= 2 * nbytes)], + dtype=torch.int32, + device="cuda", ) dist.all_reduce(fits, op=dist.ReduceOp.MIN, group=self.group) return bool(fits.item()) @@ -598,12 +957,35 @@ def ipc_a2a_multi_ready(group, enabled: bool | None = None) -> bool: return True -def ulysses_pipelined_attention(q, k, v, attend, groups: int, fill=None): +def ulysses_pipelined_attention( + q, + k, + v, + attend, + groups: int, + fill=None, + sequential=None, + signature=None, + replicated=(None, None, None), + replicated_first: bool = True, +): """Pipelined Ulysses attention over the copy-engine transport, or None - when the Ulysses group cannot use it (the caller keeps its NCCL path).""" + when the Ulysses group cannot use it (the caller keeps its NCCL path). + See ``IpcA2AMultiState.pipelined_attention`` for the arguments.""" from sglang.multimodal_gen.runtime.distributed.parallel_state import get_sp_group group = get_sp_group().ulysses_group if group is None or not ipc_a2a_multi_ready(group, enabled=True): return None - return IPC_A2A_MULTI.pipelined_attention(q, k, v, attend, groups, fill=fill) + return IPC_A2A_MULTI.pipelined_attention( + q, + k, + v, + attend, + groups, + fill=fill, + sequential=sequential, + signature=signature, + replicated=replicated, + replicated_first=replicated_first, + ) diff --git a/python/sglang/multimodal_gen/runtime/layers/attention/layer.py b/python/sglang/multimodal_gen/runtime/layers/attention/layer.py index 9b3c871dfd..b26440e2c7 100644 --- a/python/sglang/multimodal_gen/runtime/layers/attention/layer.py +++ b/python/sglang/multimodal_gen/runtime/layers/attention/layer.py @@ -68,6 +68,7 @@ from sglang.multimodal_gen.runtime.layers.usp import ( _usp_output_all_to_all, _usp_output_all_to_all_varlen, ring_attn, + uncached_a2a_staging, ) from sglang.multimodal_gen.runtime.managers.forward_context import ( ForwardContext, @@ -95,6 +96,14 @@ _VARLEN_FA_ENABLED = os.environ.get("SGLANG_VARLEN_FA", "1") != "0" # and run dense attention on the padded layout. _SP_PAD_MASK_DISABLED = envs.SGLANG_DIFFUSION_DISABLE_SP_PAD_MASK +# Dense backends the head-group pipeline runs against: each computes a head +# independently, so a head group comes out as those heads do in the full call +# (FlashAttention on Hopper, cuDNN SDPA on Blackwell). +PIPELINED_ATTENTION_BACKENDS = ( + AttentionBackendEnum.FA, + AttentionBackendEnum.DYNAMIC_CUDNN_SDPA, +) + def _resolve_sp_attention_mode( *, causal: bool, sparse_backend: bool @@ -890,6 +899,7 @@ class USPAttention(nn.Module): self.skip_sequence_parallel = skip_sequence_parallel self.enable_packed_qkv_input_a2a = bool(enable_packed_qkv_input_a2a) + self._pipeline_declined: set = set() self.sp_attention_mode, self.sp_attention_mode_is_auto = ( _resolve_sp_attention_mode( causal=causal, sparse_backend=self.backend.is_sparse @@ -1259,14 +1269,10 @@ class USPAttention(nn.Module): ) sp_size = get_ulysses_parallel_world_size() - if sp_size > 1 and not qkv_pre_all_to_all: - qkv_fast = _ipc_input_a2a_qkv(q, k, v) - if qkv_fast is not None: - q, k, v = qkv_fast - else: - q, k, v = _usp_input_all_to_all_qkv(q, k, v) - - if ( + exchange = sp_size > 1 and not qkv_pre_all_to_all + # the gathered sequence the attention below runs over + gathered_seq = q.shape[1] * (sp_size if exchange else 1) + tail_fa = ( _VARLEN_FA_ENABLED and self.backend == AttentionBackendEnum.FA and meta_pad_start is not None @@ -1274,109 +1280,121 @@ class USPAttention(nn.Module): and meta_pad_end > meta_pad_start and q.device.type == "cuda" and q.dtype in (torch.float16, torch.bfloat16) - ): - bs, seq = q.shape[0], q.shape[1] - assert 0 <= meta_pad_start < meta_pad_end <= seq - cu_tail = attn_mask_meta.get("cu_seqlens_tail") - if cu_tail is not None and meta_pad_end == seq: - # Zero-copy tail path: run varlen FA straight over the - # padded layout, each row split into [valid | pad] segments - # (contiguous reshapes only, no repacking). - assert cu_tail.numel() == 2 * bs + 1, ( - "cu_seqlens_tail does not match the batch size" + ) + gathered_mask = gathered_mask_meta = None + if not tail_fa: + # If NCCL timeout/deadlock occurs here, check whether + # attn_mask is inconsistent across SP ranks (None on some, + # Tensor on others), which causes all_gather participant + # mismatch. Upstream mask builders must ensure all ranks + # produce the same mask type. + if attn_mask is None: + # Meta-only tail-pad caller on a non-FA fallback: the + # gathered mask is fully determined by the pad span, no + # collective needed. + gathered_mask = torch.ones( + q.shape[0], gathered_seq, dtype=torch.bool, device=q.device ) - out = flash_attn_varlen_func( - q=q.reshape(bs * seq, *q.shape[2:]), - k=k.reshape(bs * seq, *k.shape[2:]), - v=v.reshape(bs * seq, *v.shape[2:]), - cu_seqlens_q=cu_tail, - cu_seqlens_k=cu_tail, - max_seqlen_q=attn_mask_meta["max_seqlen_tail"], - max_seqlen_k=attn_mask_meta["max_seqlen_tail"], + gathered_mask[:, meta_pad_start:meta_pad_end] = False + else: + gathered_mask = sequence_model_parallel_all_gather( + attn_mask.contiguous(), dim=1 + ) + gathered_rows = (q.shape[0], gathered_seq) + if ( + _VARLEN_FA_ENABLED + and self.backend == AttentionBackendEnum.FA + and gathered_mask.dtype + in (torch.bool, torch.uint8, torch.int32, torch.int64) + and q.device.type == "cuda" + and gathered_mask.device == q.device + and q.dtype in (torch.float16, torch.bfloat16) + and gathered_rows == gathered_mask.shape + and k.shape[:2] == v.shape[:2] == q.shape[:2] + ): + gathered_mask_meta = build_varlen_mask_meta(gathered_mask) + assert gathered_mask_meta["inv_indices"].shape[0] == ( + gathered_rows[0] * gathered_rows[1] + ), "gathered attn_mask shape does not match q/k/v" + if gathered_mask_meta["indices"].shape[0] == 0: + gathered_mask_meta = None + + def masked_attention(q, k, v): + """Attention over the gathered sequence; heads are independent.""" + bs, seq = q.shape[0], q.shape[1] + if tail_fa: + assert 0 <= meta_pad_start < meta_pad_end <= seq + cu_tail = attn_mask_meta.get("cu_seqlens_tail") + if cu_tail is not None and meta_pad_end == seq: + # Zero-copy tail path: run varlen FA straight over the + # padded layout, each row split into [valid | pad] + # segments (contiguous reshapes only, no repacking). + assert cu_tail.numel() == 2 * bs + 1, ( + "cu_seqlens_tail does not match the batch size" + ) + out = flash_attn_varlen_func( + q=q.reshape(bs * seq, *q.shape[2:]), + k=k.reshape(bs * seq, *k.shape[2:]), + v=v.reshape(bs * seq, *v.shape[2:]), + cu_seqlens_q=cu_tail, + cu_seqlens_k=cu_tail, + max_seqlen_q=attn_mask_meta["max_seqlen_tail"], + max_seqlen_k=attn_mask_meta["max_seqlen_tail"], + softmax_scale=self.softmax_scale, + causal=False, + ver=_fa_backend.fa_ver, + ).reshape(bs, seq, *q.shape[2:]) + # Match the packed paths: masked query rows read as zeros. + out[:, meta_pad_start:].zero_() + return out + valid_seq = seq - (meta_pad_end - meta_pad_start) + q_dense = torch.cat( + [q[:, :meta_pad_start], q[:, meta_pad_end:]], dim=1 + ) + k_dense = torch.cat( + [k[:, :meta_pad_start], k[:, meta_pad_end:]], dim=1 + ) + v_dense = torch.cat( + [v[:, :meta_pad_start], v[:, meta_pad_end:]], dim=1 + ) + cu_seqlens = torch.arange( + 0, + (bs + 1) * valid_seq, + valid_seq, + dtype=torch.int32, + device=q.device, + ) + out_dense = flash_attn_varlen_func( + q=q_dense.reshape(bs * valid_seq, *q.shape[2:]), + k=k_dense.reshape(bs * valid_seq, *k.shape[2:]), + v=v_dense.reshape(bs * valid_seq, *v.shape[2:]), + cu_seqlens_q=cu_seqlens, + cu_seqlens_k=cu_seqlens, + max_seqlen_q=valid_seq, + max_seqlen_k=valid_seq, softmax_scale=self.softmax_scale, causal=False, ver=_fa_backend.fa_ver, - ).reshape(bs, seq, *q.shape[2:]) - # Match the packed paths: masked query rows read as zeros. - out[:, meta_pad_start:].zero_() - if sp_size > 1: - out = _usp_output_all_to_all(out, head_dim=2) - return out - valid_seq = seq - (meta_pad_end - meta_pad_start) - q_dense = torch.cat([q[:, :meta_pad_start], q[:, meta_pad_end:]], dim=1) - k_dense = torch.cat([k[:, :meta_pad_start], k[:, meta_pad_end:]], dim=1) - v_dense = torch.cat([v[:, :meta_pad_start], v[:, meta_pad_end:]], dim=1) - cu_seqlens = torch.arange( - 0, - (bs + 1) * valid_seq, - valid_seq, - dtype=torch.int32, - device=q.device, - ) - out_dense = flash_attn_varlen_func( - q=q_dense.reshape(bs * valid_seq, *q.shape[2:]), - k=k_dense.reshape(bs * valid_seq, *k.shape[2:]), - v=v_dense.reshape(bs * valid_seq, *v.shape[2:]), - cu_seqlens_q=cu_seqlens, - cu_seqlens_k=cu_seqlens, - max_seqlen_q=valid_seq, - max_seqlen_k=valid_seq, - softmax_scale=self.softmax_scale, - causal=False, - ver=_fa_backend.fa_ver, - ).reshape(bs, valid_seq, *q.shape[2:]) - gap_out = out_dense.new_zeros( - bs, - meta_pad_end - meta_pad_start, - out_dense.shape[2], - out_dense.shape[3], - ) - out = torch.cat( - [ - out_dense[:, :meta_pad_start], - gap_out, - out_dense[:, meta_pad_start:], - ], - dim=1, - ) - if sp_size > 1: - out = _usp_output_all_to_all(out, head_dim=2) - return out + ).reshape(bs, valid_seq, *q.shape[2:]) + gap_out = out_dense.new_zeros( + bs, + meta_pad_end - meta_pad_start, + out_dense.shape[2], + out_dense.shape[3], + ) + return torch.cat( + [ + out_dense[:, :meta_pad_start], + gap_out, + out_dense[:, meta_pad_start:], + ], + dim=1, + ) - # If NCCL timeout/deadlock occurs here, check whether - # attn_mask is inconsistent across SP ranks (None on some, Tensor on - # others), which causes all_gather participant mismatch. Upstream - # mask builders must ensure all ranks produce the same mask type. - if attn_mask is None: - # Meta-only tail-pad caller on a non-FA fallback: the gathered - # mask is fully determined by the pad span, no collective needed. - gathered_mask = torch.ones( - q.shape[0], q.shape[1], dtype=torch.bool, device=q.device - ) - gathered_mask[:, meta_pad_start:meta_pad_end] = False - else: - gathered_mask = sequence_model_parallel_all_gather( - attn_mask.contiguous(), dim=1 - ) - if ( - _VARLEN_FA_ENABLED - and self.backend == AttentionBackendEnum.FA - and gathered_mask.dtype - in (torch.bool, torch.uint8, torch.int32, torch.int64) - and q.device.type == "cuda" - and gathered_mask.device == q.device - and q.dtype in (torch.float16, torch.bfloat16) - and q.shape[:2] == gathered_mask.shape == k.shape[:2] == v.shape[:2] - ): - bs, seq = q.shape[0], q.shape[1] - gathered_mask_meta = build_varlen_mask_meta(gathered_mask) - indices = gathered_mask_meta["indices"] - inv_indices = gathered_mask_meta["inv_indices"] - assert inv_indices.shape[0] == bs * seq, ( - "gathered attn_mask shape does not match q/k/v" - ) - if indices.shape[0] > 0: - q_unpad, k_unpad, v_unpad = fused_pack_qkv(q, k, v, indices) + if gathered_mask_meta is not None: + q_unpad, k_unpad, v_unpad = fused_pack_qkv( + q, k, v, gathered_mask_meta["indices"] + ) out_unpad = flash_attn_varlen_func( q=q_unpad, k=k_unpad, @@ -1389,33 +1407,57 @@ class USPAttention(nn.Module): causal=False, ver=_fa_backend.fa_ver, ) - out = fused_scatter_to_padded(out_unpad, inv_indices, bs, seq) - if sp_size > 1: - out = _usp_output_all_to_all(out, head_dim=2) - return out + return fused_scatter_to_padded( + out_unpad, gathered_mask_meta["inv_indices"], bs, seq + ) - q_ = q.transpose(1, 2) - k_ = k.transpose(1, 2) - v_ = v.transpose(1, 2) - mask = _prepare_sdpa_mask(gathered_mask, dtype=q_.dtype, device=q_.device) - sdpa_context = ( - sdpa_kernel(_PYTORCH_DEFAULT_CUDA_SDP_BACKENDS, set_priority=True) - if self.allow_cudnn_sdp and q_.device.type == "cuda" - else nullcontext() - ) - with sdpa_context: - out = torch.nn.functional.scaled_dot_product_attention( - q_, - k_, - v_, - attn_mask=mask, - dropout_p=0.0, - is_causal=False, - scale=self.softmax_scale, - ).transpose(1, 2) - if sp_size > 1: - out = _usp_output_all_to_all(out, head_dim=2) - return out + q_ = q.transpose(1, 2) + k_ = k.transpose(1, 2) + v_ = v.transpose(1, 2) + mask = _prepare_sdpa_mask( + gathered_mask, dtype=q_.dtype, device=q_.device + ) + sdpa_context = ( + sdpa_kernel(_PYTORCH_DEFAULT_CUDA_SDP_BACKENDS, set_priority=True) + if self.allow_cudnn_sdp and q_.device.type == "cuda" + else nullcontext() + ) + with sdpa_context: + return torch.nn.functional.scaled_dot_product_attention( + q_, + k_, + v_, + attn_mask=mask, + dropout_p=0.0, + is_causal=False, + scale=self.softmax_scale, + ).transpose(1, 2) + + def sequential(): + q_all, k_all, v_all = q, k, v + if exchange: + qkv_fast = _ipc_input_a2a_qkv(q, k, v) + if qkv_fast is not None: + q_all, k_all, v_all = qkv_fast + else: + q_all, k_all, v_all = _usp_input_all_to_all_qkv(q, k, v) + out = masked_attention(q_all, k_all, v_all) + if sp_size > 1: + out = _usp_output_all_to_all(out, head_dim=2) + return out + + if exchange: + out = self._pipelined_ulysses( + q, + k, + v, + masked_attention, + sequential, + layout="masked-tail" if tail_fa else "masked", + ) + if out is not None: + return out + return sequential() if effective_skip_sp or get_sequence_parallel_world_size() == 1: # No sequence parallelism, just run local attention. @@ -1444,6 +1486,27 @@ class USPAttention(nn.Module): q, k, v, ctx_attn_metadata, num_replicated_kv_prefix ) + if ( + sp_size > 1 + and not qkv_pre_all_to_all + and get_ring_parallel_world_size() == 1 + and not self.enable_packed_qkv_input_a2a + ): + + def attend(q_group, k_group, v_group): + return self.attn_impl.forward( + q_group, k_group, v_group, ctx_attn_metadata + ) + + def sequential(): + qs, ks, vs = _usp_input_all_to_all_qkv(q, k, v) + out = self.attn_impl.forward(qs, ks, vs, ctx_attn_metadata) + return _usp_output_all_to_all(out, head_dim=2) + + out = self._pipelined_ulysses(q, k, v, attend, sequential, layout="dense") + if out is not None: + return out + # Ulysses-style All-to-All for sequence/head sharding if sp_size > 1 and not qkv_pre_all_to_all: # -> [B, S, H_local, D] @@ -1482,6 +1545,91 @@ class USPAttention(nn.Module): return out + def _pipelined_ulysses( + self, + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + attend, + sequential, + *, + layout: str, + replicated=(None, None, None), + replicated_first: bool = True, + ) -> torch.Tensor | None: + """Ulysses attention with the exchange pipelined over head groups. + + The building block every Ulysses path of this layer goes through: q/k/v + are the call's ``[B, S_local, H, D]`` sequence shards and ``replicated`` + the rows every rank holds (see ``ulysses_pipelined_attention``). + ``attend(q, k, v)`` is the path's attention over the gathered sequence, + given one head group at a time, and ``sequential()`` the path's own + exchange-attend-exchange: the first call of each shape runs both and + keeps the pipeline only if every rank sees the same bytes. Returns + None, so the caller runs ``sequential()`` itself, when this call + cannot pipeline. + """ + groups = envs.SGLANG_DIFFUSION_ULYSSES_PIPELINE_GROUPS + if ( + not q.is_cuda + or groups in (0, 1) + or self.backend not in PIPELINED_ATTENTION_BACKENDS + or torch.compiler.is_compiling() + or torch.cuda.is_current_stream_capturing() + ): + return None + # shapes the transport already turned down stay down (too short to + # split, buffers that do not fit, a backend that failed the first-sight + # check); skip straight to the sequential path for them + key = ( + layout, + groups, + self.backend, + q.shape, + k.shape, + v.shape, + q.stride(), + k.stride(), + v.stride(), + tuple(None if t is None else (t.shape, t.stride()) for t in replicated), + replicated_first, + ) + if key in self._pipeline_declined: + return None + if ( + get_ring_parallel_world_size() > 1 + or get_request_skip_softmax_params() is not None + ): + return None + from sglang.multimodal_gen.runtime.distributed.device_communicators.ipc_a2a_multi import ( + ulysses_pipelined_attention, + ) + + def reference(): + # once per shape: leave no staging behind that later calls never use + with uncached_a2a_staging(): + return sequential() + + out = ulysses_pipelined_attention( + q, + k, + v, + attend, + groups, + sequential=reference, + signature=( + type(self.attn_impl).__name__, + self.causal, + self.softmax_scale, + layout, + ), + replicated=replicated, + replicated_first=replicated_first, + ) + if out is None: + self._pipeline_declined.add(key) + return out + def _forward_ring_tail_pad( self, q: torch.Tensor, @@ -1740,8 +1888,6 @@ class USPAttention(nn.Module): k_rep, k_shard = k[:, :num_rep], k[:, num_rep:] v_rep, v_shard = v[:, :num_rep], v[:, num_rep:] - q_shard, k_shard, v_shard = _usp_input_all_to_all_qkv(q_shard, k_shard, v_shard) - joint_mask = joint_mask_meta = None if attn_mask is not None: cache_key = ( @@ -1769,44 +1915,68 @@ class USPAttention(nn.Module): joint_mask_meta, ) - # Q and KV can have different head counts (GQA), so slice each replicated - # prefix by its own per-rank head shard to match the all-to-all'd suffix. - # For MHA (kv heads == q heads) this is identical to the q shard. - h_local = q_shard.shape[2] - kv_h_local = k_shard.shape[2] - h_start = u_rank * h_local - kv_h_start = u_rank * kv_h_local - q_rep = q_rep[:, :, h_start : h_start + h_local, :].contiguous() - k_rep = k_rep[:, :, kv_h_start : kv_h_start + kv_h_local, :].contiguous() - v_rep = v_rep[:, :, kv_h_start : kv_h_start + kv_h_local, :].contiguous() + def attend(q_group, k_group, v_group): + if joint_mask is not None: + return self._masked_local_attention( + q_group, + k_group, + v_group, + joint_mask, + attn_mask_meta=joint_mask_meta, + ) + return self.attn_impl.forward(q_group, k_group, v_group, ctx_attn_metadata) - q = torch.cat([q_rep, q_shard], dim=1) - out = self._replicated_kv_attention( - q, + def sequential(): + q_gathered, k_gathered, v_gathered = _usp_input_all_to_all_qkv( + q_shard, k_shard, v_shard + ) + # Q and KV can have different head counts (GQA), so slice each + # replicated prefix by its own per-rank head shard to match the + # all-to-all'd suffix. For MHA (kv heads == q heads) this is + # identical to the q shard. + h_local = q_gathered.shape[2] + kv_h_local = k_gathered.shape[2] + h_start = u_rank * h_local + kv_h_start = u_rank * kv_h_local + q_local = q_rep[:, :, h_start : h_start + h_local, :].contiguous() + k_local = k_rep[:, :, kv_h_start : kv_h_start + kv_h_local, :].contiguous() + v_local = v_rep[:, :, kv_h_start : kv_h_start + kv_h_local, :].contiguous() + + out = self._replicated_kv_attention( + torch.cat([q_local, q_gathered], dim=1), + k_gathered, + v_gathered, + k_local, + v_local, + ctx_attn_metadata, + attn_mask=joint_mask, + attn_mask_meta=joint_mask_meta, + ) + + out_rep = out[:, :num_rep] + out_shard = _usp_output_all_to_all(out[:, num_rep:], head_dim=2) + + if sp_size > 1: + gathered = [torch.empty_like(out_rep) for _ in range(sp_size)] + torch.distributed.all_gather( + gathered, + out_rep.contiguous(), + group=get_sp_group().ulysses_group, + ) + out_rep = torch.cat(gathered, dim=2) + + return torch.cat([out_rep, out_shard], dim=1) + + out = self._pipelined_ulysses( + q_shard, k_shard, v_shard, - k_rep, - v_rep, - ctx_attn_metadata, - attn_mask=joint_mask, - attn_mask_meta=joint_mask_meta, + attend, + sequential, + layout="replicated-prefix" + ("-masked" if joint_mask is not None else ""), + replicated=(q_rep, k_rep, v_rep), ) - - out_rep = out[:, :num_rep] - out_shard = out[:, num_rep:] - - out_shard = _usp_output_all_to_all(out_shard, head_dim=2) - - if sp_size > 1: - gathered = [torch.empty_like(out_rep) for _ in range(sp_size)] - torch.distributed.all_gather( - gathered, - out_rep.contiguous(), - group=get_sp_group().ulysses_group, - ) - out_rep = torch.cat(gathered, dim=2) - - return torch.cat([out_rep, out_shard], dim=1) + return out if out is not None else sequential() def _replicated_kv_attention( self, @@ -1998,32 +2168,52 @@ class USPAttention(nn.Module): """split form avoids materializing full K/V before Ulysses all-to-all""" u_rank = get_ulysses_parallel_rank() - if q.device.type == "cuda" and get_ulysses_parallel_world_size() > 1: - q, k_shard, v_shard = async_a2a_communicate( - [q, k_shard, v_shard], - get_ulysses_parallel_world_size(), - get_sp_group().ulysses_group, - self._get_usp_a2a_stream(), - local_seq_2_local_head=True, + def attend(q_group, k_group, v_group): + return self.attn_impl.forward(q_group, k_group, v_group, ctx_attn_metadata) + + def sequential(): + if q.device.type == "cuda" and get_ulysses_parallel_world_size() > 1: + q_gathered, k_gathered, v_gathered = async_a2a_communicate( + [q, k_shard, v_shard], + get_ulysses_parallel_world_size(), + get_sp_group().ulysses_group, + self._get_usp_a2a_stream(), + local_seq_2_local_head=True, + ) + q_gathered = q_gathered.contiguous() + k_gathered = k_gathered.contiguous() + v_gathered = v_gathered.contiguous() + else: + q_gathered = _usp_input_all_to_all(q, head_dim=2) + k_gathered = _usp_input_all_to_all(k_shard, head_dim=2) + v_gathered = _usp_input_all_to_all(v_shard, head_dim=2) + + h_kv_local = k_gathered.shape[2] + h_start = u_rank * h_kv_local + h_end = h_start + h_kv_local + k_local = k_rep[:, :, h_start:h_end, :].contiguous() + v_local = v_rep[:, :, h_start:h_end, :].contiguous() + + out = self._replicated_kv_attention( + q_gathered, + k_gathered, + v_gathered, + k_local, + v_local, + ctx_attn_metadata, ) - q = q.contiguous() - k_shard = k_shard.contiguous() - v_shard = v_shard.contiguous() - else: - q = _usp_input_all_to_all(q, head_dim=2) - k_shard = _usp_input_all_to_all(k_shard, head_dim=2) - v_shard = _usp_input_all_to_all(v_shard, head_dim=2) + return _usp_output_all_to_all(out, head_dim=2) - h_kv_local = k_shard.shape[2] - h_start = u_rank * h_kv_local - h_end = h_start + h_kv_local - k_rep = k_rep[:, :, h_start:h_end, :].contiguous() - v_rep = v_rep[:, :, h_start:h_end, :].contiguous() - - out = self._replicated_kv_attention( - q, k_shard, v_shard, k_rep, v_rep, ctx_attn_metadata + out = self._pipelined_ulysses( + q, + k_shard, + v_shard, + attend, + sequential, + layout="replicated-kv-prefix", + replicated=(None, k_rep, v_rep), ) - return _usp_output_all_to_all(out, head_dim=2) + return out if out is not None else sequential() def _forward_with_replicated_suffix( self, @@ -2049,37 +2239,58 @@ class USPAttention(nn.Module): k_shard, k_rep = k[:, :-num_rep], k[:, -num_rep:] v_shard, v_rep = v[:, :-num_rep], v[:, -num_rep:] - q_shard, k_shard, v_shard = _usp_input_all_to_all_qkv(q_shard, k_shard, v_shard) + def attend(q_group, k_group, v_group): + return self.attn_impl.forward(q_group, k_group, v_group, ctx_attn_metadata) - h_local = q_shard.shape[2] - kv_h_local = k_shard.shape[2] - h_start = u_rank * h_local - kv_h_start = u_rank * kv_h_local - q_rep = q_rep[:, :, h_start : h_start + h_local, :].contiguous() - k_rep = k_rep[:, :, kv_h_start : kv_h_start + kv_h_local, :].contiguous() - v_rep = v_rep[:, :, kv_h_start : kv_h_start + kv_h_local, :].contiguous() - - q = torch.cat([q_shard, q_rep], dim=1) - out = self._replicated_kv_attention( - q, k_shard, v_shard, k_rep, v_rep, ctx_attn_metadata, rep_first=False - ) - - out_shard = out[:, :-num_rep] - out_rep = out[:, -num_rep:] - - out_shard = _usp_output_all_to_all(out_shard, head_dim=2) - - sp_size = get_ulysses_parallel_world_size() - if sp_size > 1: - gathered = [torch.empty_like(out_rep) for _ in range(sp_size)] - torch.distributed.all_gather( - gathered, - out_rep.contiguous(), - group=get_sp_group().ulysses_group, + def sequential(): + q_gathered, k_gathered, v_gathered = _usp_input_all_to_all_qkv( + q_shard, k_shard, v_shard ) - out_rep = torch.cat(gathered, dim=2) - return torch.cat([out_shard, out_rep], dim=1) + h_local = q_gathered.shape[2] + kv_h_local = k_gathered.shape[2] + h_start = u_rank * h_local + kv_h_start = u_rank * kv_h_local + q_local = q_rep[:, :, h_start : h_start + h_local, :].contiguous() + k_local = k_rep[:, :, kv_h_start : kv_h_start + kv_h_local, :].contiguous() + v_local = v_rep[:, :, kv_h_start : kv_h_start + kv_h_local, :].contiguous() + + out = self._replicated_kv_attention( + torch.cat([q_gathered, q_local], dim=1), + k_gathered, + v_gathered, + k_local, + v_local, + ctx_attn_metadata, + rep_first=False, + ) + + out_shard = _usp_output_all_to_all(out[:, :-num_rep], head_dim=2) + out_rep = out[:, -num_rep:] + + sp_size = get_ulysses_parallel_world_size() + if sp_size > 1: + gathered = [torch.empty_like(out_rep) for _ in range(sp_size)] + torch.distributed.all_gather( + gathered, + out_rep.contiguous(), + group=get_sp_group().ulysses_group, + ) + out_rep = torch.cat(gathered, dim=2) + + return torch.cat([out_shard, out_rep], dim=1) + + out = self._pipelined_ulysses( + q_shard, + k_shard, + v_shard, + attend, + sequential, + layout="replicated-suffix", + replicated=(q_rep, k_rep, v_rep), + replicated_first=False, + ) + return out if out is not None else sequential() class _BCGBoxedTupleOutput: diff --git a/python/sglang/multimodal_gen/runtime/layers/usp.py b/python/sglang/multimodal_gen/runtime/layers/usp.py index 8103c4ce16..8ffc89a3cb 100644 --- a/python/sglang/multimodal_gen/runtime/layers/usp.py +++ b/python/sglang/multimodal_gen/runtime/layers/usp.py @@ -1,5 +1,6 @@ # Copied and adapted from: https://github.com/hao-ai-lab/FastVideo +import contextlib import logging import math from typing import TYPE_CHECKING @@ -28,6 +29,25 @@ logger = logging.getLogger(__name__) _A2A_STAGING_BUFFERS: dict[tuple[str, torch.dtype, int], torch.Tensor] = {} +# depth of open uncached_a2a_staging() blocks +_UNCACHED_STAGING = 0 + + +@contextlib.contextmanager +def uncached_a2a_staging(): + """Run the Ulysses exchanges inside the block over NCCL on per-call buffers. + + For a one-off exchange the steady state never repeats (the pipelined + attention's first-sight reference): the staging cache and the IPC + transports' staging pairs would otherwise stay allocated at its size. An + all-to-all is a permutation, so the transport never changes its bytes. + """ + global _UNCACHED_STAGING + _UNCACHED_STAGING += 1 + try: + yield + finally: + _UNCACHED_STAGING -= 1 def drop_a2a_staging_buffers() -> None: @@ -59,7 +79,8 @@ def _a2a_staging_buffer( not be shared with eager replays. """ if ( - torch.is_grad_enabled() + _UNCACHED_STAGING + or torch.is_grad_enabled() or torch.compiler.is_compiling() or device.type != "cuda" or torch.cuda.is_current_stream_capturing() @@ -88,7 +109,7 @@ def _usp_all_to_all_single(x: torch.Tensor, role: str | None = None) -> torch.Te assert ulysses_pg is not None, "Ulysses process group is not initialized." x_shape = x.shape x = x.flatten().contiguous() - if x.is_cuda: + if x.is_cuda and not _UNCACHED_STAGING: from sglang.multimodal_gen.runtime.distributed.device_communicators.ipc_a2a_multi import ( IPC_A2A_MULTI, ipc_a2a_multi_ready, @@ -148,6 +169,8 @@ def _ipc_ready_group(): ipc_a2a_ready, ) + if _UNCACHED_STAGING: + return None group = get_sp_group().ulysses_group return group if ipc_a2a_ready(group) else None diff --git a/python/sglang/multimodal_gen/test/server/gpu_cases.py b/python/sglang/multimodal_gen/test/server/gpu_cases.py index 9a25008ef1..8d1cc771b2 100644 --- a/python/sglang/multimodal_gen/test/server/gpu_cases.py +++ b/python/sglang/multimodal_gen/test/server/gpu_cases.py @@ -1562,6 +1562,7 @@ STANDALONE_FILES = { "../single_test_file/test_ar_models.py", "../single_test_file/test_ipc_a2a_2_gpu.py", "../single_test_file/test_ipc_a2a_multi_gpu.py", + "../single_test_file/test_usp_pipelined_attention_multi_gpu.py", "../single_test_file/test_encoder_fold_srt_linear_2_gpu.py", "../single_test_file/test_encoder_fold_srt_2_gpu.py", "../single_test_file/test_diffusion_bcg_tp2_zimage_turbo.py", @@ -1605,6 +1606,8 @@ STANDALONE_FILE_EST_TIMES = { "../single_test_file/test_ipc_a2a_2_gpu.py": 240.0, # 2 ranks here (4 ranks skip below four GPUs); ~50 s on H200 with warm JIT "../single_test_file/test_ipc_a2a_multi_gpu.py": 180.0, + # 2 ranks here (4 ranks skip below four GPUs); ~45 s on B300 with warm JIT + "../single_test_file/test_usp_pipelined_attention_multi_gpu.py": 120.0, "../single_test_file/test_encoder_fold_srt_linear_2_gpu.py": 120.0, "../single_test_file/test_encoder_fold_srt_2_gpu.py": 240.0, # ~60 s locally with a warm HF cache (load + one capture + 4 steps); diff --git a/python/sglang/multimodal_gen/test/single_test_file/test_ipc_a2a_multi_gpu.py b/python/sglang/multimodal_gen/test/single_test_file/test_ipc_a2a_multi_gpu.py index 8e5b366cfb..1e1385a6a7 100644 --- a/python/sglang/multimodal_gen/test/single_test_file/test_ipc_a2a_multi_gpu.py +++ b/python/sglang/multimodal_gen/test/single_test_file/test_ipc_a2a_multi_gpu.py @@ -27,6 +27,24 @@ def _attend(q, k, v): return (q.float() * 2 + k.flip(0).float() - v.roll(1, 0).float()).to(q.dtype) +def _attend_rows(q, k, v): + """`_attend` for [batch, rows, heads, head_dim] parts whose k/v may carry more + rows (a replicated KV prefix) and fewer heads (GQA) than q.""" + rep = q.shape[2] // k.shape[2] + k = k.repeat_interleave(rep, dim=2).float() + v = v.repeat_interleave(rep, dim=2).float() + rows = q.shape[1] + # elementwise only (a reduction could round differently per head count): + # the leading and the trailing rows both read, so every row counts + return ( + q.float() * 2 + + k[:, :rows] + + k.flip(1)[:, :rows] + - v.roll(1, 1)[:, -rows:] + + v[:, :rows] * 0.5 + ).to(q.dtype) + + def _worker() -> int: import torch.distributed as dist @@ -69,7 +87,7 @@ def _worker() -> int: if world == 2: cases.append((128, 16, 64, 8)) for s_local, heads, head_dim, groups in cases: - # three calls in a row: the slots alternate and the counters advance + # three calls in a row: the slots are reused and the counters advance for call in range(3): torch.manual_seed( 1000 * call + s_local @@ -132,23 +150,146 @@ def _worker() -> int: f"filled pipeline {(s_local, heads, head_dim, groups)} call {call}" ) - # groups=-1 picks a count that divides the heads per rank (4 for 16 heads at - # 2 ranks, 2 at 4) and still matches + # groups=-1 picks the most groups that divide the heads per rank and whose + # calls still fill four waves of 128-row tiles, and still matches; a shape too + # small to fill them keeps the sequential exchange + sms = torch.cuda.get_device_properties(rank).multi_processor_count + s_auto = 4 * sms * 128 // world # one head per group fills four waves torch.manual_seed(77) full = [ - torch.randn(64 * world, 16, 64, dtype=torch.bfloat16, device="cuda") + torch.randn(s_auto * world, 16, 64, dtype=torch.bfloat16, device="cuda") for _ in range(3) ] - q, k, v = (t.narrow(0, rank * 64, 64).contiguous() for t in full) + q, k, v = (t.narrow(0, rank * s_auto, s_auto).contiguous() for t in full) got = ulysses_pipelined_attention(q, k, v, _attend, -1) if got is None or not torch.equal(sequential(q, k, v), got): failures.append("auto group count") + small = torch.randn(64, 16, 64, dtype=torch.bfloat16, device="cuda") + if ulysses_pipelined_attention(small, small, small, _attend, -1) is not None: + failures.append("auto pipelined a shape too small to fill the GPU") + + # sequential=: the first call per shape and signature also runs the + # sequential exchange and keeps the pipeline only on a byte-for-byte match + torch.manual_seed(78) + full = [ + torch.randn(80 * world, 16, 64, dtype=torch.bfloat16, device="cuda") + for _ in range(3) + ] + q, k, v = (t.narrow(0, rank * 80, 80).contiguous() for t in full) + ref = sequential(q, k, v) + for call in range(2): + got = ulysses_pipelined_attention( + q, + k, + v, + _attend, + 2, + sequential=lambda: sequential(q, k, v), + signature="same", + ) + if got is None or not torch.equal(ref, got): + failures.append(f"verified pipeline call {call}") + off = ref + 1 # a "sequential" result the pipeline cannot reproduce + got = ulysses_pipelined_attention( + q, k, v, _attend, 2, sequential=lambda: off, signature="other" + ) + if got is None or not torch.equal(got, off): + failures.append("a mismatching first call did not return the sequential result") + if ( + ulysses_pipelined_attention( + q, k, v, _attend, 2, sequential=lambda: off, signature="other" + ) + is not None + ): + failures.append("a mismatched shape kept pipelining") # a head count the groups cannot split must decline, not mis-shard q = torch.randn(32, 12, 64, dtype=torch.bfloat16, device="cuda") if ulysses_pipelined_attention(q, q, q, _attend, 4) is not None: failures.append("pipelined accepted 12 heads in 4 groups") + # batch, replicated rows (prefix, suffix, KV-only prefix) and GQA: the output + # rows match attention over the global layout, computed with every head + def layout_case(batch, s_local, heads, kv_heads, rep, rep_kv, first, groups, seed): + torch.manual_seed(seed) + dims = 32 + shard = lambda h: torch.randn( + batch, s_local * world, h, dims, device="cuda" + ).bfloat16() + full = [shard(heads), shard(kv_heads), shard(kv_heads)] + reps = [ + torch.randn(batch, n, h, dims, device="cuda").bfloat16() if n else None + for n, h in ((rep, heads), (rep_kv, kv_heads), (rep_kv, kv_heads)) + ] + mine = [t[:, rank * s_local : (rank + 1) * s_local].contiguous() for t in full] + + def joined(t, r): + if r is None: + return t + return torch.cat([r, t] if first else [t, r], dim=1) + + out_all = _attend_rows(*(joined(t, r) for t, r in zip(full, reps))) + lo = (rep if first else 0) + rank * s_local + own = out_all[:, lo : lo + s_local] + if rep: + rep_rows = out_all[:, :rep] if first else out_all[:, -rep:] + own = torch.cat([rep_rows, own] if first else [own, rep_rows], dim=1) + got = ulysses_pipelined_attention( + *mine, + _attend_rows, + groups, + replicated=tuple(reps), + replicated_first=first, + ) + label = f"layout b{batch} h{heads}/{kv_heads} rep {rep}/{rep_kv} first={first} g{groups}" + if got is None: + failures.append(f"{label} returned None") + elif not torch.equal(got, own): + failures.append(label) + + for call in range(2): # every layout reuses the one pool + layout_case(2, 48, 16, 16, 0, 0, True, 2, 300 + call) + layout_case(1, 40, 16, 16, 9, 9, True, 2, 310 + call) + layout_case(2, 40, 16, 16, 7, 7, False, 2, 320 + call) + layout_case(1, 40, 16, 16, 0, 11, True, 2, 330 + call) + layout_case(2, 40, 16, 8, 5, 5, True, 2, 340 + call) + + # one pool per dtype serves every plan above, sized for the largest of them + pools = IPC_A2A_MULTI.pools + if list(pools) != [torch.bfloat16]: + failures.append(f"pipeline pools per dtype: {list(pools)}") + else: + slots = list(IPC_A2A_MULTI.slots.values()) + widest = ( + max(s.in_total for s in slots), + max(s.out_total for s in slots), + max(s.plan.groups for s in slots), + ) + if pools[torch.bfloat16].sizes != widest: + failures.append(f"pool sizes {pools[torch.bfloat16].sizes} != {widest}") + + # one set of slots is enough: a peer cannot write a call's rows or output + # blocks before this rank has read the previous call's, however far ahead + # it runs. Rank 0 attends slowly; alternating plans share the pool. + def slow_attend(q, k, v): + if rank == 0: + torch.cuda._sleep(20_000_000) + return _attend(q, k, v) + + issued = [] + for call in range(6): # queued back to back; compared only afterwards + s_local, heads, groups = ((96, 16, 2), (40, 32, 4))[call % 2] + torch.manual_seed(4000 + call) + full = [ + torch.randn(s_local * world, heads, 64, dtype=torch.bfloat16, device="cuda") + for _ in range(3) + ] + parts = [t.narrow(0, rank * s_local, s_local).contiguous() for t in full] + issued.append((parts, ulysses_pipelined_attention(*parts, slow_attend, groups))) + for call, (parts, got) in enumerate(issued): + if got is None or not torch.equal(sequential(*parts), got): + failures.append(f"skewed back-to-back call {call}") + # the plain N-rank exchange behind _usp_all_to_all_single for numel in (world * 1024, world * 4096 * 33): x = torch.randn(numel, dtype=torch.bfloat16, device="cuda") diff --git a/python/sglang/multimodal_gen/test/single_test_file/test_usp_pipelined_attention_multi_gpu.py b/python/sglang/multimodal_gen/test/single_test_file/test_usp_pipelined_attention_multi_gpu.py new file mode 100644 index 0000000000..42f28423e6 --- /dev/null +++ b/python/sglang/multimodal_gen/test/single_test_file/test_usp_pipelined_attention_multi_gpu.py @@ -0,0 +1,232 @@ +"""Every Ulysses path of USPAttention gives the same bytes with the exchange +pipelined over head groups as with the sequential exchange. + +The pipeline only engages on a same-host Ulysses group with peer-to-peer access, +so nothing in the single-GPU suite reaches it. Runs on 2 ranks, and on 4 when +four GPUs are visible: + + pytest -v python/sglang/multimodal_gen/test/single_test_file/test_usp_pipelined_attention_multi_gpu.py +""" + +from __future__ import annotations + +import os +import subprocess +import sys +import unittest + +import torch + +from sglang.multimodal_gen.runtime.platforms import current_platform +from sglang.test.test_utils import CustomTestCase + + +def _worker() -> int: + import torch.distributed as dist + + from sglang.multimodal_gen import envs + from sglang.multimodal_gen.runtime.distributed.parallel_state import ( + maybe_init_distributed_environment_and_model_parallel, + ) + + rank = int(os.environ["RANK"]) + world = int(os.environ["WORLD_SIZE"]) + torch.cuda.set_device(rank) + maybe_init_distributed_environment_and_model_parallel( + tp_size=1, sp_size=world, ulysses_degree=world + ) + + from sglang.multimodal_gen.runtime.distributed.device_communicators.ipc_a2a import ( + IPC_A2A, + ) + from sglang.multimodal_gen.runtime.distributed.device_communicators.ipc_a2a_multi import ( + IPC_A2A_MULTI, + ) + from sglang.multimodal_gen.runtime.layers import usp + from sglang.multimodal_gen.runtime.layers.attention.layer import ( + PIPELINED_ATTENTION_BACKENDS, + USPAttention, + ) + from sglang.multimodal_gen.runtime.managers.forward_context import ( + set_forward_context, + ) + from sglang.multimodal_gen.runtime.server_args import set_global_server_args + from sglang.multimodal_gen.test.unit.conftest import _make_unit_server_args + + set_global_server_args(_make_unit_server_args()) + peers = [d for d in range(world) if d != rank] + if not all(torch.cuda.can_device_access_peer(rank, d) for d in peers): + print("SKIP no peer-to-peer access between the devices", flush=True) + return 0 + + heads, kv_heads, dim, s_local, n_rep = 16, 8, 64, 96, 24 + failures, ran = [], [] + backends = [b for b in PIPELINED_ATTENTION_BACKENDS] + + def shard(batch, rows, h, seed): + torch.manual_seed(seed) # identical full tensors on every rank + full = torch.randn(batch, rows * world, h, dim, device="cuda").bfloat16() + return full[:, rank * rows : (rank + 1) * rows].contiguous() + + def replicated(batch, rows, h, seed): + torch.manual_seed(seed) + return torch.randn(batch, rows, h, dim, device="cuda").bfloat16() + + def both(label, call): + """`call()` sequentially and pipelined (two forced groups) must agree.""" + outs = [] + for groups in (0, 2): + envs.SGLANG_DIFFUSION_ULYSSES_PIPELINE_GROUPS = groups + with torch.inference_mode(), set_forward_context(0, None, None): + outs.append(call()) + envs.SGLANG_DIFFUSION_ULYSSES_PIPELINE_GROUPS = -1 + if not torch.equal(outs[0], outs[1]): + failures.append(label) + ran.append(label) + + for backend in backends: + try: + layer = USPAttention( + num_heads=heads, + head_size=dim, + causal=False, + required_attention_backend=backend, + prefix=f"test.{backend.name}", + ) + except Exception as e: # noqa: BLE001 - a backend this GPU cannot build + print(f"backend {backend.name} unavailable here: {e}", flush=True) + continue + if layer.backend is not backend: + print( + f"backend {backend.name} resolved to {layer.backend.name}", flush=True + ) + continue + name = backend.name + for batch in (1, 2): + q, k, v = (shard(batch, s_local, heads, 10 + i) for i in range(3)) + both(f"{name} dense b{batch}", lambda: layer(q, k, v)) + + rep = [replicated(batch, n_rep, heads, 20 + i) for i in range(3)] + joined = [torch.cat([r, t], dim=1) for r, t in zip(rep, (q, k, v))] + both( + f"{name} replicated prefix b{batch}", + lambda: layer(*joined, num_replicated_prefix=n_rep), + ) + mask = torch.ones(batch, n_rep + s_local, dtype=torch.bool, device="cuda") + mask[:, n_rep // 2 : n_rep] = False # padded text tokens + both( + f"{name} replicated prefix + key mask b{batch}", + lambda: layer(*joined, num_replicated_prefix=n_rep, attn_mask=mask), + ) + trailing = [torch.cat([t, r], dim=1) for r, t in zip(rep, (q, k, v))] + both( + f"{name} replicated suffix b{batch}", + lambda: layer(*trailing, num_replicated_suffix=n_rep), + ) + k_rep, v_rep = (replicated(batch, n_rep, heads, 30 + i) for i in range(2)) + both( + f"{name} replicated KV prefix b{batch}", + lambda: layer.forward_with_replicated_kv_prefix(q, k_rep, v_rep, k, v), + ) + key_mask = torch.ones(batch, s_local, dtype=torch.bool, device="cuda") + key_mask[:, -s_local // 4 :] = rank % 2 == 0 # ragged across ranks + both( + f"{name} gathered key mask b{batch}", + lambda: layer(q, k, v, attn_mask=key_mask), + ) + gqa = USPAttention( + num_heads=heads, + head_size=dim, + num_kv_heads=kv_heads, + causal=False, + required_attention_backend=backend, + prefix=f"test.gqa.{backend.name}", + ) + q = shard(1, s_local, heads, 40) + k, v = (shard(1, s_local, kv_heads, 41 + i) for i in range(2)) + both(f"{name} GQA dense", lambda: gqa(q, k, v)) + + # the first call of a shape checks the pipeline against a sequential + # exchange on per-call buffers: no staging sized for it stays behind + usp.drop_a2a_staging_buffers() + IPC_A2A.drop_staging() + checked = len(IPC_A2A_MULTI.verified) + q, k, v = (shard(1, 2 * s_local, heads, 50 + i) for i in range(3)) + envs.SGLANG_DIFFUSION_ULYSSES_PIPELINE_GROUPS = 2 + with torch.inference_mode(), set_forward_context(0, None, None): + layer(q, k, v) + envs.SGLANG_DIFFUSION_ULYSSES_PIPELINE_GROUPS = -1 + if len(IPC_A2A_MULTI.verified) != checked + 1: + failures.append(f"{name}: a new shape was not checked") + if usp._A2A_STAGING_BUFFERS or IPC_A2A.staging: + failures.append( + f"{name}: the first-sight check left sequential staging behind " + f"({list(usp._A2A_STAGING_BUFFERS)}, {len(IPC_A2A.staging)} IPC pairs)" + ) + + # a comparison where every arm quietly fell back would pass while testing + # nothing: the pipeline must have run and verified every layout + if not ran: + failures.append("no pipelined backend could be built") + if IPC_A2A_MULTI.mismatched: + failures.append( + f"pipeline disagreed with the sequential exchange: {IPC_A2A_MULTI.mismatched}" + ) + if len(IPC_A2A_MULTI.verified) < len({label.split(" b")[0] for label in ran}): + failures.append( + f"pipeline verified {len(IPC_A2A_MULTI.verified)} layouts for {len(ran)} cases" + ) + + verdict = torch.tensor([len(failures)], device="cuda") + dist.all_reduce(verdict) + if failures: + print(f"rank{rank} MISMATCH: {failures}", flush=True) + if rank == 0: + print(f"ran: {ran}", flush=True) + print( + f"USP_PIPELINE_PARITY {'FAIL' if verdict.item() else 'PASS'} world={world}", + flush=True, + ) + dist.barrier() + dist.destroy_process_group() + return 1 if verdict.item() else 0 + + +class TestUspPipelinedAttentionMultiGpu(CustomTestCase): + def _run(self, world: int, port: int): + if not current_platform.is_cuda(): + self.skipTest("the copy-engine pipeline is CUDA-only") + if torch.cuda.device_count() < world: + self.skipTest(f"needs {world} GPUs") + proc = subprocess.run( + [ + sys.executable, + "-m", + "torch.distributed.run", + f"--nproc-per-node={world}", + f"--master-port={port}", + __file__, + "--worker", + ], + capture_output=True, + text=True, + timeout=1200, + ) + print(proc.stdout[-4000:]) + if proc.returncode != 0: + print(proc.stderr[-4000:], file=sys.stderr) + self.assertEqual(proc.returncode, 0, "pipelined USPAttention diverged") + if "SKIP" not in proc.stdout: + self.assertIn(f"USP_PIPELINE_PARITY PASS world={world}", proc.stdout) + + def test_two_ranks_match_sequential_bitwise(self): + self._run(2, 29547) + + def test_four_ranks_match_sequential_bitwise(self): + self._run(4, 29557) + + +if __name__ == "__main__": + if "--worker" in sys.argv: + raise SystemExit(_worker()) + unittest.main()