[Refactor] Remove Kimi K3's own layer communication and let the consumer set an FFN's exit (#42936)
This commit is contained in:
@@ -82,6 +82,7 @@ The producer owns `ResidualUpdate`; the consumer owns `ResidualReadout` and its
|
||||
- A nonlinear update cannot generally move before reduction. The post operation of manifold-constrained hyper-connections (MHC) is one such case.
|
||||
- `applied_at_exit` requests an update at the producer exit. A deferred update must guarantee that its parameters and state outlive that producer.
|
||||
- `ResidualReadout.reads_before_dp_gather` preserves reads that must execute on the source rows before a DP gather.
|
||||
- A read that needs every row of the attention's, such as one that adds a contribution computed for every row, declares `reads_after_attn_tp_gather`. A producer that would leave its output on this rank's attention-TP slice then returns it to the attention's rows; Qwen4-exp's per-layer embedding read and Kimi K3's attention-residual bank declare it.
|
||||
|
||||
A read returns `(compute_input, residual)`. `update_and_read()` receives the producer's update together with the read, so an implementation can fuse them: the plain add+norm read does, and keeps its FP32 accumulation instead of normalizing a separately rounded snapshot. Fusion is not guaranteed. An implementation such as MHC may run the update and the read as separate steps, provided it preserves its existing rounding order.
|
||||
|
||||
@@ -202,6 +203,8 @@ hidden_states = residual_batch.final_norm(
|
||||
|
||||
`skip_empty=True` still completes owed work on an empty batch, which may be a collective that other ranks join, and skips only the norm kernel. `final_norm()` and `to_pp()` release the batch's stream after they hand on its output.
|
||||
|
||||
When the final read is more than a norm, pass it to `make_layers(final_read=...)` (or `layer_stack(final_read=...)`): it is then the last stage's consumer. Its `attn_tp_gather` gathers the rows the last FFN leaves on this rank's attention-TP slice, and with `reads_attn_tp_slices` it reads that slice itself and the last FFN leaves its output there. Kimi K3's `AttnBankOutputRead` is such a read.
|
||||
|
||||
This is a non-pipeline stack fragment. Pipeline reception uses the first boundary's `from_pp(tensors, forward_batch)` to reconstruct a stream; the sender uses `residual_batch.to_pp()`. A stream the producer already wrote travels as `hidden_states` alone and arrives written: MHC's expanded streams, and the output of an FFN on its own rows (its attention-TP slice, or an unpadded batch's rows), which writes it at its exit before a pipeline handoff. The receiver knows this from the producer's declaration, not from which tensors arrive, since a captured CUDA graph's input buffers hold every key. Dynamic completion work is resolved before transport. By default, `to_pp()` preserves a statically declared partial sum for the receiver's incoming contract to reconstruct and complete. Do not call `complete_output()` before that handoff: it would complete a sum that the receiver still expects to perform.
|
||||
|
||||
A terminal FFN normally completes its output before the final norm. The exception is a MoE finalize handoff. When the stage's fusion provider can defer the finalize at the terminal layer (for CuteDSL, `install_cutedsl_fusion(..., terminal_finalize=True)`), the exit may leave a `DeferredFinalize` for the final norm. Pass the handle straight to `residual_batch.final_norm(hidden_states, forward_batch, self.norm, finalize_norm=service)`, where `service` is the object `install_cutedsl_fusion()` returned; its `finalize(handoff, residual, gamma)` fuses finalize, all-reduce, residual add, and norm. The fused path reads `layernorm.gemma_weight`, so it currently expects a `GemmaRMSNorm` final norm. Without `finalize_norm`, `final_norm()` first completes the handoff unfused, and `capture` is then allowed. With `finalize_norm`, `final_norm()` rejects `capture` when a handoff arrives. Calling `complete_output()` first runs the producer's unfused finalize, and `snapshot()` raises on a handoff. See `python/sglang/srt/models/qwen3_5.py`.
|
||||
@@ -339,7 +342,7 @@ A backend fusion provider is passed per stage as the `fusions` option, `(declara
|
||||
|
||||
`install_cutedsl_fusion()` and `prepare_cutedsl_fusion()` find providers through the decoder layer's `attn_boundary` and `ffn_boundary` attributes.
|
||||
|
||||
Consumer fusion candidates are ordered. A candidate returning `None` must do so before modifying inputs or starting a collective, so fallback remains valid. `FfnInputFusion` declares the sum group it completes with residual add and norm. A residual read may supply its own candidates as `completing_fusions`: each `ReadoutFusion` completes a sum with the residual add, ahead of the read, and is tried before the built-in collective; one with `reads=True` also does the read. Where a stage reads its input on this rank's attention-TP slice and then gathers it, a read's `gathering_reads` may do both in one kernel. A stage may declare its own implementation of its gathers over attention TP (`declare_attn(attn_tp_gather=...)`, `declare_ffn(attn_tp_gather=...)`), tried on every batch before the built-in gather. The consumer only tries compatible candidates for the declared update/read capabilities.
|
||||
Consumer fusion candidates are ordered. A candidate returning `None` must do so before modifying inputs or starting a collective, so fallback remains valid. `FfnInputFusion` declares the sum group it completes with residual add and norm. A residual read may supply its own candidates as `completing_fusions`: each `ReadoutFusion` completes a sum with the residual add, ahead of the read, and is tried before the built-in collective; one with `reads=True` also does the read. Where a stage reads its input on this rank's attention-TP slice and then gathers it, a read's `gathering_reads` may do both in one kernel. A stage may declare its own implementation of the gather over attention TP that brings rows into it (`declare_attn(attn_tp_gather=...)`), tried on every batch before the built-in gather: at its own entry, or at its producer's exit, since an exit runs its consumer's gather. Across a pipeline boundary the producer's rank takes that gather from the neighbouring layer it builds on the meta device, so the gather may use only the communication state of the rank it runs on, not the declaring layer's weights or buffers. The consumer only tries compatible candidates for the declared update/read capabilities.
|
||||
|
||||
An `OutputTransform` declares contribution processing before residual update. Its explicit `before_reduce_scatter` setting preserves a supported implementation's order; it is not permission to commute arbitrary operations across a reduction.
|
||||
|
||||
|
||||
@@ -47,10 +47,6 @@ from sglang.srt.layers.layer_boundary.layout import (
|
||||
batches_are_unpadded,
|
||||
is_dense_ffn_fully_dp,
|
||||
)
|
||||
from sglang.srt.layers.layer_boundary.ops import (
|
||||
attn_tp_gather_input,
|
||||
update_attn_tp_gather_output,
|
||||
)
|
||||
from sglang.srt.layers.layer_boundary.prepare import (
|
||||
_attn_input_default,
|
||||
_attn_input_scattered,
|
||||
@@ -191,6 +187,7 @@ class StagePlan:
|
||||
qkv_latent_func=None,
|
||||
fusions=None,
|
||||
attn_tp_gather=None,
|
||||
exit_gather=None,
|
||||
):
|
||||
self.norm = norm
|
||||
self.edges = dict(variants)
|
||||
@@ -242,7 +239,7 @@ class StagePlan:
|
||||
else bind_exit(
|
||||
edges.outgoing,
|
||||
cp_moves=edges.cp_moves,
|
||||
attn_tp_gather=attn_tp_gather,
|
||||
attn_tp_gather=exit_gather,
|
||||
)
|
||||
)
|
||||
self.paths[variant] = StagePath(
|
||||
@@ -253,15 +250,6 @@ class StagePlan:
|
||||
returns_over_dp=out.returns_over_dp,
|
||||
writes_at_handoff=writes_at_handoff,
|
||||
)
|
||||
if attn_tp_gather is not None and not any(
|
||||
getattr(path.output_move, "func", None) is update_attn_tp_gather_output
|
||||
or getattr(path.entry.input_move, "func", None) is attn_tp_gather_input
|
||||
for path in self.paths.values()
|
||||
):
|
||||
raise ValueError(
|
||||
"an attention-TP gather is declared, but no batch variant "
|
||||
"gathers over attention TP at this stage's boundaries"
|
||||
)
|
||||
|
||||
@property
|
||||
def incoming_residual_rows(self):
|
||||
@@ -319,7 +307,7 @@ class StagePlan:
|
||||
)
|
||||
|
||||
|
||||
def _bind_stage(declaration, norm, incoming, outgoing, **options):
|
||||
def _bind_stage(declaration, norm, incoming, outgoing, *, final_read=None, **options):
|
||||
if incoming.consumer != declaration or outgoing.producer != declaration:
|
||||
raise ValueError("connections do not match the stage declaration")
|
||||
if incoming.entries.keys() != outgoing.exits.keys():
|
||||
@@ -354,6 +342,13 @@ def _bind_stage(declaration, norm, incoming, outgoing, **options):
|
||||
terminal=declaration.terminal,
|
||||
writes_at_handoff=declaration.writes_at_handoff,
|
||||
attn_tp_gather=declaration.attn_tp_gather,
|
||||
# The exit runs its consumer's gather: the next stage's, or the
|
||||
# final read's.
|
||||
exit_gather=getattr(
|
||||
outgoing.consumer if outgoing.consumer is not None else final_read,
|
||||
"attn_tp_gather",
|
||||
None,
|
||||
),
|
||||
finishes_directly=declaration.kind is StageKind.ATTENTION
|
||||
and declaration.reduction is ProducerReduction.ALWAYS_PARTIAL,
|
||||
**options,
|
||||
|
||||
@@ -45,9 +45,9 @@ class ExitRows(Enum):
|
||||
|
||||
ATTENTION = auto()
|
||||
TBO_SPLIT = auto()
|
||||
# The rows the FFN ran on, also at the stack's end: an FFN on this rank's
|
||||
# attention-TP slice leaves its output there, and the model's final read
|
||||
# must read that slice and gather it.
|
||||
# The rows the FFN ran on, also at the stack's end: the layer stack's last
|
||||
# FFN, when the model's final read reads this rank's attention-TP slice
|
||||
# and gathers it.
|
||||
SLICE = auto()
|
||||
|
||||
|
||||
|
||||
@@ -44,6 +44,10 @@ from sglang.srt.layers.layer_boundary.layout import (
|
||||
moe_gathers_over_moe_cp,
|
||||
token_axis_sizes,
|
||||
)
|
||||
from sglang.srt.layers.layer_boundary.ops import (
|
||||
attn_tp_gather_input,
|
||||
update_attn_tp_gather_output,
|
||||
)
|
||||
from sglang.srt.layers.layer_boundary.output import OutputTransform
|
||||
from sglang.srt.layers.layer_boundary.residual import ResidualReadout, ResidualUpdate
|
||||
from sglang.srt.layers.layer_boundary.residual.add_norm import (
|
||||
@@ -261,10 +265,14 @@ class StageDeclaration:
|
||||
sum, so the exit owes none. For an FFN whose output sum is fused
|
||||
with a reduction its computation needs anyway.
|
||||
attn_tp_gather: Optional implementation of the gather over attention
|
||||
TP this stage's boundaries run: given this rank's contiguous slice
|
||||
of the rows, it returns them all, in rank order, or None to leave
|
||||
the gather to the boundary. Called on every batch, so it must be
|
||||
CUDA-graph safe.
|
||||
TP that brings rows into this stage: at its own entry, or at its
|
||||
producer's exit, which runs its consumer's gather. Given this rank's
|
||||
contiguous slice of the rows, it returns them all, in rank order,
|
||||
or None to leave the gather to the boundary. Called on every batch,
|
||||
so it must be CUDA-graph safe. Across a pipeline boundary the
|
||||
producer's rank takes it from the neighbouring layer it builds on
|
||||
the meta device, so it may use only the communication state of the
|
||||
rank it runs on, not the declaring layer's weights or buffers.
|
||||
previous: Declaration whose output this stage consumes, as the stack
|
||||
records it; across pipeline ranks it is built locally.
|
||||
prepared_from: Declaration whose already-read input a branch reuses.
|
||||
@@ -346,8 +354,8 @@ def declare_attn(
|
||||
stage's input runs it, so no fused add + norm takes that input.
|
||||
Requires ALWAYS_PARTIAL.
|
||||
tp_group: Head partition; ATTN_TP by default, TP for full-TP consumers.
|
||||
attn_tp_gather: Implementation of this stage's gathers over attention
|
||||
TP, tried before the boundary's own (see StageDeclaration).
|
||||
attn_tp_gather: Implementation of the gather over attention TP into
|
||||
this stage, tried before the boundary's own (see StageDeclaration).
|
||||
|
||||
Returns:
|
||||
A StageDeclaration with no norm, tensors or execution plan.
|
||||
@@ -379,7 +387,6 @@ def declare_ffn(
|
||||
dense_tp_size=None,
|
||||
exit_rows=None,
|
||||
output_complete=False,
|
||||
attn_tp_gather=None,
|
||||
):
|
||||
"""Declare a dense or MoE FFN independently of its compute module.
|
||||
|
||||
@@ -396,8 +403,6 @@ def declare_ffn(
|
||||
the adjacent FFN kinds and TBO configuration.
|
||||
output_complete: Whether the compute completes its own output sum,
|
||||
fused with a reduction it needs anyway; the exit then owes none.
|
||||
attn_tp_gather: Implementation of this stage's gathers over attention
|
||||
TP, tried before the boundary's own (see StageDeclaration).
|
||||
|
||||
Returns:
|
||||
A StageDeclaration with no norm, tensors or execution plan.
|
||||
@@ -411,7 +416,6 @@ def declare_ffn(
|
||||
dense_tp_size=dense_tp_size,
|
||||
exit_rows=exit_rows or tbo_exit_rows(sparse, next_layer_sparse),
|
||||
output_complete=output_complete,
|
||||
attn_tp_gather=attn_tp_gather,
|
||||
)
|
||||
|
||||
|
||||
@@ -438,7 +442,10 @@ def _resolve_stage(stage, variant, following=None):
|
||||
output_complete=stage.output_complete,
|
||||
)
|
||||
exit_rows = resolve_exit_rows(stage.exit_rows)
|
||||
if exit_rows is ExitRows.ATTENTION:
|
||||
if exit_rows is ExitRows.ATTENTION or (
|
||||
following is not None
|
||||
and getattr(following.read, "reads_after_attn_tp_gather", False)
|
||||
):
|
||||
returned = attention
|
||||
elif exit_rows is ExitRows.SLICE:
|
||||
# The residual's rows during the FFN: this rank's slice for an FFN
|
||||
@@ -674,12 +681,13 @@ class _LayerStack:
|
||||
"""A layer stack under construction: what was appended, in order, and how
|
||||
to reach the layers other pipeline ranks hold on either side of it."""
|
||||
|
||||
__slots__ = ("appends", "previous_layers", "next_layers")
|
||||
__slots__ = ("appends", "previous_layers", "next_layers", "final_read")
|
||||
|
||||
def __init__(self, previous_layers=(), next_layers=()):
|
||||
def __init__(self, previous_layers=(), next_layers=(), final_read=None):
|
||||
self.appends = []
|
||||
self.previous_layers = previous_layers
|
||||
self.next_layers = next_layers
|
||||
self.final_read = final_read
|
||||
|
||||
|
||||
# The stack being built; layer_stack saves and restores an outer one.
|
||||
@@ -687,7 +695,7 @@ _stack: Optional[_LayerStack] = None
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def layer_stack(*, previous_layers=(), next_layers=()):
|
||||
def layer_stack(*, previous_layers=(), next_layers=(), final_read=None):
|
||||
"""Open a layer stack that append_stages extends in order.
|
||||
|
||||
Every stage binds when the stack closes, once its producer and its
|
||||
@@ -702,10 +710,15 @@ def layer_stack(*, previous_layers=(), next_layers=()):
|
||||
stack's first. What they build is discarded.
|
||||
next_layers: Likewise for the layers after this stack, whose first
|
||||
declared stage is the consumer of this stack's last.
|
||||
final_read: The model's final read of the stack's output, when it is
|
||||
not a plain final norm: the consumer of the last stage when no
|
||||
later layer declares one. Its ``attn_tp_gather`` gathers the rows
|
||||
that stage leaves on this rank's attention-TP slice, and with
|
||||
``reads_attn_tp_slices`` it reads that slice instead.
|
||||
"""
|
||||
global _stack
|
||||
outer = _stack
|
||||
stack = _LayerStack(previous_layers, next_layers)
|
||||
stack = _LayerStack(previous_layers, next_layers, final_read)
|
||||
_stack = stack
|
||||
try:
|
||||
yield stack
|
||||
@@ -714,6 +727,7 @@ def layer_stack(*, previous_layers=(), next_layers=()):
|
||||
stack.appends,
|
||||
previous=_neighbour_stage(stack.previous_layers, last=True),
|
||||
following=_neighbour_stage(stack.next_layers, last=False),
|
||||
final_read=stack.final_read,
|
||||
)
|
||||
finally:
|
||||
_stack = outer
|
||||
@@ -831,7 +845,7 @@ def _detached(declaration):
|
||||
return replace(declaration, previous=None, prepared_from=None)
|
||||
|
||||
|
||||
def _bind_stack(appends, *, previous, following):
|
||||
def _bind_stack(appends, *, previous, following, final_read=None):
|
||||
"""Bind every appended stage, in order, and fill in the boundaries each
|
||||
append returned."""
|
||||
if following is not None:
|
||||
@@ -857,21 +871,77 @@ def _bind_stack(appends, *, previous, following):
|
||||
sources[id(returned.declaration)] = boundary
|
||||
bound.append(boundaries)
|
||||
# The last stage's consumer is the next rank's first stage, if any;
|
||||
# without one it ends the model's layer stack.
|
||||
_bind_pending(chain, consumer=following, terminal=following is None)
|
||||
# without one it ends the model's layer stack, read by the final read.
|
||||
_bind_pending(
|
||||
chain,
|
||||
consumer=following,
|
||||
terminal=following is None,
|
||||
final_read=final_read if following is None else None,
|
||||
)
|
||||
_check_declared_gathers(appends, bound, remote_producer=previous is not None)
|
||||
for append, boundaries in zip(appends, bound):
|
||||
for returned, boundary in zip(append.boundaries, boundaries):
|
||||
returned.plan = boundary.plan
|
||||
returned.declaration = boundary.declaration
|
||||
|
||||
|
||||
def _bind_pending(chain: _Chain, *, consumer, terminal):
|
||||
def _check_declared_gathers(appends, bound, *, remote_producer):
|
||||
"""A stage that declares its gather over attention TP must have one: at
|
||||
its own entry, or at its producer's exit. A producer on another pipeline
|
||||
rank runs it there."""
|
||||
line = [
|
||||
(append.origin, boundary)
|
||||
for append, boundaries in zip(appends, bound)
|
||||
if append.prepared_from is None
|
||||
for boundary in boundaries
|
||||
]
|
||||
for (origin, consumer), producer in zip(line, [None, *(b for _, b in line)]):
|
||||
if consumer.declaration.attn_tp_gather is None or any(
|
||||
_gathers_input(path) for path in consumer.plan.paths.values()
|
||||
):
|
||||
continue
|
||||
if producer is None and remote_producer:
|
||||
continue
|
||||
if producer is not None and any(
|
||||
_gathers_output(path) for path in producer.plan.paths.values()
|
||||
):
|
||||
continue
|
||||
error = ValueError(
|
||||
"an attention-TP gather is declared, but neither this stage's entry "
|
||||
"nor its producer's exit gathers over attention TP"
|
||||
)
|
||||
_note_origin(error, origin)
|
||||
raise error
|
||||
|
||||
|
||||
def _gathers_input(path):
|
||||
return getattr(path.entry.input_move, "func", None) is attn_tp_gather_input
|
||||
|
||||
|
||||
def _gathers_output(path):
|
||||
return getattr(path.output_move, "func", None) is update_attn_tp_gather_output
|
||||
|
||||
|
||||
def _ended(declaration, final_read):
|
||||
"""The stack's last stage as the final read takes it: an FFN on this
|
||||
rank's attention-TP slice leaves its output there for a read that reads
|
||||
the slice and gathers what it read."""
|
||||
if declaration.kind is StageKind.FFN and getattr(
|
||||
final_read, "reads_attn_tp_slices", False
|
||||
):
|
||||
return replace(declaration, exit_rows=ExitRows.SLICE)
|
||||
return declaration
|
||||
|
||||
|
||||
def _bind_pending(chain: _Chain, *, consumer, terminal, final_read=None):
|
||||
pending = chain.pending
|
||||
if pending is None:
|
||||
return
|
||||
chain.pending = None
|
||||
try:
|
||||
declaration = replace(pending.declaration, terminal=terminal)
|
||||
if terminal:
|
||||
declaration = _ended(declaration, final_read)
|
||||
if pending.predecessor is None:
|
||||
incoming = _incoming(declaration)
|
||||
else:
|
||||
@@ -882,7 +952,12 @@ def _bind_pending(chain: _Chain, *, consumer, terminal):
|
||||
)
|
||||
outgoing = _connect(declaration, consumer, residual_from=incoming)
|
||||
bound = _bind_stage(
|
||||
declaration, pending.norm, incoming, outgoing, **pending.options
|
||||
declaration,
|
||||
pending.norm,
|
||||
incoming,
|
||||
outgoing,
|
||||
final_read=final_read,
|
||||
**pending.options,
|
||||
)
|
||||
except Exception as error:
|
||||
_note_origin(error, pending.origin)
|
||||
|
||||
@@ -70,6 +70,10 @@ class ResidualReadout(Protocol):
|
||||
is_plain_norm: Read is normalization/optional quantization without changing
|
||||
the residual, allowing compatible fused add+norm implementations.
|
||||
reads_before_dp_gather: Preserve this read on source rows before a DP gather.
|
||||
reads_after_attn_tp_gather: Optional. The read needs every row of the
|
||||
attention's, so it runs after any gather over attention TP: a
|
||||
producer that would leave its output on this rank's attention-TP
|
||||
slice returns it to the attention's rows instead.
|
||||
completing_fusions: Optional. ReadoutFusion kernels that complete the
|
||||
sum the input owes with the residual add ahead of this read, tried
|
||||
before the boundary's own all-reduce or reduce-scatter.
|
||||
|
||||
@@ -96,6 +96,9 @@ class AttnBankState:
|
||||
# A read on this rank's attention-TP slice first tries the bank's kernels
|
||||
# that also run the collective around it.
|
||||
fuses_slice_collectives: bool = False
|
||||
# The bank stays on each rank's attention-TP slice across layers;
|
||||
# otherwise every rank writes every row and the attention reads them all.
|
||||
reads_slices: bool = False
|
||||
|
||||
def _aggregate(
|
||||
self, contribution, residual, norm, *, score_proj, score_norm, write
|
||||
@@ -216,6 +219,10 @@ class _AttnReadout(_BankReadout):
|
||||
"""The attention input: the bank aggregation and this layer's input norm.
|
||||
A write layer snapshots the residual this read forms."""
|
||||
|
||||
@property
|
||||
def reads_after_attn_tp_gather(self):
|
||||
return not self.state.reads_slices
|
||||
|
||||
@property
|
||||
def gathering_reads(self):
|
||||
state = self.state
|
||||
@@ -269,20 +276,29 @@ class AttnBankOutputRead:
|
||||
side's scoring parameters, then the final norm. A callable final norm for
|
||||
`residual_batch.final_norm`.
|
||||
|
||||
A stack that ends on this rank's attention-TP slice of the rows is read
|
||||
there, against the bank rows this rank wrote, and what it read is
|
||||
gathered: by the bank's kernel that does both when it takes the batch,
|
||||
else by ``attn_tp_gather`` when that does, else over the attention-TP
|
||||
group."""
|
||||
With ``reads_attn_tp_slices``, the stack may end on this rank's
|
||||
attention-TP slice of the rows: such a batch is read there, against the
|
||||
bank rows this rank wrote, and what it read is gathered: by the bank's
|
||||
kernel that does both when it takes the batch, else by ``attn_tp_gather``
|
||||
when that does, else over the attention-TP group. Otherwise the stack's
|
||||
last FFN brings its output to the attention's rows, in ``attn_tp_gather``
|
||||
when that takes the batch."""
|
||||
|
||||
def __init__(
|
||||
self, bank: AttnBank, score_proj, score_norm, norm, attn_tp_gather=None
|
||||
self,
|
||||
bank: AttnBank,
|
||||
score_proj,
|
||||
score_norm,
|
||||
norm,
|
||||
attn_tp_gather=None,
|
||||
reads_attn_tp_slices=False,
|
||||
):
|
||||
self.bank = bank
|
||||
self.score_proj = score_proj
|
||||
self.score_norm = score_norm
|
||||
self.norm = norm
|
||||
self.attn_tp_gather = attn_tp_gather
|
||||
self.reads_attn_tp_slices = reads_attn_tp_slices
|
||||
|
||||
def __call__(self, hidden_states, residual=None):
|
||||
bank = self.bank.require()
|
||||
|
||||
@@ -44,6 +44,8 @@ class GatedResidualState:
|
||||
ffn_mix: The FFN stage's read.
|
||||
attn_combine: Writes the attention output back into the streams.
|
||||
ffn_combine: Writes the FFN output back into the streams.
|
||||
attn_reads_every_row: Whether attn_mix needs every row of the
|
||||
attention's, as a contribution computed for all of them does.
|
||||
"""
|
||||
|
||||
expand: Callable
|
||||
@@ -51,6 +53,7 @@ class GatedResidualState:
|
||||
ffn_mix: Callable
|
||||
attn_combine: Callable
|
||||
ffn_combine: Callable
|
||||
attn_reads_every_row: bool = False
|
||||
# Produced by a stage's read and consumed by that same stage's write-back.
|
||||
normed: Optional[torch.Tensor] = None
|
||||
|
||||
@@ -110,6 +113,10 @@ class _AttnReadout:
|
||||
def __init__(self, state: GatedResidualState):
|
||||
self.state = state
|
||||
|
||||
@property
|
||||
def reads_after_attn_tp_gather(self):
|
||||
return self.state.attn_reads_every_row
|
||||
|
||||
def init_residual(self, hidden_states):
|
||||
return self.state.expand(hidden_states)
|
||||
|
||||
|
||||
+104
-562
File diff suppressed because it is too large
Load Diff
@@ -37,7 +37,6 @@ from sglang.srt.layers.hyperconnection import (
|
||||
HyperConnectionConfig,
|
||||
)
|
||||
from sglang.srt.layers.layer_boundary import (
|
||||
ExitRows,
|
||||
GatedResidualState,
|
||||
append_stages,
|
||||
declare_attn,
|
||||
@@ -1434,13 +1433,7 @@ def _has_ple(layer_id, config) -> bool:
|
||||
return (layer_id + 1) in config.ple_layer_ids
|
||||
|
||||
|
||||
def _ffn_exit_rows(layer_id, config):
|
||||
"""A PLE layer's read adds an embedding computed for every row, so the FFN
|
||||
before it hands its output on as full attention rows."""
|
||||
return ExitRows.ATTENTION if _has_ple(layer_id + 1, config) else None
|
||||
|
||||
|
||||
def _build_qwen4_exp_stages(residual, *, sparse, layer_id, config):
|
||||
def _build_qwen4_exp_stages(residual, *, sparse):
|
||||
"""The attention and FFN stage boundaries of one gated hyper-connection layer.
|
||||
|
||||
Each read normalizes the streams itself, so neither stage binds a norm, and
|
||||
@@ -1458,7 +1451,6 @@ def _build_qwen4_exp_stages(residual, *, sparse, layer_id, config):
|
||||
next_layer_sparse=sparse,
|
||||
read=residual.ffn_readout,
|
||||
update=residual.ffn_update,
|
||||
exit_rows=_ffn_exit_rows(layer_id, config),
|
||||
),
|
||||
None,
|
||||
),
|
||||
@@ -1526,10 +1518,10 @@ class Qwen4ExpLayerExtensionMixin:
|
||||
ffn_mix=self.mlp_hyper_connection.mix,
|
||||
attn_combine=self.attn_hyper_connection.combine,
|
||||
ffn_combine=self.mlp_hyper_connection.combine,
|
||||
# The PLE embedding is computed for every row.
|
||||
attn_reads_every_row=self.ple is not None,
|
||||
).residual_ops(),
|
||||
sparse=isinstance(self.mlp, Qwen2MoeSparseMoeBlock),
|
||||
layer_id=layer_id,
|
||||
config=config,
|
||||
)
|
||||
|
||||
def _widen_streams(self, hidden_states: torch.Tensor) -> torch.Tensor:
|
||||
|
||||
@@ -1590,13 +1590,15 @@ def make_layers(
|
||||
prefix: str = "",
|
||||
return_tuple: bool = False,
|
||||
offloader_kwargs: Optional[Dict[str, Any]] = None,
|
||||
final_read: Optional[Any] = None,
|
||||
) -> Tuple[torch.nn.Module, int, int]:
|
||||
"""Make a list of layers with the given layer function.
|
||||
|
||||
The local layers are built inside one layer stack, so layers that declare
|
||||
stage boundaries connect in order without naming their neighbours. Across
|
||||
a pipeline stage boundary the stack learns the neighbouring stage from the
|
||||
layer itself, built again on the meta device.
|
||||
layer itself, built again on the meta device. ``final_read`` is the
|
||||
stack's terminal read when it is not a plain final norm (see layer_stack).
|
||||
"""
|
||||
# circular imports
|
||||
from sglang.srt.distributed import get_pp_indices
|
||||
@@ -1623,6 +1625,7 @@ def make_layers(
|
||||
with layer_stack(
|
||||
previous_layers=[neighbour(idx) for idx in reversed(range(start_layer))],
|
||||
next_layers=[neighbour(idx) for idx in range(end_layer, num_hidden_layers)],
|
||||
final_read=final_read,
|
||||
):
|
||||
modules = torch.nn.ModuleList(
|
||||
[PPMissingLayer(return_tuple=return_tuple) for _ in range(start_layer)]
|
||||
@@ -1649,6 +1652,7 @@ def make_pp_layers(
|
||||
prefix: str = "",
|
||||
return_tuple: bool = False,
|
||||
offloader_kwargs: Optional[Dict[str, Any]] = None,
|
||||
final_read: Optional[Any] = None,
|
||||
) -> Tuple[torch.nn.Module, int, int]:
|
||||
"""Make this pipeline stage's layers, and return them with the stage's range.
|
||||
|
||||
@@ -1663,6 +1667,7 @@ def make_pp_layers(
|
||||
prefix=prefix,
|
||||
return_tuple=return_tuple,
|
||||
offloader_kwargs=offloader_kwargs,
|
||||
final_read=final_read,
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -457,48 +457,73 @@ class TestUnpaddedBatches(CustomTestCase):
|
||||
self.assertNotIn(BatchVariant.UNPADDED, ffn.plan.paths)
|
||||
|
||||
|
||||
class TestSliceExitRows(CustomTestCase):
|
||||
"""An FFN on this rank's attention-TP slice that declares ExitRows.SLICE
|
||||
leaves its output there, the stack's last one too, where the final read
|
||||
then takes it; the next layer's attention reads it there and gathers what
|
||||
it read, in the declared gather when that takes the batch."""
|
||||
class TestRowsTheConsumerReads(CustomTestCase):
|
||||
"""An FFN on this rank's attention-TP slice leaves its output there unless
|
||||
the stage after it needs the attention's rows: a read that needs every row
|
||||
(reads_after_attn_tp_gather) has the FFN's exit gather them, in that
|
||||
stage's declared gather. At the stack's end the model's final read takes
|
||||
that place: one that reads the slice (reads_attn_tp_slices) takes the
|
||||
last FFN's output there, and its own gather serves the last FFN's exit."""
|
||||
|
||||
def build(self, exit_rows, *, disable_attn_tp_gather=False, gather=None):
|
||||
class EveryRowRead(NormQuantReadout):
|
||||
reads_after_attn_tp_gather = True
|
||||
|
||||
def build(
|
||||
self,
|
||||
*,
|
||||
final_read=None,
|
||||
disable_attn_tp_gather=False,
|
||||
gather=None,
|
||||
read=None,
|
||||
):
|
||||
parallel = fixture.parallel_of(
|
||||
attn_dp=1, attn_tp=2, disable_attn_tp_gather=disable_attn_tp_gather
|
||||
)
|
||||
attn_read = {} if read is None else dict(read=read)
|
||||
with fixture.planning(parallel, a2a=True, boundary_reduction="ar"):
|
||||
with layer_stack():
|
||||
with layer_stack(final_read=final_read):
|
||||
return [
|
||||
stage
|
||||
for idx in range(2)
|
||||
for stage in append_stages(
|
||||
(
|
||||
declare_attn(attn_tp_gather=gather if idx else None),
|
||||
declare_attn(
|
||||
attn_tp_gather=gather if idx else None, **attn_read
|
||||
),
|
||||
fixture.Norm(),
|
||||
),
|
||||
(
|
||||
declare_ffn(
|
||||
sparse=True,
|
||||
next_layer_sparse=True,
|
||||
exit_rows=exit_rows,
|
||||
),
|
||||
declare_ffn(sparse=True, next_layer_sparse=True),
|
||||
fixture.Norm(),
|
||||
),
|
||||
)
|
||||
]
|
||||
|
||||
def test_the_last_ffn_stays_on_its_slice(self):
|
||||
for exit_rows in (ExitRows.SLICE, None):
|
||||
with self.subTest(exit_rows=exit_rows):
|
||||
last = self.build(exit_rows)[-1].plan.paths[BatchVariant.ORDINARY]
|
||||
if exit_rows is ExitRows.SLICE:
|
||||
self.assertIs(last.output_move, keep_output)
|
||||
def test_the_last_ffn_stays_on_its_slice_for_a_read_of_the_slice(self):
|
||||
for reads_slices in (True, False):
|
||||
with self.subTest(reads_slices=reads_slices):
|
||||
final_read = SimpleNamespace(reads_attn_tp_slices=reads_slices)
|
||||
last = self.build(final_read=final_read)[-1]
|
||||
path = last.plan.paths[BatchVariant.ORDINARY]
|
||||
if reads_slices:
|
||||
self.assertIs(path.output_move, keep_output)
|
||||
else:
|
||||
self.assertIs(last.output_move.func, update_attn_tp_gather_output)
|
||||
self.assertIs(path.output_move.func, update_attn_tp_gather_output)
|
||||
|
||||
def test_the_last_ffn_gathers_in_the_final_read_gather(self):
|
||||
def gather(hidden_states):
|
||||
return None
|
||||
|
||||
last = self.build(final_read=SimpleNamespace(attn_tp_gather=gather))[-1]
|
||||
move = last.plan.paths[BatchVariant.ORDINARY].output_move
|
||||
self.assertIs(move.func, update_attn_tp_gather_output)
|
||||
self.assertIs(move.keywords["gather"], gather)
|
||||
|
||||
def test_an_unpadded_batch_leaves_the_last_ffn_on_the_attention_rows(self):
|
||||
last = self.build(ExitRows.SLICE, disable_attn_tp_gather=True)[-1]
|
||||
last = self.build(
|
||||
final_read=SimpleNamespace(reads_attn_tp_slices=True),
|
||||
disable_attn_tp_gather=True,
|
||||
)[-1]
|
||||
unpadded = last.plan.paths[BatchVariant.UNPADDED]
|
||||
self.assertEqual(unpadded.output.layout.sharded, frozenset())
|
||||
self.assertIs(unpadded.output_move, keep_output)
|
||||
@@ -507,10 +532,24 @@ class TestSliceExitRows(CustomTestCase):
|
||||
def gather(hidden_states):
|
||||
return None
|
||||
|
||||
_, _, attention, _ = self.build(ExitRows.SLICE, gather=gather)
|
||||
first_ffn, attention = self.build(gather=gather)[1:3]
|
||||
move = attention.plan.paths[BatchVariant.ORDINARY].entry.input_move
|
||||
self.assertIs(move.func, attn_tp_gather_input)
|
||||
self.assertIs(move.keywords["gather"], gather)
|
||||
path = first_ffn.plan.paths[BatchVariant.ORDINARY]
|
||||
self.assertIs(path.output_move, keep_output)
|
||||
|
||||
def test_a_read_of_every_row_has_the_ffn_gather_in_its_gather(self):
|
||||
def gather(hidden_states):
|
||||
return None
|
||||
|
||||
first_ffn, attention = self.build(gather=gather, read=self.EveryRowRead())[1:3]
|
||||
move = first_ffn.plan.paths[BatchVariant.ORDINARY].output_move
|
||||
self.assertIs(move.func, update_attn_tp_gather_output)
|
||||
self.assertIs(move.keywords["gather"], gather)
|
||||
entry = attention.plan.paths[BatchVariant.ORDINARY].entry
|
||||
self.assertIsNone(entry.input_move)
|
||||
self.assertEqual(entry.input_rows.sharded, frozenset())
|
||||
|
||||
def test_a_read_that_gathers_needs_a_plain_add_or_a_written_stream(self):
|
||||
def gather_read(hidden_states, residual, norm):
|
||||
@@ -536,7 +575,7 @@ class TestSliceExitRows(CustomTestCase):
|
||||
with self.subTest(update=type(update).__name__):
|
||||
with (
|
||||
fixture.planning(parallel, a2a=True, boundary_reduction="ar"),
|
||||
layer_stack(),
|
||||
layer_stack(final_read=SimpleNamespace(reads_attn_tp_slices=True)),
|
||||
):
|
||||
stages = [
|
||||
stage
|
||||
@@ -548,7 +587,6 @@ class TestSliceExitRows(CustomTestCase):
|
||||
sparse=True,
|
||||
next_layer_sparse=True,
|
||||
update=update,
|
||||
exit_rows=ExitRows.SLICE,
|
||||
),
|
||||
fixture.Norm(),
|
||||
),
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
"""A stack read through the attention-residual bank's stage reads computes
|
||||
what the decoder computed when its MLP folded the pending add itself."""
|
||||
"""A stack read through the attention-residual bank's stage reads matches a
|
||||
reference that drives AttnResidual directly, with each MLP adding the pending
|
||||
residual itself."""
|
||||
|
||||
import unittest
|
||||
from types import SimpleNamespace
|
||||
@@ -11,7 +12,6 @@ import torch
|
||||
from sglang.srt.layers import attn_residual
|
||||
from sglang.srt.layers.layer_boundary import (
|
||||
BatchVariant,
|
||||
ExitRows,
|
||||
ReadoutFusion,
|
||||
SumGroup,
|
||||
append_stages,
|
||||
@@ -85,7 +85,7 @@ class _Layer:
|
||||
def mlp(self, x):
|
||||
return torch.sin(x) * self.mlp_weight
|
||||
|
||||
def ops(self, bank):
|
||||
def ops(self, bank, *, reads_slices=False):
|
||||
return AttnBankState(
|
||||
bank,
|
||||
self.attn_proj,
|
||||
@@ -93,6 +93,7 @@ class _Layer:
|
||||
self.ffn_proj,
|
||||
self.ffn_score_norm,
|
||||
writes_block=self.writes_block,
|
||||
reads_slices=reads_slices,
|
||||
).residual_ops()
|
||||
|
||||
|
||||
@@ -100,9 +101,9 @@ LAYER_LIST = [_Layer(i) for i in range(LAYERS)]
|
||||
OUT_PROJ, OUT_SCORE_NORM, FINAL_NORM = _Proj(99), _Norm(99), _Norm(100)
|
||||
|
||||
|
||||
def _legacy(hidden, layers, bank_rows=None, *, final=True):
|
||||
"""The decoder's own order: the MLP adds the pending prefix to its output,
|
||||
so each layer hands on the whole head."""
|
||||
def _reference(hidden, layers, bank_rows=None, *, final=True):
|
||||
"""The reference order: the MLP adds the pending prefix to its output, so
|
||||
each layer hands on the whole head."""
|
||||
bank = attn_residual.AttnResidual(
|
||||
hidden, -(-(layers.stop) // BLOCK), block_residual=bank_rows
|
||||
)
|
||||
@@ -193,8 +194,8 @@ class TestAttnBankResidualOps(CustomTestCase):
|
||||
def assert_identical(self, got, want):
|
||||
torch.testing.assert_close(got, want, rtol=0, atol=0)
|
||||
|
||||
def test_the_stack_matches_the_decoders_own_order(self):
|
||||
want = _legacy(self.hidden, slice(0, LAYERS))
|
||||
def test_the_stack_matches_the_reference_order(self):
|
||||
want = _reference(self.hidden, slice(0, LAYERS))
|
||||
for read_after_write in (False, True):
|
||||
for writes_stream in ((), (1, 3), (LAYERS - 1,)):
|
||||
with self.subTest(
|
||||
@@ -209,7 +210,7 @@ class TestAttnBankResidualOps(CustomTestCase):
|
||||
self.assert_identical(got, want)
|
||||
|
||||
def test_a_pipeline_rank_continues_from_the_head_and_the_bank(self):
|
||||
want = _legacy(self.hidden, slice(0, LAYERS))
|
||||
want = _reference(self.hidden, slice(0, LAYERS))
|
||||
# Split before a layer that writes a block and before one that reads,
|
||||
# after a layer that adds the residual itself and after one that
|
||||
# leaves the add to the next read.
|
||||
@@ -223,12 +224,12 @@ class TestAttnBankResidualOps(CustomTestCase):
|
||||
writes_stream=writes_stream,
|
||||
final=False,
|
||||
)
|
||||
legacy_head, legacy_rows = _legacy(
|
||||
want_head, want_rows = _reference(
|
||||
self.hidden, slice(0, split), final=False
|
||||
)
|
||||
# The wire carries what the decoder's own order sent.
|
||||
self.assert_identical(head, legacy_head)
|
||||
self.assert_identical(bank_rows, legacy_rows)
|
||||
# The wire carries what the reference order sends.
|
||||
self.assert_identical(head, want_head)
|
||||
self.assert_identical(bank_rows, want_rows)
|
||||
got = _staged(
|
||||
head,
|
||||
slice(split, LAYERS),
|
||||
@@ -346,9 +347,9 @@ class TestAttnBankSpMoeStages(CustomTestCase):
|
||||
"""A latent MoE dispatched over an a2a backend with attention TP runs on
|
||||
this rank's shard of the rows: the FFN's entry reduce-scatters the
|
||||
attention output and slices the residual, and each MoE layer's exit
|
||||
gathers its stream back to every row, as the decoder did."""
|
||||
gathers its stream back to every row, which the bank's next read needs."""
|
||||
|
||||
def build(self, exit_rows):
|
||||
def build(self, *, reads_slices=False):
|
||||
holder = AttnBank()
|
||||
with (
|
||||
fixture.planning(
|
||||
@@ -371,12 +372,14 @@ class TestAttnBankSpMoeStages(CustomTestCase):
|
||||
sparse=True,
|
||||
next_layer_sparse=True,
|
||||
output_complete=True,
|
||||
exit_rows=exit_rows,
|
||||
),
|
||||
fixture.Norm(),
|
||||
),
|
||||
)[1]
|
||||
for ops in (LAYER_LIST[i].ops(holder) for i in range(3))
|
||||
for ops in (
|
||||
LAYER_LIST[i].ops(holder, reads_slices=reads_slices)
|
||||
for i in range(3)
|
||||
)
|
||||
]
|
||||
|
||||
def entry_step(self, ffn):
|
||||
@@ -384,7 +387,7 @@ class TestAttnBankSpMoeStages(CustomTestCase):
|
||||
return prepare.keywords["step"]
|
||||
|
||||
def test_each_moe_layer_returns_to_every_row(self):
|
||||
for ffn in self.build(ExitRows.ATTENTION):
|
||||
for ffn in self.build():
|
||||
step = self.entry_step(ffn)
|
||||
self.assertIs(step.func, comm_ops._attn_tp_reduce_scatter_update_read)
|
||||
self.assertTrue(step.keywords["scatters_residual"])
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
"""Kimi K3's own kernels for a layer's sums and gathers serve its
|
||||
attention-residual bank only: the standard residual path keeps the
|
||||
boundary's collectives, as the decoder did, even where those kernels run.
|
||||
boundary's collectives, even where those kernels run.
|
||||
With SGLANG_K3_SP_ATTN_RES the bank and the stream stay on each rank's shard
|
||||
across SP-MoE layers."""
|
||||
|
||||
@@ -16,7 +16,10 @@ from sglang.srt.layers.communication import k3_ar_fusion, k3_sp_collective
|
||||
from sglang.srt.layers.layer_boundary import BatchVariant, layer_stack
|
||||
from sglang.srt.layers.layer_boundary import prepare as comm_ops
|
||||
from sglang.srt.layers.layer_boundary.ops import attn_tp_gather_input, keep_output
|
||||
from sglang.srt.layers.layer_boundary.residual.attn_bank import AttnBank
|
||||
from sglang.srt.layers.layer_boundary.residual.attn_bank import (
|
||||
AttnBank,
|
||||
AttnBankOutputRead,
|
||||
)
|
||||
from sglang.srt.models import kimi_k3
|
||||
from sglang.test.ci.ci_register import register_cpu_ci
|
||||
from sglang.test.test_utils import CustomTestCase
|
||||
@@ -53,11 +56,26 @@ def moe_layer(*, bank, sp_moe, all_reduce_fusion):
|
||||
|
||||
def declare(layers, *, sp_moe, carries=False, config=CONFIG):
|
||||
"""Declare ``layers`` (index -> layer) in one layer stack, as if the
|
||||
stack carried the bank's shards (``carries``)."""
|
||||
stack carried the bank's shards (``carries``). A bank stack ends on the
|
||||
model's final read of the bank, which gathers an SP-MoE layer's rows in
|
||||
K3's tuned all-gather."""
|
||||
bank = AttnBank()
|
||||
final_read = (
|
||||
AttnBankOutputRead(
|
||||
bank,
|
||||
None,
|
||||
None,
|
||||
fixture.Norm(),
|
||||
attn_tp_gather=k3_sp_collective.all_gather if sp_moe else None,
|
||||
reads_attn_tp_slices=carries,
|
||||
)
|
||||
if any(layer.use_attn_residuals for layer in layers.values())
|
||||
else None
|
||||
)
|
||||
with (
|
||||
patch.object(k3_sp_collective, "enabled", return_value=True),
|
||||
patch.object(k3_ar_fusion, "enabled", return_value=True),
|
||||
patch.object(kimi_k3, "_shards_moe_rows", return_value=sp_moe),
|
||||
patch.object(kimi_k3, "_carries_bank_slices", return_value=carries),
|
||||
envs.SGLANG_K3_SP_ATTN_RES.override(carries),
|
||||
fixture.planning(
|
||||
@@ -65,7 +83,7 @@ def declare(layers, *, sp_moe, carries=False, config=CONFIG):
|
||||
a2a=sp_moe,
|
||||
boundary_reduction="ar",
|
||||
),
|
||||
layer_stack(),
|
||||
layer_stack(final_read=final_read),
|
||||
):
|
||||
for idx, layer in layers.items():
|
||||
layer._declare_stages(config, idx, bank)
|
||||
|
||||
@@ -9,14 +9,14 @@ from sglang.srt.layers.layer_boundary import layer_stack
|
||||
from sglang.srt.layers.layer_boundary.contracts import BatchVariant
|
||||
from sglang.srt.layers.layer_boundary.layout import TokenAxis
|
||||
from sglang.srt.layers.layer_boundary.residual.gated import GatedResidualState
|
||||
from sglang.srt.models.qwen4_exp import _build_qwen4_exp_stages
|
||||
from sglang.srt.models.qwen4_exp import _build_qwen4_exp_stages, _has_ple
|
||||
from sglang.test.ci.ci_register import register_cpu_ci
|
||||
from sglang.test.test_utils import CustomTestCase
|
||||
|
||||
register_cpu_ci(est_time=5, suite="base-a-test-cpu")
|
||||
|
||||
|
||||
def _residual_ops():
|
||||
def _residual_ops(*, attn_reads_every_row):
|
||||
def unused(*args, **kwargs):
|
||||
raise AssertionError("construction only")
|
||||
|
||||
@@ -26,6 +26,7 @@ def _residual_ops():
|
||||
ffn_mix=unused,
|
||||
attn_combine=unused,
|
||||
ffn_combine=unused,
|
||||
attn_reads_every_row=attn_reads_every_row,
|
||||
).residual_ops()
|
||||
|
||||
|
||||
@@ -45,7 +46,8 @@ class TestQwen4ExpPleRows(CustomTestCase):
|
||||
):
|
||||
layers = [
|
||||
_build_qwen4_exp_stages(
|
||||
_residual_ops(), sparse=True, layer_id=layer_id, config=config
|
||||
_residual_ops(attn_reads_every_row=_has_ple(layer_id, config)),
|
||||
sparse=True,
|
||||
)
|
||||
for layer_id in range(config.num_hidden_layers)
|
||||
]
|
||||
|
||||
@@ -158,9 +158,8 @@ class TestKimiAttentionParallelGroups(CustomTestCase):
|
||||
self.assertEqual(
|
||||
module.all_reduce_fusion, active_fusion
|
||||
)
|
||||
self.assertEqual(
|
||||
module.o_proj.reduce_results, not active_fusion
|
||||
)
|
||||
# The stage boundary completes the attention-TP sum.
|
||||
self.assertFalse(module.o_proj.reduce_results)
|
||||
self.assertEqual(
|
||||
module.o_proj.use_dp_attention_reduce,
|
||||
not active_fusion,
|
||||
|
||||
@@ -20,14 +20,13 @@ from sglang.test.test_utils import CustomTestCase
|
||||
register_cpu_ci(est_time=15, stage="weekly", runner_config="cpu")
|
||||
|
||||
|
||||
def build_mlp(group=None, *, width=16, quant_config=None, reduce=True):
|
||||
def build_mlp(group=None, *, width=16, quant_config=None):
|
||||
kwargs = {} if group is None else dict(parallel_group=group)
|
||||
return kimi_k3.KimiK3MLP(
|
||||
width,
|
||||
2 * width,
|
||||
"silu",
|
||||
quant_config=quant_config,
|
||||
reduce_results=reduce,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
@@ -128,64 +127,60 @@ class TestKimiMLPParallelGroups(CustomTestCase):
|
||||
"replicated",
|
||||
"shared_experts_tp",
|
||||
):
|
||||
for reduce in (False, True):
|
||||
module = build_mlp(group, reduce=reduce)
|
||||
auto_attn = group is None and dense_attn and dp > 1
|
||||
r, s = (
|
||||
(0, 1)
|
||||
if group == "replicated"
|
||||
else (rank % 2, 2)
|
||||
if group == "shared_experts_tp"
|
||||
else (rank % (4 // dp), 4 // dp)
|
||||
if group == "attn_tp" or auto_attn
|
||||
else (rank, 4)
|
||||
)
|
||||
self.assertEqual(module._dense_attn_tp, auto_attn)
|
||||
for name in ("gate_up_proj", "down_proj"):
|
||||
layer = getattr(module, name)
|
||||
self.assertEqual(rank_size(layer), (r, s))
|
||||
expected, _ = load_projection(layer)
|
||||
torch.testing.assert_close(layer.weight, expected)
|
||||
x = (
|
||||
(
|
||||
torch.arange(2 * expected.shape[1]).reshape(
|
||||
2, -1
|
||||
)
|
||||
% 17
|
||||
- 8
|
||||
module = build_mlp(group)
|
||||
auto_attn = group is None and dense_attn and dp > 1
|
||||
r, s = (
|
||||
(0, 1)
|
||||
if group == "replicated"
|
||||
else (rank % 2, 2)
|
||||
if group == "shared_experts_tp"
|
||||
else (rank % (4 // dp), 4 // dp)
|
||||
if group == "attn_tp" or auto_attn
|
||||
else (rank, 4)
|
||||
)
|
||||
self.assertEqual(module._dense_attn_tp, auto_attn)
|
||||
for name in ("gate_up_proj", "down_proj"):
|
||||
layer = getattr(module, name)
|
||||
self.assertEqual(rank_size(layer), (r, s))
|
||||
expected, _ = load_projection(layer)
|
||||
torch.testing.assert_close(layer.weight, expected)
|
||||
x = (
|
||||
(
|
||||
torch.arange(2 * expected.shape[1]).reshape(
|
||||
2, -1
|
||||
)
|
||||
/ 128
|
||||
).to(layer.weight.dtype)
|
||||
tp, attn = Mock(), Mock()
|
||||
tp.all_reduce.side_effect = lambda v: v * 4
|
||||
attn.all_reduce.side_effect = lambda v: (
|
||||
v * (4 // dp)
|
||||
% 17
|
||||
- 8
|
||||
)
|
||||
row = name == "down_proj"
|
||||
reference = F.linear(x, expected)
|
||||
if row and reduce and s > 1:
|
||||
reference *= (4 // dp) if auto_attn else 4
|
||||
with (
|
||||
parallel_scope(tp_group=tp, attn_tp_group=attn),
|
||||
patch.object(
|
||||
linear,
|
||||
"use_symmetric_memory",
|
||||
return_value=nullcontext(),
|
||||
) as allocation,
|
||||
):
|
||||
actual = layer(x)[0]
|
||||
torch.testing.assert_close(actual, reference)
|
||||
if row:
|
||||
self.assertEqual(layer.reduce_results, reduce)
|
||||
self.assertEqual(
|
||||
layer.use_dp_attention_reduce, auto_attn
|
||||
)
|
||||
self.assertIs(
|
||||
allocation.call_args.args[0],
|
||||
attn if auto_attn else tp,
|
||||
)
|
||||
else:
|
||||
allocation.assert_not_called()
|
||||
/ 128
|
||||
).to(layer.weight.dtype)
|
||||
tp, attn = Mock(), Mock()
|
||||
tp.all_reduce.side_effect = lambda v: v * 4
|
||||
attn.all_reduce.side_effect = lambda v: v * (4 // dp)
|
||||
row = name == "down_proj"
|
||||
reference = F.linear(x, expected)
|
||||
with (
|
||||
parallel_scope(tp_group=tp, attn_tp_group=attn),
|
||||
patch.object(
|
||||
linear,
|
||||
"use_symmetric_memory",
|
||||
return_value=nullcontext(),
|
||||
) as allocation,
|
||||
):
|
||||
actual = layer(x)[0]
|
||||
torch.testing.assert_close(actual, reference)
|
||||
if row:
|
||||
# The stage boundary completes the sum.
|
||||
self.assertFalse(layer.reduce_results)
|
||||
self.assertEqual(
|
||||
layer.use_dp_attention_reduce, auto_attn
|
||||
)
|
||||
self.assertIs(
|
||||
allocation.call_args.args[0],
|
||||
attn if auto_attn else tp,
|
||||
)
|
||||
else:
|
||||
allocation.assert_not_called()
|
||||
|
||||
def test_moe_shared_consumer_preserves_selection_and_execution_handle(self):
|
||||
for rank in range(4):
|
||||
|
||||
Reference in New Issue
Block a user