[Refactor] Declare pipeline neighbours from a shared declaration function (#43459)
This commit is contained in:
@@ -122,7 +122,7 @@ with layer_stack():
|
||||
|
||||
Every stage binds when the stack closes, once its producer and its consumer are known: the stage appended before it and the one appended after it. The last stage ends the model's layer stack and is marked terminal, unless a later layer declares a stage. A boundary's `declaration` is usable at once; when the stack closes, it is replaced by the declaration with its place in the stack (its `previous` and `terminal`), and the plan is bound. An error raised while binding carries a note naming the `append_stages()` call it came from. A terminal stage leaves no work for a following layer, except the final-norm finalize handoff described in [Enter and leave the layer stack](#enter-and-leave-the-layer-stack). A NextN/MTP draft module is its own stack: its layer enters the stack and ends it, even when it reuses the target's decoder class.
|
||||
|
||||
Pipeline parallelism splits one layer stack across ranks, so the first and last local stages have neighbours that another rank builds. `make_layers()` finds them without communicating: when its stack closes, it builds the layer before the rank's first local layer and the one after its last again, with the model's own layer function inside `building_neighbour_layer()` (on the meta device), reads the stages they declare and discards them. A stage's declaration depends only on the config and the global layer index, so both ranks see the same boundary. A layer that declares no stage is passed over in favour of the next one out. These neighbours are built only if the stack appended stages, and after the local layers. A neighbour is never loaded or run, so a layer constructor must run on the meta device, must skip the host and device resources only a running layer needs (tables, streams, engines, communicators) when `is_building_neighbour_layer()` is true, and must not depend on which pipeline rank builds it. The meta device alone does not stop a constructor that names a device explicitly.
|
||||
Pipeline parallelism splits one layer stack across ranks, so the first and last local stages have neighbours that another rank builds. The model gives `make_layers()` (or `make_pp_layers()`) its shared declaration function, `stage_facts`: given a global layer index, it returns the stages that layer declares, in order (an empty tuple for a layer that declares none), from the config, the layer index, construction options known before the layer is built, and the fixed parallel topology, never from built modules or provider state. The layer declares its own stages with the same function, so both ranks derive a boundary from one source; under pipeline parallelism `make_layers()` also checks each layer it builds against the function. When the stack closes, it asks the function for the stages next to it, nearest layer first, passing over a layer that declares none, and keeps only what binding reads of them (`facts_of()`: the capabilities a read, update and output transform declare, not the objects). A model that does not give the function yet is on a transitional list: its neighbours are built again with the model's own layer function inside `building_neighbour_layer()` (on the meta device), which must then skip the host and device resources only a running layer needs when `is_building_neighbour_layer()` is true and must not depend on which pipeline rank builds it. A model off that list must give the function to run under pipeline parallelism.
|
||||
|
||||
Each item may include a third element containing constructor options: an attention stage accepts `qkv_latent_func` (for example `{"qkv_latent_func": self.self_attn.prepare_qkv_latent}`) and `fusions`; an FFN stage accepts only `fusions` (see [Fusion providers](#fusion-providers-and-output-decisions)). An option a stage does not take is rejected when appended. Pass declarations fresh from `declare_attn()` / `declare_ffn()`: one taken from a bound boundary already has its place in a stack, and is rejected.
|
||||
|
||||
@@ -251,7 +251,7 @@ Required attention and MoE collectives are unaffected. These are permissions, no
|
||||
Before you open a pull request for a new model, check each item:
|
||||
|
||||
- Every decoder layer appends its stages inside a layer stack: `make_layers()`, or `layer_stack()` around any other construction, including a NextN/MTP draft's own layer.
|
||||
- The layer constructor runs on the meta device, allocates nothing when `is_building_neighbour_layer()` is true, and declares the same stages whichever pipeline rank builds it: `make_layers()` builds the layers next to a rank's own that way to read their stages.
|
||||
- The model gives `make_layers()` its shared declaration function (`stage_facts`), and each layer declares its stages with it: another pipeline rank binds the stages next to its own from that function alone.
|
||||
- The model forward calls `residual_batch.start()` before the first layer, and leaves the stack through `residual_batch.final_norm()`, `to_pp()`, or `take_output()`.
|
||||
- Pipeline reception calls `from_pp(tensors, forward_batch)` on the first local boundary.
|
||||
- Split prefill starts the stream only in the segment that runs the first layer and completes it only in the final segment.
|
||||
@@ -332,6 +332,18 @@ Configuration validation rejects unsupported model/parallel combinations. Bounda
|
||||
|
||||
### Fusion providers and output decisions
|
||||
|
||||
Each exit has its own policy, and what it decides from fixed facts (the parallel configuration, the declarations and the bound path) is decided when the plan is built and recorded on the path as `ExitFacts`; only what depends on the batch is decided for it:
|
||||
|
||||
| Exit | Entry point | May decide |
|
||||
|---|---|---|
|
||||
| FFN | `exit()` or `finish()` | Complete the sum now, possibly in an attention-DP or CP / input-scattered reduce-scatter on the way back; leave it to the next input (an all-reduce, or under attention DP the scatter back to this rank's tokens); hand a MoE finalize to the next input |
|
||||
| Operation-scheduled (TBO) | `complete_now()` | Only complete it now: under attention DP the move back is a plain scatter; never defers or hands off |
|
||||
| Mixer | `finish()` | Leave its declared sum to the next stage's input (before an FFN); carry it to the next mixer's input (without attention DP); or complete it |
|
||||
| Stack end | `exit()` or `finish()` | Complete it now; a MoE finalize handoff may reach a final norm that accepts it |
|
||||
| Pipeline handoff | `exit()` or `finish()`, then `to_pp()` | Complete it now; an FFN on its own rows also writes it into the residual |
|
||||
|
||||
Fixed per path: whether an FFN may leave its sum to the next input at all, whether a reduce-scatter completes it without an attention-DP one, which sum an input-scattered batch's next input completes, whether the attention-DP return is the reduce-scatterv, whether the post-expert sum is one all-reduce, and whether a mixer carries its sum. Decided per batch: its padding, whether the attention-DP reduce-scatter is usable (an elastic EP scale-up changes the DP group), whether it has tokens, and whether a fused consumer takes the sum.
|
||||
|
||||
An output decision is made once. Its flags control whether compute performs reduction/finalize work; the matching completion action either executes that work or records exactly what the next boundary owes. Do not set a skip flag independently of a completion action.
|
||||
|
||||
A backend fusion provider is passed per stage as the `fusions` option, `(declaration, norm, {"fusions": provider})`; give a layer's attention and FFN stages the same provider. The boundary calls it as follows:
|
||||
@@ -342,7 +354,7 @@ A backend fusion provider is passed per stage as the `fusions` option, `(declara
|
||||
|
||||
`install_cutedsl_fusion()` and `prepare_cutedsl_fusion()` find providers through the decoder layer's `attn_boundary` and `ffn_boundary` attributes.
|
||||
|
||||
Consumer fusion candidates are ordered. A candidate returning `None` must do so before modifying inputs or starting a collective, so fallback remains valid. `FfnInputFusion` declares the sum group it completes with residual add and norm. A residual read may supply its own candidates as `completing_fusions`: each `ReadoutFusion` completes a sum with the residual add, ahead of the read, and is tried before the built-in collective; one with `reads=True` also does the read. Where a stage reads its input on this rank's attention-TP slice and then gathers it, a read's `gathering_reads` may do both in one kernel. A stage may declare its own implementation of the gather over attention TP that brings rows into it (`declare_attn(attn_tp_gather=...)`), tried on every batch before the built-in gather: at its own entry, or at its producer's exit, since an exit runs its consumer's gather. Across a pipeline boundary the producer's rank takes that gather from the neighbouring layer it builds on the meta device, so the gather may use only the communication state of the rank it runs on, not the declaring layer's weights or buffers. The consumer only tries compatible candidates for the declared update/read capabilities.
|
||||
Consumer fusion candidates are ordered. A candidate returning `None` must do so before modifying inputs or starting a collective, so fallback remains valid. `FfnInputFusion` declares the sum group it completes with residual add and norm. A residual read may supply its own candidates as `completing_fusions`: each `ReadoutFusion` completes a sum with the residual add, ahead of the read, and is tried before the built-in collective; one with `reads=True` also does the read. Where a stage reads its input on this rank's attention-TP slice and then gathers it, a read's `gathering_reads` may do both in one kernel. A stage may declare its own implementation of the gather over attention TP that brings rows into it (`declare_attn(attn_tp_gather=...)`), tried on every batch before the built-in gather: at its own entry, or at its producer's exit, since an exit runs its consumer's gather. Across a pipeline boundary the producer's rank takes that gather from the neighbouring stage's declaration, so the gather may use only the communication state of the rank it runs on, not the declaring layer's weights or buffers. The consumer only tries compatible candidates for the declared update/read capabilities.
|
||||
|
||||
An `OutputTransform` declares contribution processing before residual update. Its explicit `before_reduce_scatter` setting preserves a supported implementation's order; it is not permission to commute arbitrary operations across a reduction.
|
||||
|
||||
|
||||
@@ -24,7 +24,6 @@ from sglang.srt.layers.layer_boundary.boundary import (
|
||||
)
|
||||
from sglang.srt.layers.layer_boundary.construction import (
|
||||
BatchVariant,
|
||||
VariantEdges,
|
||||
)
|
||||
from sglang.srt.layers.layer_boundary.contracts import (
|
||||
EdgeContract,
|
||||
@@ -35,11 +34,10 @@ from sglang.srt.layers.layer_boundary.contracts import (
|
||||
OutputContract,
|
||||
ProducerReduction,
|
||||
ReadoutFusion,
|
||||
StageContract,
|
||||
StageKind,
|
||||
StagePath,
|
||||
)
|
||||
from sglang.srt.layers.layer_boundary.exit import ExitDecision, FfnExit, MixerExit
|
||||
from sglang.srt.layers.layer_boundary.exit import MixerExit
|
||||
from sglang.srt.layers.layer_boundary.factories import (
|
||||
append_stages,
|
||||
declare_attn,
|
||||
@@ -52,8 +50,6 @@ from sglang.srt.layers.layer_boundary.layout import (
|
||||
TokenAxis,
|
||||
batch_gathers_over_moe_cp,
|
||||
is_dense_ffn_fully_dp,
|
||||
moe_gathers_over_moe_cp,
|
||||
token_axis_sizes,
|
||||
)
|
||||
from sglang.srt.layers.layer_boundary.ops import (
|
||||
move_rows,
|
||||
@@ -64,11 +60,9 @@ from sglang.srt.layers.layer_boundary.output import (
|
||||
UnreducedOutput,
|
||||
complete_owed,
|
||||
)
|
||||
from sglang.srt.layers.layer_boundary.residual import LayerResidualOps
|
||||
from sglang.srt.layers.layer_boundary.residual.add_norm import (
|
||||
FUSE_ALLREDUCE_MAX_BATCH_SIZE,
|
||||
NORM_QUANT_READOUT,
|
||||
NORM_READOUT,
|
||||
PLAIN_ADD,
|
||||
PLAIN_RESIDUAL_OPS,
|
||||
)
|
||||
@@ -94,21 +88,16 @@ __all__ = [
|
||||
"ExitRows",
|
||||
"ProducerReduction",
|
||||
"FUSE_ALLREDUCE_MAX_BATCH_SIZE",
|
||||
"ExitDecision",
|
||||
"FfnExit",
|
||||
"FfnInputFusion",
|
||||
"ReadoutFusion",
|
||||
"DeferredFinalize",
|
||||
"GatedResidualState",
|
||||
"IHCState",
|
||||
"LayerResidualOps",
|
||||
"Layout",
|
||||
"MHCState",
|
||||
"MixerExit",
|
||||
"NORM_QUANT_READOUT",
|
||||
"NORM_READOUT",
|
||||
"PLAIN_RESIDUAL_OPS",
|
||||
"StageContract",
|
||||
"EntryPath",
|
||||
"InputContract",
|
||||
"StageKind",
|
||||
@@ -123,8 +112,6 @@ __all__ = [
|
||||
"batch_gathers_over_moe_cp",
|
||||
"move_rows",
|
||||
"complete_owed",
|
||||
"moe_gathers_over_moe_cp",
|
||||
"tbo_split_moves",
|
||||
"token_axis_sizes",
|
||||
"tp_reduce_scatter",
|
||||
]
|
||||
|
||||
@@ -235,9 +235,6 @@ class StagePlan:
|
||||
return _bound_for(self._next_input_rows, self.variant_for(forward_batch))
|
||||
return self.path_for(forward_batch).entry.input_rows
|
||||
|
||||
def produced(self, forward_batch):
|
||||
return self.path_for(forward_batch).output
|
||||
|
||||
@cached_property
|
||||
def output(self):
|
||||
return ExitPolicy(self)
|
||||
|
||||
@@ -27,7 +27,9 @@ from sglang.srt.layers.layer_boundary.residual.add_norm import NORM_READOUT, PLA
|
||||
|
||||
|
||||
class ProducerReduction(Enum):
|
||||
"""Who completes the sum a stage's output owes; compute never does.
|
||||
"""Who completes the sum a stage's output owes: exactly one completer for
|
||||
each output, the boundary. Compute completes it only where an FFN fuses
|
||||
it with a reduction its computation needs (``output_complete``).
|
||||
|
||||
ALWAYS_PARTIAL is attention-only: finish() hands the partial sum to the
|
||||
next stage's input as its declared sum. EXIT_SCOPED, for an FFN or a
|
||||
|
||||
@@ -5,9 +5,9 @@ from __future__ import annotations
|
||||
import contextlib
|
||||
import os
|
||||
import sys
|
||||
from dataclasses import dataclass, replace
|
||||
from dataclasses import dataclass, fields, is_dataclass, replace
|
||||
from types import MappingProxyType
|
||||
from typing import Callable, Mapping, NamedTuple, Optional
|
||||
from typing import Callable, Mapping, NamedTuple, Optional, Tuple
|
||||
|
||||
from sglang.srt.environ import envs
|
||||
from sglang.srt.layers import layernorm_sp
|
||||
@@ -29,6 +29,7 @@ from sglang.srt.layers.layer_boundary.contracts import (
|
||||
StageContract,
|
||||
StageKind,
|
||||
)
|
||||
from sglang.srt.layers.layer_boundary.facts import facts_of
|
||||
from sglang.srt.layers.layer_boundary.layout import (
|
||||
Layout,
|
||||
SumGroup,
|
||||
@@ -301,11 +302,14 @@ class StageDeclaration:
|
||||
contiguous slice of the rows, it returns them all, in rank order,
|
||||
or None to leave the gather to the boundary. Called on every batch,
|
||||
so it must be CUDA-graph safe. Across a pipeline boundary the
|
||||
producer's rank takes it from the neighbouring layer it builds on
|
||||
the meta device, so it may use only the communication state of the
|
||||
rank it runs on, not the declaring layer's weights or buffers.
|
||||
previous: Declaration whose output this stage consumes, as the stack
|
||||
records it; across pipeline ranks it is built locally.
|
||||
producer's rank takes it from the neighbouring stage's declaration
|
||||
(the model's shared declaration function, or, for a model on the
|
||||
transitional list, the layer built again on the meta device), so it
|
||||
may use only the communication state of the rank it runs on, not
|
||||
the declaring layer's weights or buffers.
|
||||
previous: The stage whose output this stage consumes, as the stack
|
||||
records it: only the facts binding reads of it (see facts_of),
|
||||
on this rank or another.
|
||||
prepared_from: Declaration whose already-read input a branch reuses.
|
||||
Mutually exclusive with previous; avoids a second update/read.
|
||||
|
||||
@@ -738,6 +742,19 @@ def _connect_line(line, origins, *, before=None, after=None, arrivals=None):
|
||||
if arrivals is None:
|
||||
arrivals = {v: _stack_arrival(v) for v in variants}
|
||||
if before is not None:
|
||||
if (
|
||||
before.output_transform is not None
|
||||
and before.kind is StageKind.ATTENTION
|
||||
and before.reduction is ProducerReduction.ALWAYS_PARTIAL
|
||||
):
|
||||
# Its transform runs at the next stage's input, which would be
|
||||
# on this rank, without the module that declares it.
|
||||
error = NotImplementedError(
|
||||
"a pipeline rank that ends on an attention transforming the "
|
||||
"output whose sum it leaves"
|
||||
)
|
||||
_note_origin(error, origins[0])
|
||||
raise error
|
||||
flow = resolve(0, before, arrivals, line[0])
|
||||
arrivals = {v: _arrival(before, flow[v], v) for v in variants}
|
||||
flows = []
|
||||
@@ -830,11 +847,14 @@ def layer_stack(*, previous_layers=(), next_layers=(), final_read=None):
|
||||
stage ends the model's layer stack unless a later layer declares a stage.
|
||||
|
||||
Args:
|
||||
previous_layers: Callables that build, nearest first, the layers before
|
||||
this stack that another pipeline rank holds. Called only if this
|
||||
stack appended stages, after its own layers are built, until one of
|
||||
them declares a stage: its last stage is the producer of this
|
||||
stack's first. What they build is discarded.
|
||||
previous_layers: For the layers before this stack that another
|
||||
pipeline rank holds, nearest first, callables that return the
|
||||
stages the layer declares as DeclaredStages (from the model's
|
||||
shared declaration function), or else build the layer so that it
|
||||
appends them. Called only if this stack appended stages, after its
|
||||
own layers are built, until one of them declares a stage: its last
|
||||
stage is the producer of this stack's first. Only what binding
|
||||
reads of that stage is kept (see facts_of).
|
||||
next_layers: Likewise for the layers after this stack, whose first
|
||||
declared stage is the consumer of this stack's last.
|
||||
final_read: The model's final read of the stack's output (a
|
||||
@@ -861,21 +881,94 @@ def layer_stack(*, previous_layers=(), next_layers=(), final_read=None):
|
||||
_stack = outer
|
||||
|
||||
|
||||
def _neighbour_stage(build_layers, *, last):
|
||||
class DeclaredStages(NamedTuple):
|
||||
"""The stages a neighbouring layer declares, from the model's shared
|
||||
declaration function, which stand in for building the layer."""
|
||||
|
||||
stages: Tuple[StageDeclaration, ...]
|
||||
|
||||
|
||||
def check_declared_stages(stack, appended: int, expected, where: str) -> None:
|
||||
"""Check that what a layer appended to ``stack`` since it held
|
||||
``appended`` appends is what its shared declaration function says it
|
||||
declares, as far as binding reads it: another pipeline rank binds this
|
||||
layer's stages from that function alone."""
|
||||
declared = [
|
||||
facts_of(declaration)
|
||||
for append in stack.appends[appended:]
|
||||
if append.prepared_from is None
|
||||
for declaration in append.declarations
|
||||
]
|
||||
expected = [facts_of(declaration) for declaration in expected]
|
||||
if declared != expected:
|
||||
raise ValueError(
|
||||
f"{where} declares stages its shared declaration function does "
|
||||
f"not: {_first_difference(declared, expected)}"
|
||||
)
|
||||
|
||||
|
||||
def _first_difference(declared, expected) -> str:
|
||||
"""Where the stages a layer declares first differ from those its shared
|
||||
declaration function gives, field by field."""
|
||||
if len(declared) != len(expected):
|
||||
return (
|
||||
f"it declares {[stage.kind.name for stage in declared]}, the "
|
||||
f"function {[stage.kind.name for stage in expected]}"
|
||||
)
|
||||
index, (stage, given) = next(
|
||||
(index, pair)
|
||||
for index, pair in enumerate(zip(declared, expected))
|
||||
if pair[0] != pair[1]
|
||||
)
|
||||
return f"stage {index} ({stage.kind.name}): " + "; ".join(
|
||||
f"{name} is {value!r}, the function says {given_value!r}"
|
||||
for name, value, given_value in _differing_fields(stage, given)
|
||||
)
|
||||
|
||||
|
||||
def _differing_fields(value, given, prefix=""):
|
||||
"""(dotted name, value, given value) for each leaf field that differs,
|
||||
descending into declarations and the facts they hold."""
|
||||
if is_dataclass(value):
|
||||
names = [field.name for field in fields(value)]
|
||||
else:
|
||||
names = getattr(type(value), "__struct_fields__", None)
|
||||
if names is None or type(value) is not type(given):
|
||||
return [(prefix.rstrip("."), value, given)]
|
||||
return [
|
||||
difference
|
||||
for name in names
|
||||
if getattr(value, name) != getattr(given, name)
|
||||
for difference in _differing_fields(
|
||||
getattr(value, name), getattr(given, name), f"{prefix}{name}."
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def _neighbour_stage(layers, *, last):
|
||||
"""The stage a neighbouring layer declares next to this stack: the last
|
||||
one of the nearest layer before it, or the first of the nearest after.
|
||||
Branches are side paths, so they never stand next to the stack."""
|
||||
global _stack
|
||||
for build_layer in build_layers:
|
||||
for layer in layers:
|
||||
outer = _stack
|
||||
_stack = _LayerStack()
|
||||
try:
|
||||
build_layer()
|
||||
appends = [a for a in _stack.appends if a.prepared_from is None]
|
||||
result = layer()
|
||||
declared = (
|
||||
result.stages
|
||||
if isinstance(result, DeclaredStages)
|
||||
else [
|
||||
declaration
|
||||
for append in _stack.appends
|
||||
if append.prepared_from is None
|
||||
for declaration in append.declarations
|
||||
]
|
||||
)
|
||||
finally:
|
||||
_stack = outer
|
||||
if appends:
|
||||
return appends[-1].declarations[-1] if last else appends[0].declarations[0]
|
||||
if declared:
|
||||
return facts_of(declared[-1] if last else declared[0])
|
||||
return None
|
||||
|
||||
|
||||
@@ -938,15 +1031,6 @@ def _return_before_trailing_attention(line):
|
||||
return
|
||||
|
||||
|
||||
def _detached(declaration):
|
||||
"""The declaration as the stage after it records it: without its own
|
||||
history, so a bound stage holds its producer's facts and not the chain of
|
||||
declarations before it."""
|
||||
if declaration is None:
|
||||
return None
|
||||
return replace(declaration, previous=None, prepared_from=None)
|
||||
|
||||
|
||||
def _bind_stack(appends, *, previous, following, final_read=None):
|
||||
"""Bind every appended stage and fill in the boundaries each append
|
||||
returned.
|
||||
@@ -973,13 +1057,13 @@ def _bind_stack(appends, *, previous, following, final_read=None):
|
||||
bound = {id(append): [None] * len(append.bindings) for append in appends}
|
||||
# A returned declaration's stage as bound, for the branches that start from it.
|
||||
sources = {}
|
||||
producer = _detached(_handed_off(previous))
|
||||
producer = facts_of(_handed_off(previous))
|
||||
chained = []
|
||||
for append, index in stages:
|
||||
chained.append(
|
||||
replace(append.declarations[index], previous=producer, prepared_from=None)
|
||||
)
|
||||
producer = _detached(chained[-1])
|
||||
producer = facts_of(chained[-1])
|
||||
connections = _connect_line(
|
||||
chained,
|
||||
[append.origin for append, _ in stages],
|
||||
@@ -1010,8 +1094,8 @@ def _bind_stack(appends, *, previous, following, final_read=None):
|
||||
chained.append(
|
||||
replace(
|
||||
declaration,
|
||||
previous=chained[-1] if chained else None,
|
||||
prepared_from=None if chained else source_declaration,
|
||||
previous=facts_of(chained[-1]) if chained else None,
|
||||
prepared_from=None if chained else facts_of(source_declaration),
|
||||
)
|
||||
)
|
||||
try:
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
# Copyright 2023-2024 SGLang Team
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
# ==============================================================================
|
||||
"""What binding reads of a stage it does not run: the stage before or after
|
||||
a stack on another pipeline rank, and the producer a bound stage records.
|
||||
|
||||
A stage's read, update and output transform are objects that hold the
|
||||
layer's modules and state. Binding a neighbouring stage reads only the
|
||||
capabilities they declare, so a neighbour, and the producer a stage keeps,
|
||||
is recorded as those values alone: no module, bound method or declaration
|
||||
history stays referenced once the stack is bound."""
|
||||
|
||||
from typing import Optional
|
||||
|
||||
import msgspec
|
||||
|
||||
|
||||
class UpdateFacts(msgspec.Struct, frozen=True):
|
||||
"""The capabilities a ResidualUpdate declares (see ResidualUpdate)."""
|
||||
|
||||
is_plain_add: bool
|
||||
applied_at_exit: bool
|
||||
outlives_layer: bool
|
||||
writes_stream: bool
|
||||
quantized_sum: bool = False
|
||||
|
||||
@classmethod
|
||||
def of(cls, update) -> "UpdateFacts":
|
||||
if isinstance(update, cls):
|
||||
return update
|
||||
return cls(
|
||||
is_plain_add=update.is_plain_add,
|
||||
applied_at_exit=update.applied_at_exit,
|
||||
outlives_layer=update.outlives_layer,
|
||||
writes_stream=update.writes_stream,
|
||||
quantized_sum=update.quantized_sum,
|
||||
)
|
||||
|
||||
|
||||
class ReadFacts(msgspec.Struct, frozen=True):
|
||||
"""The capabilities a ResidualReadout declares (see ResidualReadout).
|
||||
Binding takes kernels only from the reads of the stages it binds, never
|
||||
from a stage recorded by its facts, so these supply none."""
|
||||
|
||||
is_plain_norm: bool
|
||||
reads_before_dp_gather: bool
|
||||
reads_after_attn_tp_gather: bool
|
||||
completing_fusions = ()
|
||||
gathering_reads = ()
|
||||
|
||||
@classmethod
|
||||
def of(cls, read) -> "ReadFacts":
|
||||
if isinstance(read, cls):
|
||||
return read
|
||||
return cls(
|
||||
is_plain_norm=read.is_plain_norm,
|
||||
reads_before_dp_gather=read.reads_before_dp_gather,
|
||||
reads_after_attn_tp_gather=read.reads_after_attn_tp_gather,
|
||||
)
|
||||
|
||||
|
||||
class TransformFacts(msgspec.Struct, frozen=True):
|
||||
"""What an OutputTransform declares (see OutputTransform)."""
|
||||
|
||||
before_reduce_scatter: bool = False
|
||||
|
||||
@classmethod
|
||||
def of(cls, transform) -> Optional["TransformFacts"]:
|
||||
if transform is None or isinstance(transform, cls):
|
||||
return transform
|
||||
return cls(before_reduce_scatter=transform.before_reduce_scatter)
|
||||
|
||||
|
||||
def facts_of(declaration):
|
||||
"""``declaration`` with its read, update and output transform reduced to
|
||||
the capabilities they declare, and without the declarations it was
|
||||
chained to. None stays None."""
|
||||
if declaration is None:
|
||||
return None
|
||||
from dataclasses import replace
|
||||
|
||||
return replace(
|
||||
declaration,
|
||||
read=ReadFacts.of(declaration.read),
|
||||
update=UpdateFacts.of(declaration.update),
|
||||
output_transform=TransformFacts.of(declaration.output_transform),
|
||||
previous=None,
|
||||
prepared_from=None,
|
||||
)
|
||||
@@ -335,7 +335,6 @@ def keep_output(
|
||||
hidden_states: torch.Tensor,
|
||||
residual: torch.Tensor,
|
||||
forward_batch: ForwardBatch,
|
||||
**kwargs,
|
||||
):
|
||||
return hidden_states, residual
|
||||
|
||||
@@ -346,7 +345,6 @@ def update_attn_tp_gather_output(
|
||||
forward_batch: ForwardBatch,
|
||||
update: ResidualUpdate = PLAIN_ADD,
|
||||
gather: Optional[Callable] = None,
|
||||
**kwargs,
|
||||
):
|
||||
"""Write the output into the residual on this rank's slice, then gather it
|
||||
over attention TP (see attn_tp_gather_with)."""
|
||||
@@ -371,7 +369,6 @@ def residual_slice_output(
|
||||
sums: bool,
|
||||
gathers_back: bool,
|
||||
update: ResidualUpdate,
|
||||
**kwargs,
|
||||
):
|
||||
"""Bring the FFN output onto the slice of the rows the residual is on:
|
||||
a reduce-scatter that also completes its sum when ``sums``, else this
|
||||
@@ -405,7 +402,6 @@ def attn_cp_take_back_output(
|
||||
hidden_states: torch.Tensor,
|
||||
residual: torch.Tensor,
|
||||
forward_batch: ForwardBatch,
|
||||
**kwargs,
|
||||
):
|
||||
"""DSA and MLA CP: this rank's shard of a complete output gathered in
|
||||
equal shards over the attention-CP group."""
|
||||
@@ -418,7 +414,6 @@ def attn_cp_reduce_scatter_output(
|
||||
hidden_states: torch.Tensor,
|
||||
residual: torch.Tensor,
|
||||
forward_batch: ForwardBatch,
|
||||
**kwargs,
|
||||
):
|
||||
"""DSA and MLA CP: sum the FFN output over the attention-CP group and
|
||||
keep this rank's shard."""
|
||||
@@ -429,7 +424,6 @@ def dp_cp_take_back_output(
|
||||
hidden_states: torch.Tensor,
|
||||
residual: torch.Tensor,
|
||||
forward_batch: ForwardBatch,
|
||||
**kwargs,
|
||||
):
|
||||
"""This rank's CP shard of the rows the DP gather put in its DP group's
|
||||
slot, at the shard's padded length with the padding zeroed."""
|
||||
@@ -450,7 +444,6 @@ def moe_cp_take_back_output(
|
||||
hidden_states: torch.Tensor,
|
||||
residual: torch.Tensor,
|
||||
forward_batch: ForwardBatch,
|
||||
**kwargs,
|
||||
):
|
||||
"""Return a MoE output computed on the MoE-CP-gathered rows to this rank's attention rows.
|
||||
|
||||
|
||||
@@ -174,7 +174,7 @@ class StageBoundary:
|
||||
completed or carried on. An FFN whose boundary has fusions (which may
|
||||
take a MoE finalize handoff) finishes through exit() instead."""
|
||||
if self.kind is StageKind.ATTENTION and self.plan.finishes_directly:
|
||||
produced = self.plan.produced(forward_batch)
|
||||
produced = self.plan.path_for(forward_batch).output
|
||||
return stream_of(forward_batch).record(
|
||||
hidden_states,
|
||||
produced.update,
|
||||
@@ -238,9 +238,6 @@ class StageBoundary:
|
||||
)
|
||||
return hidden_states
|
||||
|
||||
def snapshot(self, hidden_states, forward_batch):
|
||||
return stream_of(forward_batch).snapshot(hidden_states)
|
||||
|
||||
def capture_output(self, hidden_states, forward_batch, *, skip_empty=False):
|
||||
"""Capture an output while preserving the caller's reduction timing.
|
||||
|
||||
|
||||
@@ -607,7 +607,7 @@ class LongcatFlashModel(nn.Module):
|
||||
for i in range(total_num_layers):
|
||||
if i in self.layers_to_capture:
|
||||
aux_hidden_states.append(
|
||||
self.layers[i].attn_boundary.snapshot(hidden_states, forward_batch)
|
||||
residual_batch.snapshot(hidden_states, forward_batch)
|
||||
)
|
||||
with get_global_expert_distribution_recorder().with_current_layer(i):
|
||||
layer = self.layers[i]
|
||||
|
||||
@@ -18,6 +18,7 @@
|
||||
"""Inference-only NemotronH model."""
|
||||
|
||||
from collections.abc import Iterable
|
||||
from functools import partial
|
||||
|
||||
import torch
|
||||
from torch import nn
|
||||
@@ -77,7 +78,7 @@ from sglang.srt.model_loader.weight_utils import (
|
||||
replace_prefix,
|
||||
replace_substrings,
|
||||
)
|
||||
from sglang.srt.models.nemotron_h_utils import make_stage_boundary
|
||||
from sglang.srt.models.nemotron_h_utils import make_stage_boundary, stage_facts
|
||||
from sglang.srt.models.utils import WeightsMapper
|
||||
from sglang.srt.runtime_context import get_exec, get_parallel
|
||||
from sglang.srt.utils import (
|
||||
@@ -690,6 +691,7 @@ class NemotronHModel(nn.Module):
|
||||
len(config.hybrid_override_pattern),
|
||||
get_layer,
|
||||
prefix=f"{prefix}.layers",
|
||||
stage_facts=partial(stage_facts, config.hybrid_override_pattern),
|
||||
)
|
||||
if self.pp_group.is_last_rank:
|
||||
self.norm_f = RMSNorm(config.hidden_size, eps=config.layer_norm_epsilon)
|
||||
|
||||
@@ -37,15 +37,18 @@ def _declaration(pattern: str, layer_idx: int):
|
||||
)
|
||||
|
||||
|
||||
def stage_facts(pattern: str, layer_idx: int):
|
||||
"""The one stage each Nemotron-H layer declares, from the pattern alone:
|
||||
the model's shared declaration function, which its layers declare with
|
||||
too (see make_layers)."""
|
||||
return (_declaration(pattern, layer_idx),)
|
||||
|
||||
|
||||
def make_stage_boundary(layer_norm: RMSNorm, *, pattern: str, layer_idx: int):
|
||||
from sglang.srt.layers import layernorm_sp
|
||||
|
||||
if get_parallel().attn_cp_size > 1 or layernorm_sp.layernorm_sp_enabled():
|
||||
raise NotImplementedError("a Nemotron stage with attention CP or LayerNorm SP")
|
||||
(boundary,) = append_stages(
|
||||
(
|
||||
_declaration(pattern, layer_idx),
|
||||
layer_norm,
|
||||
),
|
||||
)
|
||||
(declaration,) = stage_facts(pattern, layer_idx)
|
||||
(boundary,) = append_stages((declaration, layer_norm))
|
||||
return boundary
|
||||
|
||||
@@ -1591,18 +1591,27 @@ def make_layers(
|
||||
return_tuple: bool = False,
|
||||
offloader_kwargs: Optional[Dict[str, Any]] = None,
|
||||
final_read: Optional[Any] = None,
|
||||
stage_facts: Optional[Callable[[int], Sequence[Any]]] = None,
|
||||
) -> Tuple[torch.nn.Module, int, int]:
|
||||
"""Make a list of layers with the given layer function.
|
||||
|
||||
The local layers are built inside one layer stack, so layers that declare
|
||||
stage boundaries connect in order without naming their neighbours. Across
|
||||
a pipeline stage boundary the stack learns the neighbouring stage from the
|
||||
layer itself, built again on the meta device. ``final_read`` is the
|
||||
stack's terminal read when it is not a plain final norm (see layer_stack).
|
||||
a pipeline stage boundary the stack learns the neighbouring stage from
|
||||
``stage_facts``: the model's shared declaration function, which returns
|
||||
the stages the layer at a global index declares without building it (an
|
||||
empty sequence when it declares none), and which the layer itself uses
|
||||
for its own declarations. A model on the transitional list may omit it;
|
||||
the stack then builds the layer again on the meta device to read them.
|
||||
``final_read`` is the stack's terminal read when it is not a plain final
|
||||
norm (see layer_stack).
|
||||
"""
|
||||
# circular imports
|
||||
from sglang.srt.distributed import get_pp_indices
|
||||
from sglang.srt.layers.layer_boundary.factories import layer_stack
|
||||
from sglang.srt.layers.layer_boundary.factories import (
|
||||
check_declared_stages,
|
||||
layer_stack,
|
||||
)
|
||||
from sglang.srt.layers.utils import PPMissingLayer
|
||||
from sglang.srt.utils.offloader import get_offloader
|
||||
|
||||
@@ -1618,22 +1627,29 @@ def make_layers(
|
||||
)
|
||||
|
||||
def neighbour(idx):
|
||||
if stage_facts is not None:
|
||||
return functools.partial(_declared_stages, stage_facts, idx)
|
||||
return functools.partial(
|
||||
_build_neighbour_layer, layer_fn, idx, add_prefix(idx, prefix)
|
||||
)
|
||||
|
||||
def build(stack, idx):
|
||||
appended = len(stack.appends)
|
||||
layer = layer_fn(idx=idx, prefix=add_prefix(idx, prefix))
|
||||
if stage_facts is not None and (pp_size or 1) > 1:
|
||||
# Another rank binds this layer's stages from stage_facts alone.
|
||||
check_declared_stages(stack, appended, stage_facts(idx), f"layer {idx}")
|
||||
return layer
|
||||
|
||||
with layer_stack(
|
||||
previous_layers=[neighbour(idx) for idx in reversed(range(start_layer))],
|
||||
next_layers=[neighbour(idx) for idx in range(end_layer, num_hidden_layers)],
|
||||
final_read=final_read,
|
||||
):
|
||||
) as stack:
|
||||
modules = torch.nn.ModuleList(
|
||||
[PPMissingLayer(return_tuple=return_tuple) for _ in range(start_layer)]
|
||||
+ get_offloader().wrap_modules(
|
||||
(
|
||||
layer_fn(idx=idx, prefix=add_prefix(idx, prefix))
|
||||
for idx in range(start_layer, end_layer)
|
||||
),
|
||||
(build(stack, idx) for idx in range(start_layer, end_layer)),
|
||||
**(offloader_kwargs or {}),
|
||||
)
|
||||
+ [
|
||||
@@ -1653,10 +1669,12 @@ def make_pp_layers(
|
||||
return_tuple: bool = False,
|
||||
offloader_kwargs: Optional[Dict[str, Any]] = None,
|
||||
final_read: Optional[Any] = None,
|
||||
stage_facts: Optional[Callable[[int], Sequence[Any]]] = None,
|
||||
) -> Tuple[torch.nn.Module, int, int]:
|
||||
"""Make this pipeline stage's layers, and return them with the stage's range.
|
||||
|
||||
Layers outside ``[start_layer, end_layer)`` are ``PPMissingLayer`` stand-ins.
|
||||
``stage_facts`` is the model's shared declaration function (see make_layers).
|
||||
"""
|
||||
parallel = get_parallel()
|
||||
return make_layers(
|
||||
@@ -1668,15 +1686,81 @@ def make_pp_layers(
|
||||
return_tuple=return_tuple,
|
||||
offloader_kwargs=offloader_kwargs,
|
||||
final_read=final_read,
|
||||
stage_facts=stage_facts,
|
||||
)
|
||||
|
||||
|
||||
# The models whose layers a pipeline rank still builds on the meta device to
|
||||
# read the stages they declare, until each gives make_layers its shared
|
||||
# declaration function (stage_facts). Any other model must give one to run
|
||||
# under pipeline parallelism.
|
||||
_NEIGHBOUR_BUILD_MODULES = frozenset(
|
||||
(
|
||||
"sglang.srt.models.apertus",
|
||||
"sglang.srt.models.arcee",
|
||||
"sglang.srt.models.bailing_moe",
|
||||
"sglang.srt.models.bailing_moe_linear",
|
||||
"sglang.srt.models.bailing_moe_v3",
|
||||
"sglang.srt.models.deepseek_v2",
|
||||
"sglang.srt.models.dots3_common.modeling",
|
||||
"sglang.srt.models.ernie45_moe_vl",
|
||||
"sglang.srt.models.exaone4",
|
||||
"sglang.srt.models.exaone_moe",
|
||||
"sglang.srt.models.gemma4_causal",
|
||||
"sglang.srt.models.gigachat35",
|
||||
"sglang.srt.models.glm4",
|
||||
"sglang.srt.models.glm4_moe",
|
||||
"sglang.srt.models.glm4_moe_lite",
|
||||
"sglang.srt.models.glm5_next",
|
||||
"sglang.srt.models.gpt_oss",
|
||||
"sglang.srt.models.granitemoehybrid",
|
||||
"sglang.srt.models.kimi_k3",
|
||||
"sglang.srt.models.kimi_linear",
|
||||
"sglang.srt.models.laguna",
|
||||
"sglang.srt.models.llada2",
|
||||
"sglang.srt.models.llama",
|
||||
"sglang.srt.models.mimo_v2",
|
||||
"sglang.srt.models.minimax_m2",
|
||||
"sglang.srt.models.minimax_m3",
|
||||
"sglang.srt.models.ministral3",
|
||||
"sglang.srt.models.mixtral",
|
||||
"sglang.srt.models.nanbeige",
|
||||
"sglang.srt.models.nemotron_nas",
|
||||
"sglang.srt.models.qwen2",
|
||||
"sglang.srt.models.qwen2_moe",
|
||||
"sglang.srt.models.qwen3_5",
|
||||
"sglang.srt.models.sarvam_moe",
|
||||
"sglang.srt.models.sdar",
|
||||
"sglang.srt.models.sdar_moe",
|
||||
"sglang.srt.models.spark2_5",
|
||||
"sglang.srt.models.step3p5",
|
||||
"sglang.srt.models.xllm",
|
||||
"sglang.srt.models.zaya",
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _declared_stages(stage_facts, idx: int):
|
||||
"""The stages the layer at ``idx`` declares, from the model's shared
|
||||
declaration function, standing in for building it."""
|
||||
from sglang.srt.layers.layer_boundary.factories import DeclaredStages
|
||||
|
||||
return DeclaredStages(tuple(stage_facts(idx)))
|
||||
|
||||
|
||||
def _build_neighbour_layer(layer_fn: LayerFn, idx: int, prefix: str) -> None:
|
||||
"""Build a layer another pipeline stage holds, only for the stage
|
||||
boundaries it declares (see building_neighbour_layer). RoPE modules it
|
||||
adds to the shared cache are meta, so they are dropped again."""
|
||||
from sglang.srt.layers.rotary_embedding.factory import _ROPE_DICT
|
||||
|
||||
module = getattr(getattr(layer_fn, "func", layer_fn), "__module__", None)
|
||||
if module not in _NEIGHBOUR_BUILD_MODULES:
|
||||
raise ValueError(
|
||||
f"{module} declares stage boundaries under pipeline parallelism "
|
||||
"without a shared declaration function: give make_layers its "
|
||||
"stage_facts, the stages a layer declares without building it"
|
||||
)
|
||||
cached = set(_ROPE_DICT)
|
||||
try:
|
||||
with building_neighbour_layer():
|
||||
|
||||
@@ -22,6 +22,7 @@ from sglang.srt.layers.layer_boundary import (
|
||||
layer_stack,
|
||||
)
|
||||
from sglang.srt.layers.layer_boundary import prepare as boundary_prepare
|
||||
from sglang.srt.layers.layer_boundary.facts import facts_of
|
||||
from sglang.srt.layers.layer_boundary.layout import SumGroup, TokenAxis
|
||||
from sglang.srt.layers.layer_boundary.ops import (
|
||||
attn_tp_gather_input,
|
||||
@@ -37,6 +38,7 @@ from sglang.srt.layers.layer_boundary.residual.add_norm import (
|
||||
from sglang.srt.layers.layer_boundary.residual.stream import DeclaredSum
|
||||
from sglang.srt.layers.rotary_embedding import factory as rope_factory
|
||||
from sglang.srt.model_executor.forward_batch_info import PPProxyTensors
|
||||
from sglang.srt.utils import common
|
||||
from sglang.srt.utils.common import is_building_neighbour_layer, make_layers
|
||||
from sglang.test.ci.ci_register import register_cpu_ci
|
||||
from sglang.test.communicator_patch import patch_communicator
|
||||
@@ -169,7 +171,7 @@ class TestAppendStages(CustomTestCase):
|
||||
self.assertTrue(all(s.plan is not None for s in branch))
|
||||
# The branch reads the MoE's input; the next layer follows the main
|
||||
# line's MoE, not the branch's dense FFN.
|
||||
self.assertEqual(branch[0].declaration.prepared_from, moe.declaration)
|
||||
self.assertEqual(branch[0].declaration.prepared_from, facts_of(moe.declaration))
|
||||
self.assertTrue(following[0].declaration.previous.sparse)
|
||||
|
||||
def test_a_nested_stack_leaves_the_outer_one_untouched(self):
|
||||
@@ -214,7 +216,7 @@ class TestAppendStages(CustomTestCase):
|
||||
(declare_ffn(), fixture.Norm()), prepared_from=first.declaration
|
||||
)
|
||||
self.assertIsNot(first.declaration, second.declaration)
|
||||
self.assertIs(branch.declaration.prepared_from, first.declaration)
|
||||
self.assertEqual(branch.declaration.prepared_from, facts_of(first.declaration))
|
||||
|
||||
@unittest.skipUnless(hasattr(BaseException, "add_note"), "needs add_note")
|
||||
def test_an_error_at_close_names_the_append_it_binds(self):
|
||||
@@ -302,6 +304,15 @@ class TestMakeLayers(CustomTestCase):
|
||||
self.planning = fixture.planning(fixture.parallel_of(attn_dp=1, attn_tp=2))
|
||||
self.planning.__enter__()
|
||||
self.addCleanup(self.planning.__exit__, None, None, None)
|
||||
# These layers are built again on the meta device, as a model's on the
|
||||
# transitional list are, unless a test gives a declaration function.
|
||||
allowed = patch.object(
|
||||
common,
|
||||
"_NEIGHBOUR_BUILD_MODULES",
|
||||
common._NEIGHBOUR_BUILD_MODULES | {__name__},
|
||||
)
|
||||
allowed.start()
|
||||
self.addCleanup(allowed.stop)
|
||||
|
||||
def build(self, pp_rank=None, pp_size=None, *, declares=True):
|
||||
"""Build the layers; layer 1 has a MoE. Returns each layer_fn call as
|
||||
@@ -357,6 +368,56 @@ class TestMakeLayers(CustomTestCase):
|
||||
self.assertFalse(last[2][0].plan.enters_stack)
|
||||
self.assertTrue(last[3][1].plan.terminal)
|
||||
|
||||
def test_a_shared_declaration_function_stands_in_for_the_neighbours(self):
|
||||
built, stages = [], {}
|
||||
|
||||
def layer_fn(idx, prefix):
|
||||
built.append(idx)
|
||||
stages[idx] = layer(sparse=idx == 1)
|
||||
return nn.Identity()
|
||||
|
||||
def stage_facts(idx):
|
||||
# The stages layer() declares, without building the layer.
|
||||
return (
|
||||
declare_attn(),
|
||||
declare_ffn(sparse=idx == 1, next_layer_sparse=idx == 1),
|
||||
)
|
||||
|
||||
make_layers(
|
||||
NUM_LAYERS,
|
||||
layer_fn,
|
||||
pp_rank=1,
|
||||
pp_size=2,
|
||||
prefix="layers",
|
||||
stage_facts=stage_facts,
|
||||
)
|
||||
self.assertEqual(built, [2, 3])
|
||||
# Layer 2's attention follows layer 1's MoE, on the other stage.
|
||||
self.assertTrue(stages[2][0].declaration.previous.sparse)
|
||||
|
||||
def test_a_layer_must_declare_what_its_declaration_function_says(self):
|
||||
def layer_fn(idx, prefix):
|
||||
layer(sparse=idx == 2)
|
||||
return nn.Identity()
|
||||
|
||||
with self.assertRaisesRegex(
|
||||
ValueError,
|
||||
r"layer 2 declares stages .*: stage 1 \(FFN\): sparse is True, "
|
||||
"the function says False",
|
||||
):
|
||||
make_layers(
|
||||
NUM_LAYERS,
|
||||
layer_fn,
|
||||
pp_rank=1,
|
||||
pp_size=2,
|
||||
stage_facts=lambda idx: (declare_attn(), declare_ffn()),
|
||||
)
|
||||
|
||||
def test_a_model_off_the_list_needs_a_declaration_function(self):
|
||||
with patch.object(common, "_NEIGHBOUR_BUILD_MODULES", frozenset()):
|
||||
with self.assertRaisesRegex(ValueError, "stage_facts"):
|
||||
self.build(pp_rank=1, pp_size=2)
|
||||
|
||||
def test_layers_without_stages_build_no_neighbour(self):
|
||||
calls, _ = self.build(pp_rank=1, pp_size=2, declares=False)
|
||||
self.assertEqual([call[0] for call in calls], [2, 3])
|
||||
|
||||
@@ -22,6 +22,8 @@ from sglang.srt.layers.layer_boundary import (
|
||||
layer_stack,
|
||||
)
|
||||
from sglang.srt.layers.layer_boundary.contracts import BatchVariant
|
||||
from sglang.srt.layers.layer_boundary.facts import facts_of
|
||||
from sglang.srt.layers.layer_boundary.output import OutputTransform
|
||||
from sglang.srt.layers.layer_boundary.residual import batch
|
||||
from sglang.srt.model_executor.forward_batch_info import ForwardMode
|
||||
from sglang.test.boundary_fixtures import build_stages
|
||||
@@ -189,9 +191,12 @@ class TestIndependentStageConstruction(CustomTestCase):
|
||||
)
|
||||
# The branch leaves the stage it reads from as it was.
|
||||
self.assertEqual(source.plan.edges, alone.plan.edges)
|
||||
self.assertEqual(first.declaration.prepared_from, source.declaration)
|
||||
self.assertIs(attn.declaration.previous, first.declaration)
|
||||
self.assertIs(last.declaration.previous, attn.declaration)
|
||||
# Each records the stage before it by what binding reads of it.
|
||||
self.assertEqual(
|
||||
first.declaration.prepared_from, facts_of(source.declaration)
|
||||
)
|
||||
self.assertEqual(attn.declaration.previous, facts_of(first.declaration))
|
||||
self.assertEqual(last.declaration.previous, facts_of(attn.declaration))
|
||||
for v, edge in first.plan.edges.items():
|
||||
source_input = source.plan.edges[v].incoming
|
||||
self.assertEqual(
|
||||
@@ -449,6 +454,33 @@ class TestPipelineCuts(CustomTestCase):
|
||||
)
|
||||
return True
|
||||
|
||||
def test_a_rank_after_an_attention_that_transforms_its_output_is_rejected(self):
|
||||
# The attention leaves its sum, and its transform runs at the next
|
||||
# stage's input: on the receiving rank, which does not hold it.
|
||||
transformed = partial(declare_attn, output_transform=OutputTransform(abs))
|
||||
for (name, (parallel, a2a)), kind in itertools.product(
|
||||
CUT_CONFIGS.items(), CUT_STAGES
|
||||
):
|
||||
with (
|
||||
self.subTest(config=name, receiver=kind),
|
||||
fixture.planning(parallel, a2a=a2a),
|
||||
):
|
||||
# Without the transform, the same rank binds.
|
||||
self.assertIsNone(bind_rank((kind,), before=("attention",))[1])
|
||||
with (
|
||||
self.assertRaisesRegex(
|
||||
NotImplementedError,
|
||||
"a pipeline rank that ends on an attention transforming "
|
||||
"the output whose sum it leaves",
|
||||
),
|
||||
layer_stack(
|
||||
previous_layers=[
|
||||
lambda: append_stages((transformed(), fixture.Norm()))
|
||||
]
|
||||
),
|
||||
):
|
||||
append_stages(*cut_stages((kind,)))
|
||||
|
||||
def assert_rows_follow(self, stages, rank):
|
||||
"""Within one rank's stages, each side of a boundary takes the
|
||||
residual on the rows the other side leaves it on."""
|
||||
|
||||
Reference in New Issue
Block a user