[Refactor] Declare pipeline neighbours from a shared declaration function (#43459)

This commit is contained in:
Cheng Wan
2026-10-10 12:42:46 -07:00
committed by GitHub
parent 3b717c0f13
commit 3b16eeedce
14 changed files with 438 additions and 85 deletions
+15 -3
View File
@@ -122,7 +122,7 @@ with layer_stack():
Every stage binds when the stack closes, once its producer and its consumer are known: the stage appended before it and the one appended after it. The last stage ends the model's layer stack and is marked terminal, unless a later layer declares a stage. A boundary's `declaration` is usable at once; when the stack closes, it is replaced by the declaration with its place in the stack (its `previous` and `terminal`), and the plan is bound. An error raised while binding carries a note naming the `append_stages()` call it came from. A terminal stage leaves no work for a following layer, except the final-norm finalize handoff described in [Enter and leave the layer stack](#enter-and-leave-the-layer-stack). A NextN/MTP draft module is its own stack: its layer enters the stack and ends it, even when it reuses the target's decoder class.
Pipeline parallelism splits one layer stack across ranks, so the first and last local stages have neighbours that another rank builds. `make_layers()` finds them without communicating: when its stack closes, it builds the layer before the rank's first local layer and the one after its last again, with the model's own layer function inside `building_neighbour_layer()` (on the meta device), reads the stages they declare and discards them. A stage's declaration depends only on the config and the global layer index, so both ranks see the same boundary. A layer that declares no stage is passed over in favour of the next one out. These neighbours are built only if the stack appended stages, and after the local layers. A neighbour is never loaded or run, so a layer constructor must run on the meta device, must skip the host and device resources only a running layer needs (tables, streams, engines, communicators) when `is_building_neighbour_layer()` is true, and must not depend on which pipeline rank builds it. The meta device alone does not stop a constructor that names a device explicitly.
Pipeline parallelism splits one layer stack across ranks, so the first and last local stages have neighbours that another rank builds. The model gives `make_layers()` (or `make_pp_layers()`) its shared declaration function, `stage_facts`: given a global layer index, it returns the stages that layer declares, in order (an empty tuple for a layer that declares none), from the config, the layer index, construction options known before the layer is built, and the fixed parallel topology, never from built modules or provider state. The layer declares its own stages with the same function, so both ranks derive a boundary from one source; under pipeline parallelism `make_layers()` also checks each layer it builds against the function. When the stack closes, it asks the function for the stages next to it, nearest layer first, passing over a layer that declares none, and keeps only what binding reads of them (`facts_of()`: the capabilities a read, update and output transform declare, not the objects). A model that does not give the function yet is on a transitional list: its neighbours are built again with the model's own layer function inside `building_neighbour_layer()` (on the meta device), which must then skip the host and device resources only a running layer needs when `is_building_neighbour_layer()` is true and must not depend on which pipeline rank builds it. A model off that list must give the function to run under pipeline parallelism.
Each item may include a third element containing constructor options: an attention stage accepts `qkv_latent_func` (for example `{"qkv_latent_func": self.self_attn.prepare_qkv_latent}`) and `fusions`; an FFN stage accepts only `fusions` (see [Fusion providers](#fusion-providers-and-output-decisions)). An option a stage does not take is rejected when appended. Pass declarations fresh from `declare_attn()` / `declare_ffn()`: one taken from a bound boundary already has its place in a stack, and is rejected.
@@ -251,7 +251,7 @@ Required attention and MoE collectives are unaffected. These are permissions, no
Before you open a pull request for a new model, check each item:
- Every decoder layer appends its stages inside a layer stack: `make_layers()`, or `layer_stack()` around any other construction, including a NextN/MTP draft's own layer.
- The layer constructor runs on the meta device, allocates nothing when `is_building_neighbour_layer()` is true, and declares the same stages whichever pipeline rank builds it: `make_layers()` builds the layers next to a rank's own that way to read their stages.
- The model gives `make_layers()` its shared declaration function (`stage_facts`), and each layer declares its stages with it: another pipeline rank binds the stages next to its own from that function alone.
- The model forward calls `residual_batch.start()` before the first layer, and leaves the stack through `residual_batch.final_norm()`, `to_pp()`, or `take_output()`.
- Pipeline reception calls `from_pp(tensors, forward_batch)` on the first local boundary.
- Split prefill starts the stream only in the segment that runs the first layer and completes it only in the final segment.
@@ -332,6 +332,18 @@ Configuration validation rejects unsupported model/parallel combinations. Bounda
### Fusion providers and output decisions
Each exit has its own policy, and what it decides from fixed facts (the parallel configuration, the declarations and the bound path) is decided when the plan is built and recorded on the path as `ExitFacts`; only what depends on the batch is decided for it:
| Exit | Entry point | May decide |
|---|---|---|
| FFN | `exit()` or `finish()` | Complete the sum now, possibly in an attention-DP or CP / input-scattered reduce-scatter on the way back; leave it to the next input (an all-reduce, or under attention DP the scatter back to this rank's tokens); hand a MoE finalize to the next input |
| Operation-scheduled (TBO) | `complete_now()` | Only complete it now: under attention DP the move back is a plain scatter; never defers or hands off |
| Mixer | `finish()` | Leave its declared sum to the next stage's input (before an FFN); carry it to the next mixer's input (without attention DP); or complete it |
| Stack end | `exit()` or `finish()` | Complete it now; a MoE finalize handoff may reach a final norm that accepts it |
| Pipeline handoff | `exit()` or `finish()`, then `to_pp()` | Complete it now; an FFN on its own rows also writes it into the residual |
Fixed per path: whether an FFN may leave its sum to the next input at all, whether a reduce-scatter completes it without an attention-DP one, which sum an input-scattered batch's next input completes, whether the attention-DP return is the reduce-scatterv, whether the post-expert sum is one all-reduce, and whether a mixer carries its sum. Decided per batch: its padding, whether the attention-DP reduce-scatter is usable (an elastic EP scale-up changes the DP group), whether it has tokens, and whether a fused consumer takes the sum.
An output decision is made once. Its flags control whether compute performs reduction/finalize work; the matching completion action either executes that work or records exactly what the next boundary owes. Do not set a skip flag independently of a completion action.
A backend fusion provider is passed per stage as the `fusions` option, `(declaration, norm, {"fusions": provider})`; give a layer's attention and FFN stages the same provider. The boundary calls it as follows:
@@ -342,7 +354,7 @@ A backend fusion provider is passed per stage as the `fusions` option, `(declara
`install_cutedsl_fusion()` and `prepare_cutedsl_fusion()` find providers through the decoder layer's `attn_boundary` and `ffn_boundary` attributes.
Consumer fusion candidates are ordered. A candidate returning `None` must do so before modifying inputs or starting a collective, so fallback remains valid. `FfnInputFusion` declares the sum group it completes with residual add and norm. A residual read may supply its own candidates as `completing_fusions`: each `ReadoutFusion` completes a sum with the residual add, ahead of the read, and is tried before the built-in collective; one with `reads=True` also does the read. Where a stage reads its input on this rank's attention-TP slice and then gathers it, a read's `gathering_reads` may do both in one kernel. A stage may declare its own implementation of the gather over attention TP that brings rows into it (`declare_attn(attn_tp_gather=...)`), tried on every batch before the built-in gather: at its own entry, or at its producer's exit, since an exit runs its consumer's gather. Across a pipeline boundary the producer's rank takes that gather from the neighbouring layer it builds on the meta device, so the gather may use only the communication state of the rank it runs on, not the declaring layer's weights or buffers. The consumer only tries compatible candidates for the declared update/read capabilities.
Consumer fusion candidates are ordered. A candidate returning `None` must do so before modifying inputs or starting a collective, so fallback remains valid. `FfnInputFusion` declares the sum group it completes with residual add and norm. A residual read may supply its own candidates as `completing_fusions`: each `ReadoutFusion` completes a sum with the residual add, ahead of the read, and is tried before the built-in collective; one with `reads=True` also does the read. Where a stage reads its input on this rank's attention-TP slice and then gathers it, a read's `gathering_reads` may do both in one kernel. A stage may declare its own implementation of the gather over attention TP that brings rows into it (`declare_attn(attn_tp_gather=...)`), tried on every batch before the built-in gather: at its own entry, or at its producer's exit, since an exit runs its consumer's gather. Across a pipeline boundary the producer's rank takes that gather from the neighbouring stage's declaration, so the gather may use only the communication state of the rank it runs on, not the declaring layer's weights or buffers. The consumer only tries compatible candidates for the declared update/read capabilities.
An `OutputTransform` declares contribution processing before residual update. Its explicit `before_reduce_scatter` setting preserves a supported implementation's order; it is not permission to commute arbitrary operations across a reduction.
@@ -24,7 +24,6 @@ from sglang.srt.layers.layer_boundary.boundary import (
)
from sglang.srt.layers.layer_boundary.construction import (
BatchVariant,
VariantEdges,
)
from sglang.srt.layers.layer_boundary.contracts import (
EdgeContract,
@@ -35,11 +34,10 @@ from sglang.srt.layers.layer_boundary.contracts import (
OutputContract,
ProducerReduction,
ReadoutFusion,
StageContract,
StageKind,
StagePath,
)
from sglang.srt.layers.layer_boundary.exit import ExitDecision, FfnExit, MixerExit
from sglang.srt.layers.layer_boundary.exit import MixerExit
from sglang.srt.layers.layer_boundary.factories import (
append_stages,
declare_attn,
@@ -52,8 +50,6 @@ from sglang.srt.layers.layer_boundary.layout import (
TokenAxis,
batch_gathers_over_moe_cp,
is_dense_ffn_fully_dp,
moe_gathers_over_moe_cp,
token_axis_sizes,
)
from sglang.srt.layers.layer_boundary.ops import (
move_rows,
@@ -64,11 +60,9 @@ from sglang.srt.layers.layer_boundary.output import (
UnreducedOutput,
complete_owed,
)
from sglang.srt.layers.layer_boundary.residual import LayerResidualOps
from sglang.srt.layers.layer_boundary.residual.add_norm import (
FUSE_ALLREDUCE_MAX_BATCH_SIZE,
NORM_QUANT_READOUT,
NORM_READOUT,
PLAIN_ADD,
PLAIN_RESIDUAL_OPS,
)
@@ -94,21 +88,16 @@ __all__ = [
"ExitRows",
"ProducerReduction",
"FUSE_ALLREDUCE_MAX_BATCH_SIZE",
"ExitDecision",
"FfnExit",
"FfnInputFusion",
"ReadoutFusion",
"DeferredFinalize",
"GatedResidualState",
"IHCState",
"LayerResidualOps",
"Layout",
"MHCState",
"MixerExit",
"NORM_QUANT_READOUT",
"NORM_READOUT",
"PLAIN_RESIDUAL_OPS",
"StageContract",
"EntryPath",
"InputContract",
"StageKind",
@@ -123,8 +112,6 @@ __all__ = [
"batch_gathers_over_moe_cp",
"move_rows",
"complete_owed",
"moe_gathers_over_moe_cp",
"tbo_split_moves",
"token_axis_sizes",
"tp_reduce_scatter",
]
@@ -235,9 +235,6 @@ class StagePlan:
return _bound_for(self._next_input_rows, self.variant_for(forward_batch))
return self.path_for(forward_batch).entry.input_rows
def produced(self, forward_batch):
return self.path_for(forward_batch).output
@cached_property
def output(self):
return ExitPolicy(self)
@@ -27,7 +27,9 @@ from sglang.srt.layers.layer_boundary.residual.add_norm import NORM_READOUT, PLA
class ProducerReduction(Enum):
"""Who completes the sum a stage's output owes; compute never does.
"""Who completes the sum a stage's output owes: exactly one completer for
each output, the boundary. Compute completes it only where an FFN fuses
it with a reduction its computation needs (``output_complete``).
ALWAYS_PARTIAL is attention-only: finish() hands the partial sum to the
next stage's input as its declared sum. EXIT_SCOPED, for an FFN or a
@@ -5,9 +5,9 @@ from __future__ import annotations
import contextlib
import os
import sys
from dataclasses import dataclass, replace
from dataclasses import dataclass, fields, is_dataclass, replace
from types import MappingProxyType
from typing import Callable, Mapping, NamedTuple, Optional
from typing import Callable, Mapping, NamedTuple, Optional, Tuple
from sglang.srt.environ import envs
from sglang.srt.layers import layernorm_sp
@@ -29,6 +29,7 @@ from sglang.srt.layers.layer_boundary.contracts import (
StageContract,
StageKind,
)
from sglang.srt.layers.layer_boundary.facts import facts_of
from sglang.srt.layers.layer_boundary.layout import (
Layout,
SumGroup,
@@ -301,11 +302,14 @@ class StageDeclaration:
contiguous slice of the rows, it returns them all, in rank order,
or None to leave the gather to the boundary. Called on every batch,
so it must be CUDA-graph safe. Across a pipeline boundary the
producer's rank takes it from the neighbouring layer it builds on
the meta device, so it may use only the communication state of the
rank it runs on, not the declaring layer's weights or buffers.
previous: Declaration whose output this stage consumes, as the stack
records it; across pipeline ranks it is built locally.
producer's rank takes it from the neighbouring stage's declaration
(the model's shared declaration function, or, for a model on the
transitional list, the layer built again on the meta device), so it
may use only the communication state of the rank it runs on, not
the declaring layer's weights or buffers.
previous: The stage whose output this stage consumes, as the stack
records it: only the facts binding reads of it (see facts_of),
on this rank or another.
prepared_from: Declaration whose already-read input a branch reuses.
Mutually exclusive with previous; avoids a second update/read.
@@ -738,6 +742,19 @@ def _connect_line(line, origins, *, before=None, after=None, arrivals=None):
if arrivals is None:
arrivals = {v: _stack_arrival(v) for v in variants}
if before is not None:
if (
before.output_transform is not None
and before.kind is StageKind.ATTENTION
and before.reduction is ProducerReduction.ALWAYS_PARTIAL
):
# Its transform runs at the next stage's input, which would be
# on this rank, without the module that declares it.
error = NotImplementedError(
"a pipeline rank that ends on an attention transforming the "
"output whose sum it leaves"
)
_note_origin(error, origins[0])
raise error
flow = resolve(0, before, arrivals, line[0])
arrivals = {v: _arrival(before, flow[v], v) for v in variants}
flows = []
@@ -830,11 +847,14 @@ def layer_stack(*, previous_layers=(), next_layers=(), final_read=None):
stage ends the model's layer stack unless a later layer declares a stage.
Args:
previous_layers: Callables that build, nearest first, the layers before
this stack that another pipeline rank holds. Called only if this
stack appended stages, after its own layers are built, until one of
them declares a stage: its last stage is the producer of this
stack's first. What they build is discarded.
previous_layers: For the layers before this stack that another
pipeline rank holds, nearest first, callables that return the
stages the layer declares as DeclaredStages (from the model's
shared declaration function), or else build the layer so that it
appends them. Called only if this stack appended stages, after its
own layers are built, until one of them declares a stage: its last
stage is the producer of this stack's first. Only what binding
reads of that stage is kept (see facts_of).
next_layers: Likewise for the layers after this stack, whose first
declared stage is the consumer of this stack's last.
final_read: The model's final read of the stack's output (a
@@ -861,21 +881,94 @@ def layer_stack(*, previous_layers=(), next_layers=(), final_read=None):
_stack = outer
def _neighbour_stage(build_layers, *, last):
class DeclaredStages(NamedTuple):
"""The stages a neighbouring layer declares, from the model's shared
declaration function, which stand in for building the layer."""
stages: Tuple[StageDeclaration, ...]
def check_declared_stages(stack, appended: int, expected, where: str) -> None:
"""Check that what a layer appended to ``stack`` since it held
``appended`` appends is what its shared declaration function says it
declares, as far as binding reads it: another pipeline rank binds this
layer's stages from that function alone."""
declared = [
facts_of(declaration)
for append in stack.appends[appended:]
if append.prepared_from is None
for declaration in append.declarations
]
expected = [facts_of(declaration) for declaration in expected]
if declared != expected:
raise ValueError(
f"{where} declares stages its shared declaration function does "
f"not: {_first_difference(declared, expected)}"
)
def _first_difference(declared, expected) -> str:
"""Where the stages a layer declares first differ from those its shared
declaration function gives, field by field."""
if len(declared) != len(expected):
return (
f"it declares {[stage.kind.name for stage in declared]}, the "
f"function {[stage.kind.name for stage in expected]}"
)
index, (stage, given) = next(
(index, pair)
for index, pair in enumerate(zip(declared, expected))
if pair[0] != pair[1]
)
return f"stage {index} ({stage.kind.name}): " + "; ".join(
f"{name} is {value!r}, the function says {given_value!r}"
for name, value, given_value in _differing_fields(stage, given)
)
def _differing_fields(value, given, prefix=""):
"""(dotted name, value, given value) for each leaf field that differs,
descending into declarations and the facts they hold."""
if is_dataclass(value):
names = [field.name for field in fields(value)]
else:
names = getattr(type(value), "__struct_fields__", None)
if names is None or type(value) is not type(given):
return [(prefix.rstrip("."), value, given)]
return [
difference
for name in names
if getattr(value, name) != getattr(given, name)
for difference in _differing_fields(
getattr(value, name), getattr(given, name), f"{prefix}{name}."
)
]
def _neighbour_stage(layers, *, last):
"""The stage a neighbouring layer declares next to this stack: the last
one of the nearest layer before it, or the first of the nearest after.
Branches are side paths, so they never stand next to the stack."""
global _stack
for build_layer in build_layers:
for layer in layers:
outer = _stack
_stack = _LayerStack()
try:
build_layer()
appends = [a for a in _stack.appends if a.prepared_from is None]
result = layer()
declared = (
result.stages
if isinstance(result, DeclaredStages)
else [
declaration
for append in _stack.appends
if append.prepared_from is None
for declaration in append.declarations
]
)
finally:
_stack = outer
if appends:
return appends[-1].declarations[-1] if last else appends[0].declarations[0]
if declared:
return facts_of(declared[-1] if last else declared[0])
return None
@@ -938,15 +1031,6 @@ def _return_before_trailing_attention(line):
return
def _detached(declaration):
"""The declaration as the stage after it records it: without its own
history, so a bound stage holds its producer's facts and not the chain of
declarations before it."""
if declaration is None:
return None
return replace(declaration, previous=None, prepared_from=None)
def _bind_stack(appends, *, previous, following, final_read=None):
"""Bind every appended stage and fill in the boundaries each append
returned.
@@ -973,13 +1057,13 @@ def _bind_stack(appends, *, previous, following, final_read=None):
bound = {id(append): [None] * len(append.bindings) for append in appends}
# A returned declaration's stage as bound, for the branches that start from it.
sources = {}
producer = _detached(_handed_off(previous))
producer = facts_of(_handed_off(previous))
chained = []
for append, index in stages:
chained.append(
replace(append.declarations[index], previous=producer, prepared_from=None)
)
producer = _detached(chained[-1])
producer = facts_of(chained[-1])
connections = _connect_line(
chained,
[append.origin for append, _ in stages],
@@ -1010,8 +1094,8 @@ def _bind_stack(appends, *, previous, following, final_read=None):
chained.append(
replace(
declaration,
previous=chained[-1] if chained else None,
prepared_from=None if chained else source_declaration,
previous=facts_of(chained[-1]) if chained else None,
prepared_from=None if chained else facts_of(source_declaration),
)
)
try:
@@ -0,0 +1,99 @@
# Copyright 2023-2024 SGLang Team
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
"""What binding reads of a stage it does not run: the stage before or after
a stack on another pipeline rank, and the producer a bound stage records.
A stage's read, update and output transform are objects that hold the
layer's modules and state. Binding a neighbouring stage reads only the
capabilities they declare, so a neighbour, and the producer a stage keeps,
is recorded as those values alone: no module, bound method or declaration
history stays referenced once the stack is bound."""
from typing import Optional
import msgspec
class UpdateFacts(msgspec.Struct, frozen=True):
"""The capabilities a ResidualUpdate declares (see ResidualUpdate)."""
is_plain_add: bool
applied_at_exit: bool
outlives_layer: bool
writes_stream: bool
quantized_sum: bool = False
@classmethod
def of(cls, update) -> "UpdateFacts":
if isinstance(update, cls):
return update
return cls(
is_plain_add=update.is_plain_add,
applied_at_exit=update.applied_at_exit,
outlives_layer=update.outlives_layer,
writes_stream=update.writes_stream,
quantized_sum=update.quantized_sum,
)
class ReadFacts(msgspec.Struct, frozen=True):
"""The capabilities a ResidualReadout declares (see ResidualReadout).
Binding takes kernels only from the reads of the stages it binds, never
from a stage recorded by its facts, so these supply none."""
is_plain_norm: bool
reads_before_dp_gather: bool
reads_after_attn_tp_gather: bool
completing_fusions = ()
gathering_reads = ()
@classmethod
def of(cls, read) -> "ReadFacts":
if isinstance(read, cls):
return read
return cls(
is_plain_norm=read.is_plain_norm,
reads_before_dp_gather=read.reads_before_dp_gather,
reads_after_attn_tp_gather=read.reads_after_attn_tp_gather,
)
class TransformFacts(msgspec.Struct, frozen=True):
"""What an OutputTransform declares (see OutputTransform)."""
before_reduce_scatter: bool = False
@classmethod
def of(cls, transform) -> Optional["TransformFacts"]:
if transform is None or isinstance(transform, cls):
return transform
return cls(before_reduce_scatter=transform.before_reduce_scatter)
def facts_of(declaration):
"""``declaration`` with its read, update and output transform reduced to
the capabilities they declare, and without the declarations it was
chained to. None stays None."""
if declaration is None:
return None
from dataclasses import replace
return replace(
declaration,
read=ReadFacts.of(declaration.read),
update=UpdateFacts.of(declaration.update),
output_transform=TransformFacts.of(declaration.output_transform),
previous=None,
prepared_from=None,
)
@@ -335,7 +335,6 @@ def keep_output(
hidden_states: torch.Tensor,
residual: torch.Tensor,
forward_batch: ForwardBatch,
**kwargs,
):
return hidden_states, residual
@@ -346,7 +345,6 @@ def update_attn_tp_gather_output(
forward_batch: ForwardBatch,
update: ResidualUpdate = PLAIN_ADD,
gather: Optional[Callable] = None,
**kwargs,
):
"""Write the output into the residual on this rank's slice, then gather it
over attention TP (see attn_tp_gather_with)."""
@@ -371,7 +369,6 @@ def residual_slice_output(
sums: bool,
gathers_back: bool,
update: ResidualUpdate,
**kwargs,
):
"""Bring the FFN output onto the slice of the rows the residual is on:
a reduce-scatter that also completes its sum when ``sums``, else this
@@ -405,7 +402,6 @@ def attn_cp_take_back_output(
hidden_states: torch.Tensor,
residual: torch.Tensor,
forward_batch: ForwardBatch,
**kwargs,
):
"""DSA and MLA CP: this rank's shard of a complete output gathered in
equal shards over the attention-CP group."""
@@ -418,7 +414,6 @@ def attn_cp_reduce_scatter_output(
hidden_states: torch.Tensor,
residual: torch.Tensor,
forward_batch: ForwardBatch,
**kwargs,
):
"""DSA and MLA CP: sum the FFN output over the attention-CP group and
keep this rank's shard."""
@@ -429,7 +424,6 @@ def dp_cp_take_back_output(
hidden_states: torch.Tensor,
residual: torch.Tensor,
forward_batch: ForwardBatch,
**kwargs,
):
"""This rank's CP shard of the rows the DP gather put in its DP group's
slot, at the shard's padded length with the padding zeroed."""
@@ -450,7 +444,6 @@ def moe_cp_take_back_output(
hidden_states: torch.Tensor,
residual: torch.Tensor,
forward_batch: ForwardBatch,
**kwargs,
):
"""Return a MoE output computed on the MoE-CP-gathered rows to this rank's attention rows.
@@ -174,7 +174,7 @@ class StageBoundary:
completed or carried on. An FFN whose boundary has fusions (which may
take a MoE finalize handoff) finishes through exit() instead."""
if self.kind is StageKind.ATTENTION and self.plan.finishes_directly:
produced = self.plan.produced(forward_batch)
produced = self.plan.path_for(forward_batch).output
return stream_of(forward_batch).record(
hidden_states,
produced.update,
@@ -238,9 +238,6 @@ class StageBoundary:
)
return hidden_states
def snapshot(self, hidden_states, forward_batch):
return stream_of(forward_batch).snapshot(hidden_states)
def capture_output(self, hidden_states, forward_batch, *, skip_empty=False):
"""Capture an output while preserving the caller's reduction timing.
+1 -1
View File
@@ -607,7 +607,7 @@ class LongcatFlashModel(nn.Module):
for i in range(total_num_layers):
if i in self.layers_to_capture:
aux_hidden_states.append(
self.layers[i].attn_boundary.snapshot(hidden_states, forward_batch)
residual_batch.snapshot(hidden_states, forward_batch)
)
with get_global_expert_distribution_recorder().with_current_layer(i):
layer = self.layers[i]
+3 -1
View File
@@ -18,6 +18,7 @@
"""Inference-only NemotronH model."""
from collections.abc import Iterable
from functools import partial
import torch
from torch import nn
@@ -77,7 +78,7 @@ from sglang.srt.model_loader.weight_utils import (
replace_prefix,
replace_substrings,
)
from sglang.srt.models.nemotron_h_utils import make_stage_boundary
from sglang.srt.models.nemotron_h_utils import make_stage_boundary, stage_facts
from sglang.srt.models.utils import WeightsMapper
from sglang.srt.runtime_context import get_exec, get_parallel
from sglang.srt.utils import (
@@ -690,6 +691,7 @@ class NemotronHModel(nn.Module):
len(config.hybrid_override_pattern),
get_layer,
prefix=f"{prefix}.layers",
stage_facts=partial(stage_facts, config.hybrid_override_pattern),
)
if self.pp_group.is_last_rank:
self.norm_f = RMSNorm(config.hidden_size, eps=config.layer_norm_epsilon)
+9 -6
View File
@@ -37,15 +37,18 @@ def _declaration(pattern: str, layer_idx: int):
)
def stage_facts(pattern: str, layer_idx: int):
"""The one stage each Nemotron-H layer declares, from the pattern alone:
the model's shared declaration function, which its layers declare with
too (see make_layers)."""
return (_declaration(pattern, layer_idx),)
def make_stage_boundary(layer_norm: RMSNorm, *, pattern: str, layer_idx: int):
from sglang.srt.layers import layernorm_sp
if get_parallel().attn_cp_size > 1 or layernorm_sp.layernorm_sp_enabled():
raise NotImplementedError("a Nemotron stage with attention CP or LayerNorm SP")
(boundary,) = append_stages(
(
_declaration(pattern, layer_idx),
layer_norm,
),
)
(declaration,) = stage_facts(pattern, layer_idx)
(boundary,) = append_stages((declaration, layer_norm))
return boundary
+93 -9
View File
@@ -1591,18 +1591,27 @@ def make_layers(
return_tuple: bool = False,
offloader_kwargs: Optional[Dict[str, Any]] = None,
final_read: Optional[Any] = None,
stage_facts: Optional[Callable[[int], Sequence[Any]]] = None,
) -> Tuple[torch.nn.Module, int, int]:
"""Make a list of layers with the given layer function.
The local layers are built inside one layer stack, so layers that declare
stage boundaries connect in order without naming their neighbours. Across
a pipeline stage boundary the stack learns the neighbouring stage from the
layer itself, built again on the meta device. ``final_read`` is the
stack's terminal read when it is not a plain final norm (see layer_stack).
a pipeline stage boundary the stack learns the neighbouring stage from
``stage_facts``: the model's shared declaration function, which returns
the stages the layer at a global index declares without building it (an
empty sequence when it declares none), and which the layer itself uses
for its own declarations. A model on the transitional list may omit it;
the stack then builds the layer again on the meta device to read them.
``final_read`` is the stack's terminal read when it is not a plain final
norm (see layer_stack).
"""
# circular imports
from sglang.srt.distributed import get_pp_indices
from sglang.srt.layers.layer_boundary.factories import layer_stack
from sglang.srt.layers.layer_boundary.factories import (
check_declared_stages,
layer_stack,
)
from sglang.srt.layers.utils import PPMissingLayer
from sglang.srt.utils.offloader import get_offloader
@@ -1618,22 +1627,29 @@ def make_layers(
)
def neighbour(idx):
if stage_facts is not None:
return functools.partial(_declared_stages, stage_facts, idx)
return functools.partial(
_build_neighbour_layer, layer_fn, idx, add_prefix(idx, prefix)
)
def build(stack, idx):
appended = len(stack.appends)
layer = layer_fn(idx=idx, prefix=add_prefix(idx, prefix))
if stage_facts is not None and (pp_size or 1) > 1:
# Another rank binds this layer's stages from stage_facts alone.
check_declared_stages(stack, appended, stage_facts(idx), f"layer {idx}")
return layer
with layer_stack(
previous_layers=[neighbour(idx) for idx in reversed(range(start_layer))],
next_layers=[neighbour(idx) for idx in range(end_layer, num_hidden_layers)],
final_read=final_read,
):
) as stack:
modules = torch.nn.ModuleList(
[PPMissingLayer(return_tuple=return_tuple) for _ in range(start_layer)]
+ get_offloader().wrap_modules(
(
layer_fn(idx=idx, prefix=add_prefix(idx, prefix))
for idx in range(start_layer, end_layer)
),
(build(stack, idx) for idx in range(start_layer, end_layer)),
**(offloader_kwargs or {}),
)
+ [
@@ -1653,10 +1669,12 @@ def make_pp_layers(
return_tuple: bool = False,
offloader_kwargs: Optional[Dict[str, Any]] = None,
final_read: Optional[Any] = None,
stage_facts: Optional[Callable[[int], Sequence[Any]]] = None,
) -> Tuple[torch.nn.Module, int, int]:
"""Make this pipeline stage's layers, and return them with the stage's range.
Layers outside ``[start_layer, end_layer)`` are ``PPMissingLayer`` stand-ins.
``stage_facts`` is the model's shared declaration function (see make_layers).
"""
parallel = get_parallel()
return make_layers(
@@ -1668,15 +1686,81 @@ def make_pp_layers(
return_tuple=return_tuple,
offloader_kwargs=offloader_kwargs,
final_read=final_read,
stage_facts=stage_facts,
)
# The models whose layers a pipeline rank still builds on the meta device to
# read the stages they declare, until each gives make_layers its shared
# declaration function (stage_facts). Any other model must give one to run
# under pipeline parallelism.
_NEIGHBOUR_BUILD_MODULES = frozenset(
(
"sglang.srt.models.apertus",
"sglang.srt.models.arcee",
"sglang.srt.models.bailing_moe",
"sglang.srt.models.bailing_moe_linear",
"sglang.srt.models.bailing_moe_v3",
"sglang.srt.models.deepseek_v2",
"sglang.srt.models.dots3_common.modeling",
"sglang.srt.models.ernie45_moe_vl",
"sglang.srt.models.exaone4",
"sglang.srt.models.exaone_moe",
"sglang.srt.models.gemma4_causal",
"sglang.srt.models.gigachat35",
"sglang.srt.models.glm4",
"sglang.srt.models.glm4_moe",
"sglang.srt.models.glm4_moe_lite",
"sglang.srt.models.glm5_next",
"sglang.srt.models.gpt_oss",
"sglang.srt.models.granitemoehybrid",
"sglang.srt.models.kimi_k3",
"sglang.srt.models.kimi_linear",
"sglang.srt.models.laguna",
"sglang.srt.models.llada2",
"sglang.srt.models.llama",
"sglang.srt.models.mimo_v2",
"sglang.srt.models.minimax_m2",
"sglang.srt.models.minimax_m3",
"sglang.srt.models.ministral3",
"sglang.srt.models.mixtral",
"sglang.srt.models.nanbeige",
"sglang.srt.models.nemotron_nas",
"sglang.srt.models.qwen2",
"sglang.srt.models.qwen2_moe",
"sglang.srt.models.qwen3_5",
"sglang.srt.models.sarvam_moe",
"sglang.srt.models.sdar",
"sglang.srt.models.sdar_moe",
"sglang.srt.models.spark2_5",
"sglang.srt.models.step3p5",
"sglang.srt.models.xllm",
"sglang.srt.models.zaya",
)
)
def _declared_stages(stage_facts, idx: int):
"""The stages the layer at ``idx`` declares, from the model's shared
declaration function, standing in for building it."""
from sglang.srt.layers.layer_boundary.factories import DeclaredStages
return DeclaredStages(tuple(stage_facts(idx)))
def _build_neighbour_layer(layer_fn: LayerFn, idx: int, prefix: str) -> None:
"""Build a layer another pipeline stage holds, only for the stage
boundaries it declares (see building_neighbour_layer). RoPE modules it
adds to the shared cache are meta, so they are dropped again."""
from sglang.srt.layers.rotary_embedding.factory import _ROPE_DICT
module = getattr(getattr(layer_fn, "func", layer_fn), "__module__", None)
if module not in _NEIGHBOUR_BUILD_MODULES:
raise ValueError(
f"{module} declares stage boundaries under pipeline parallelism "
"without a shared declaration function: give make_layers its "
"stage_facts, the stages a layer declares without building it"
)
cached = set(_ROPE_DICT)
try:
with building_neighbour_layer():
@@ -22,6 +22,7 @@ from sglang.srt.layers.layer_boundary import (
layer_stack,
)
from sglang.srt.layers.layer_boundary import prepare as boundary_prepare
from sglang.srt.layers.layer_boundary.facts import facts_of
from sglang.srt.layers.layer_boundary.layout import SumGroup, TokenAxis
from sglang.srt.layers.layer_boundary.ops import (
attn_tp_gather_input,
@@ -37,6 +38,7 @@ from sglang.srt.layers.layer_boundary.residual.add_norm import (
from sglang.srt.layers.layer_boundary.residual.stream import DeclaredSum
from sglang.srt.layers.rotary_embedding import factory as rope_factory
from sglang.srt.model_executor.forward_batch_info import PPProxyTensors
from sglang.srt.utils import common
from sglang.srt.utils.common import is_building_neighbour_layer, make_layers
from sglang.test.ci.ci_register import register_cpu_ci
from sglang.test.communicator_patch import patch_communicator
@@ -169,7 +171,7 @@ class TestAppendStages(CustomTestCase):
self.assertTrue(all(s.plan is not None for s in branch))
# The branch reads the MoE's input; the next layer follows the main
# line's MoE, not the branch's dense FFN.
self.assertEqual(branch[0].declaration.prepared_from, moe.declaration)
self.assertEqual(branch[0].declaration.prepared_from, facts_of(moe.declaration))
self.assertTrue(following[0].declaration.previous.sparse)
def test_a_nested_stack_leaves_the_outer_one_untouched(self):
@@ -214,7 +216,7 @@ class TestAppendStages(CustomTestCase):
(declare_ffn(), fixture.Norm()), prepared_from=first.declaration
)
self.assertIsNot(first.declaration, second.declaration)
self.assertIs(branch.declaration.prepared_from, first.declaration)
self.assertEqual(branch.declaration.prepared_from, facts_of(first.declaration))
@unittest.skipUnless(hasattr(BaseException, "add_note"), "needs add_note")
def test_an_error_at_close_names_the_append_it_binds(self):
@@ -302,6 +304,15 @@ class TestMakeLayers(CustomTestCase):
self.planning = fixture.planning(fixture.parallel_of(attn_dp=1, attn_tp=2))
self.planning.__enter__()
self.addCleanup(self.planning.__exit__, None, None, None)
# These layers are built again on the meta device, as a model's on the
# transitional list are, unless a test gives a declaration function.
allowed = patch.object(
common,
"_NEIGHBOUR_BUILD_MODULES",
common._NEIGHBOUR_BUILD_MODULES | {__name__},
)
allowed.start()
self.addCleanup(allowed.stop)
def build(self, pp_rank=None, pp_size=None, *, declares=True):
"""Build the layers; layer 1 has a MoE. Returns each layer_fn call as
@@ -357,6 +368,56 @@ class TestMakeLayers(CustomTestCase):
self.assertFalse(last[2][0].plan.enters_stack)
self.assertTrue(last[3][1].plan.terminal)
def test_a_shared_declaration_function_stands_in_for_the_neighbours(self):
built, stages = [], {}
def layer_fn(idx, prefix):
built.append(idx)
stages[idx] = layer(sparse=idx == 1)
return nn.Identity()
def stage_facts(idx):
# The stages layer() declares, without building the layer.
return (
declare_attn(),
declare_ffn(sparse=idx == 1, next_layer_sparse=idx == 1),
)
make_layers(
NUM_LAYERS,
layer_fn,
pp_rank=1,
pp_size=2,
prefix="layers",
stage_facts=stage_facts,
)
self.assertEqual(built, [2, 3])
# Layer 2's attention follows layer 1's MoE, on the other stage.
self.assertTrue(stages[2][0].declaration.previous.sparse)
def test_a_layer_must_declare_what_its_declaration_function_says(self):
def layer_fn(idx, prefix):
layer(sparse=idx == 2)
return nn.Identity()
with self.assertRaisesRegex(
ValueError,
r"layer 2 declares stages .*: stage 1 \(FFN\): sparse is True, "
"the function says False",
):
make_layers(
NUM_LAYERS,
layer_fn,
pp_rank=1,
pp_size=2,
stage_facts=lambda idx: (declare_attn(), declare_ffn()),
)
def test_a_model_off_the_list_needs_a_declaration_function(self):
with patch.object(common, "_NEIGHBOUR_BUILD_MODULES", frozenset()):
with self.assertRaisesRegex(ValueError, "stage_facts"):
self.build(pp_rank=1, pp_size=2)
def test_layers_without_stages_build_no_neighbour(self):
calls, _ = self.build(pp_rank=1, pp_size=2, declares=False)
self.assertEqual([call[0] for call in calls], [2, 3])
@@ -22,6 +22,8 @@ from sglang.srt.layers.layer_boundary import (
layer_stack,
)
from sglang.srt.layers.layer_boundary.contracts import BatchVariant
from sglang.srt.layers.layer_boundary.facts import facts_of
from sglang.srt.layers.layer_boundary.output import OutputTransform
from sglang.srt.layers.layer_boundary.residual import batch
from sglang.srt.model_executor.forward_batch_info import ForwardMode
from sglang.test.boundary_fixtures import build_stages
@@ -189,9 +191,12 @@ class TestIndependentStageConstruction(CustomTestCase):
)
# The branch leaves the stage it reads from as it was.
self.assertEqual(source.plan.edges, alone.plan.edges)
self.assertEqual(first.declaration.prepared_from, source.declaration)
self.assertIs(attn.declaration.previous, first.declaration)
self.assertIs(last.declaration.previous, attn.declaration)
# Each records the stage before it by what binding reads of it.
self.assertEqual(
first.declaration.prepared_from, facts_of(source.declaration)
)
self.assertEqual(attn.declaration.previous, facts_of(first.declaration))
self.assertEqual(last.declaration.previous, facts_of(attn.declaration))
for v, edge in first.plan.edges.items():
source_input = source.plan.edges[v].incoming
self.assertEqual(
@@ -449,6 +454,33 @@ class TestPipelineCuts(CustomTestCase):
)
return True
def test_a_rank_after_an_attention_that_transforms_its_output_is_rejected(self):
# The attention leaves its sum, and its transform runs at the next
# stage's input: on the receiving rank, which does not hold it.
transformed = partial(declare_attn, output_transform=OutputTransform(abs))
for (name, (parallel, a2a)), kind in itertools.product(
CUT_CONFIGS.items(), CUT_STAGES
):
with (
self.subTest(config=name, receiver=kind),
fixture.planning(parallel, a2a=a2a),
):
# Without the transform, the same rank binds.
self.assertIsNone(bind_rank((kind,), before=("attention",))[1])
with (
self.assertRaisesRegex(
NotImplementedError,
"a pipeline rank that ends on an attention transforming "
"the output whose sum it leaves",
),
layer_stack(
previous_layers=[
lambda: append_stages((transformed(), fixture.Norm()))
]
),
):
append_stages(*cut_stages((kind,)))
def assert_rows_follow(self, stages, rank):
"""Within one rank's stages, each side of a boundary takes the
residual on the rows the other side leaves it on."""