diff --git a/docs/docs/advanced_features/server_arguments.mdx b/docs/docs/advanced_features/server_arguments.mdx index ba9f9e14c8..533b1025a5 100644 --- a/docs/docs/advanced_features/server_arguments.mdx +++ b/docs/docs/advanced_features/server_arguments.mdx @@ -2868,7 +2868,7 @@ Combining `--enable-response-store` with `--disaggregation-mode=prefill` or `dec ### FFN boundary reduction -Use `--boundary-reduction` to choose which optional FFN output reductions may combine the sum with token redistribution. The selected policy applies to the dense and MoE FFN stages of decoders built with independently constructed stage boundaries (`declare_ffn` / `make_stages`; see [Layer boundaries](/docs/developer_guide/layer_boundary)). Other models keep their built-in FFN output communication and ignore this option. +Use `--boundary-reduction` to choose which optional FFN output reductions may combine the sum with token redistribution. The selected policy applies to the dense and MoE FFN stages of decoders built with independently constructed stage boundaries (`declare_ffn` / `append_stages`; see [Layer boundaries](/docs/developer_guide/layer_boundary)). Other models keep their built-in FFN output communication and ignore this option. | Value | Behavior | | --- | --- | diff --git a/docs/docs/developer_guide/layer_boundary.mdx b/docs/docs/developer_guide/layer_boundary.mdx index 75491032a1..06121eaecb 100644 --- a/docs/docs/developer_guide/layer_boundary.mdx +++ b/docs/docs/developer_guide/layer_boundary.mdx @@ -97,41 +97,48 @@ The following constructor fragment assumes the model has already initialized its ```python from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) -self.attn_boundary, self.ffn_boundary = make_stages( +self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), (declare_ffn(), self.post_attention_layernorm), - previous=declare_ffn() if layer_id != 0 else None, - terminal=layer_id == config.num_hidden_layers - 1, ) ``` -`previous` describes the actual external producer. You do not need an executable object from the previous layer or another pipeline rank. For a sparse or custom preceding stage, build `previous` with the same arguments the producing layer uses for its own declaration, including `next_layer_sparse`, for example `declare_ffn(sparse=prev_sparse, next_layer_sparse=this_sparse)`; the two-batch overlap (TBO) exit rows derive from that pair. +`append_stages()` extends the open layer stack: the stack records the last stage appended, and the next append takes it as its producer, so a layer needs no description of its neighbours. `make_layers()` (and `make_pp_layers()`, which calls it) opens a stack around the layers it builds. A model that builds its layers another way, such as a hand-written `nn.ModuleList` or a NextN/multi-token-prediction (MTP) draft, opens the stack itself; appending outside an open stack raises: -`terminal=True` marks the end of the model's layer stack, not every Python decoder layer. A pipeline partition that has more layers downstream is not terminal merely because its local module list ends. A NextN/multi-token-prediction (MTP) draft module is its own stack: its layer enters with `previous=None` and is terminal even when it reuses the target's decoder class. In that case, use the draft's layer count, for example `terminal=layer_id == (1 if is_nextn else config.num_hidden_layers) - 1`. A terminal stage leaves no work for a following layer, except the final-norm finalize handoff described in [Enter and leave the layer stack](#enter-and-leave-the-layer-stack). +```python +from sglang.srt.layers.layer_boundary import layer_stack -`make_stages()` returns independent boundaries and retains no runtime sequence object. Each item may include a third element containing constructor options: an attention stage accepts `qkv_latent_func` (for example `{"qkv_latent_func": self.self_attn.prepare_qkv_latent}`) and `fusions`; an FFN stage accepts only `fusions` (see [Fusion providers](#fusion-providers-and-output-decisions)). Declarations passed into this assembler must not already carry `previous`, `prepared_from`, or `terminal`; provide those to the assembler. +with layer_stack(): + self.layers = nn.ModuleList( + DecoderLayer(config, layer_id) for layer_id in range(config.num_hidden_layers) + ) +``` -For a single independently constructed stage, use `make_attn_stage()` or `make_ffn_stage()`. Their declaration includes its source; an optional `following` declaration describes the local consumer and must name that producer in its `previous` field. +Every stage binds when the stack closes, once its producer and its consumer are known: the stage appended before it and the one appended after it. The last stage ends the model's layer stack and is marked terminal, unless a later layer declares a stage. A boundary's `declaration` is usable at once; when the stack closes, it is replaced by the declaration with its place in the stack (its `previous` and `terminal`), and the plan is bound. An error raised while binding carries a note naming the `append_stages()` call it came from. A terminal stage leaves no work for a following layer, except the final-norm finalize handoff described in [Enter and leave the layer stack](#enter-and-leave-the-layer-stack). A NextN/MTP draft module is its own stack: its layer enters the stack and ends it, even when it reuses the target's decoder class. + +Pipeline parallelism splits one layer stack across ranks, so the first and last local stages have neighbours that another rank builds. `make_layers()` finds them without communicating: when its stack closes, it builds the layer before the rank's first local layer and the one after its last again, with the model's own layer function inside `building_neighbour_layer()` (on the meta device), reads the stages they declare and discards them. A stage's declaration depends only on the config and the global layer index, so both ranks see the same boundary. A layer that declares no stage is passed over in favour of the next one out. These neighbours are built only if the stack appended stages, and after the local layers. A neighbour is never loaded or run, so a layer constructor must run on the meta device, must skip the host and device resources only a running layer needs (tables, streams, engines, communicators) when `is_building_neighbour_layer()` is true, and must not depend on which pipeline rank builds it. The meta device alone does not stop a constructor that names a device explicitly. + +Each item may include a third element containing constructor options: an attention stage accepts `qkv_latent_func` (for example `{"qkv_latent_func": self.self_attn.prepare_qkv_latent}`) and `fusions`; an FFN stage accepts only `fusions` (see [Fusion providers](#fusion-providers-and-output-decisions)). An option a stage does not take is rejected when appended. Pass declarations fresh from `declare_attn()` / `declare_ffn()`: one taken from a bound boundary already has its place in a stack, and is rejected. ### Single-stage mixers and heterogeneous stacks -For heterogeneous layer stacks, declare the actual neighbouring stage kinds; the framework does not require an alternating attention/FFN pattern. `make_stages(..., following=declaration)` describes the external consumer after the last local stage, just as `previous` describes the producer before the first. +For heterogeneous layer stacks, append the actual stages in order; the framework does not require an alternating attention/FFN pattern, and a layer may append a single stage. -For a single-stage `ProducerReduction.EXIT_SCOPED` mixer, `following` decides whether the mixer leaves its attention-TP sum to the next stage: always before an FFN, and per batch before another mixer. The next stage makes the same decision from its own `previous`, so both must describe the same neighbours. Otherwise a pipeline handoff reduces the mixer's output twice: `to_pp()` exports it complete and the receiver's `from_pp()` declares the sum again. +A single-stage `ProducerReduction.EXIT_SCOPED` mixer's consumer decides whether the mixer leaves its attention-TP sum to the next stage: always before an FFN, and per batch before another mixer. The stack binds the mixer only once its consumer is known, so the two stages always agree, across a pipeline boundary too: there the consumer is the next rank's first stage, which `make_layers()` reads from the neighbouring layer. Bound without its consumer, the mixer would complete the sum before `to_pp()`, and the receiving FFN's `from_pp()` would declare the sum again. -`models/nemotron_h_utils.py` (used by `models/nemotron_h.py`) builds each Nemotron-H stage as `make_stages((decl, norm), previous=..., following=..., terminal=...)`, with mixers declared as `declare_attn(reduction=ProducerReduction.EXIT_SCOPED, gathers_attn_tp_input=False)`. +`models/nemotron_h_utils.py` (used by `models/nemotron_h.py`) builds each Nemotron-H stage as `append_stages((decl, norm))`, with mixers declared as `declare_attn(reduction=ProducerReduction.EXIT_SCOPED, gathers_attn_tp_input=False)`. ### More than two stages and branches There is no two-stage limit. For example, LongCat constructs a dense FFN, attention, and another dense FFN from an input already prepared for its MoE branch: ```python -first_ffn, second_attn, second_ffn = make_stages( +first_ffn, second_attn, second_ffn = append_stages( (declare_ffn(), first_ffn_norm), (declare_attn(), second_attn_norm), (declare_ffn(), second_ffn_norm), @@ -139,7 +146,7 @@ first_ffn, second_attn, second_ffn = make_stages( ) ``` -This fragment assumes the source boundary and the three norms already exist. `prepared_from` means the source has already performed its read. Enter through `branch_input(source, hidden_states, forward_batch)`, not `prepare()`, to move that input and fork the residual stream without normalizing twice. The first stage's norm is therefore never applied; LongCat passes the MoE stage's norm there. +This fragment assumes the source boundary and the three norms already exist. `prepared_from` means the source has already performed its read. A branch is a side path that the model merges back explicitly: it neither extends the stack nor waits for a consumer, so the next append still follows the source. Enter through `branch_input(source, hidden_states, forward_batch)`, not `prepare()`, to move that input and fork the residual stream without normalizing twice. The first stage's norm is therefore never applied; LongCat passes the MoE stage's norm there. The branch adapter also provides `branch_output()` to complete the sum a branch's contribution owes and place it on the branch's exit rows and `merge_branch()` to combine it with another branch's output. These methods encode row movement and residual ownership; the model still determines its compute schedule. Branch transport currently supports only ordinary token rows: construction rejects context-parallel, input-scattered, and sequence-parallel variants. See `python/sglang/srt/models/longcat_flash.py` for a complete integration. @@ -218,7 +225,7 @@ Snapshots currently require a plain residual update. Producer-specific finalize A separate BF16 snapshot followed by norm can change results relative to a fused add+norm kernel's FP32 accumulation. Capture from the existing update/read operation when that ordering matters. -A capture callback has the signature `capture(value, *, owned=False)`, as `AuxHiddenStateList.capture` does. With `owned=False` the value may alias the mutable residual or a reusable communication buffer, and the collector must copy it before retaining it. The boundary passes `owned=True` only for storage nothing else will write: a capture move that gathers rows (`EntryPath.capture_move_allocates`), a residual sum computed only for the capture, or a residual that the following local stage's input path certifies it leaves untouched. `make_stages()` copies that certificate into the producer's entry at construction; no plan is consulted during forward. `residual_batch.final_norm(..., capture=...)` always passes a borrowed value. +A capture callback has the signature `capture(value, *, owned=False)`, as `AuxHiddenStateList.capture` does. With `owned=False` the value may alias the mutable residual or a reusable communication buffer, and the collector must copy it before retaining it. The boundary passes `owned=True` only for storage nothing else will write: a capture move that gathers rows (`EntryPath.capture_move_allocates`), a residual sum computed only for the capture, or a residual that the following local stage's input path certifies it leaves untouched. `append_stages()` copies that certificate into the producer's entry at construction; no plan is consulted during forward. `residual_batch.final_norm(..., capture=...)` always passes a borrowed value. ### Choose a reduction policy @@ -238,9 +245,8 @@ Required attention and MoE collectives are unaffected. These are permissions, no Before you open a pull request for a new model, check each item: -- Every layer passes the real producer as `previous`, including `sparse`/`next_layer_sparse`, and `previous=None` only where the stack starts. -- `terminal` is set only on the last layer of the model's stack, using the draft's own layer count for NextN/MTP modules. -- A single-stage mixer or FFN also passes `following`, and both neighbours describe each other consistently. +- Every decoder layer appends its stages inside a layer stack: `make_layers()`, or `layer_stack()` around any other construction, including a NextN/MTP draft's own layer. +- The layer constructor runs on the meta device, allocates nothing when `is_building_neighbour_layer()` is true, and declares the same stages whichever pipeline rank builds it: `make_layers()` builds the layers next to a rank's own that way to read their stages. - The model forward calls `residual_batch.start()` before the first layer, and leaves the stack through `residual_batch.final_norm()`, `to_pp()`, or `take_output()`. - Pipeline reception calls `from_pp(tensors, forward_batch)` on the first local boundary. - Split prefill starts the stream only in the segment that runs the first layer and completes it only in the final segment. @@ -255,6 +261,7 @@ Common errors and their usual cause: | `output does not belong to this residual stream; change layer outputs through boundary accessors` | The model replaced a layer's output object (for example rebinding it to a new tensor) or passed another microbatch's output. This check compares object identity, so it does not catch in-place changes; the interface still forbids modifying layer outputs in place | | `write the residual update before taking the final output` | `take_output()` ran while a contribution was still pending; fold or norm it first | | `no stage boundary path for the active batch` | The active batch variant (CP, input-scattered, sequence parallel) is not supported by this stage; add a server-argument check that rejects the configuration | +| `append_stages needs an open layer stack` | The layers are built outside `make_layers()`; wrap their construction in `layer_stack()` | | `a prepared branch must enter through branch_input, not prepare` | A `prepared_from` stage was entered with `prepare()` | | `snapshot requires a plain residual update` | A snapshot was taken on a nonlinear (for example MHC) update | @@ -279,7 +286,7 @@ flowchart TD Construction and execution have different lifetimes: -1. **At model initialization**, `declare_attn()` and `declare_ffn()` describe compute behavior. `make_stages()` connects those declarations and binds each stage's own norm and hooks. +1. **At model initialization**, `declare_attn()` and `declare_ffn()` describe compute behavior. `append_stages()` connects those declarations in stack order and binds each stage's own norm and hooks. 2. **During construction**, factories resolve token rows and reduction contracts for supported batch variants. `bind_entry()` binds consumer work; `bind_exit()` binds output transport. `StagePlan` retains those paths. 3. **At each forward**, batch facts select an existing path. An exit can make a batch-dependent completion decision, but it does not rebuild the declarations. The decision supplies the matching completion action; the only flag it publishes to compute is `defer_moe_finalize`, for a MoE that may hand off its finalize. 4. **Across stages**, `ForwardBatch.residual_stream` carries residual state and the producer's actual contribution. Stages do not store forward tensors on their reusable plans. @@ -353,4 +360,4 @@ Place changes according to their responsibility: 4. Put backend completion in `fusions/` and specialized exit and entry behavior in `adapters/`. 5. Keep models on stage factories and boundary methods. Avoid reading a neighbour's execution plan or reconstructing a collective decision during forward. -Focused tests live in `test/registered/unit/layer_boundary/`. Cover numerical output and residual state, collective group/row correctness, and supported fallback behavior for the changed path. Relevant examples include `test_ffn_exit.py`, `test_reduction_fusion.py`, and `test_terminal_stages.py`. A declaration-only assertion cannot establish that a new parallel path moves and reduces the correct values. +Focused tests live in `test/registered/unit/layer_boundary/`. Cover numerical output and residual state, collective group/row correctness, and supported fallback behavior for the changed path. Relevant examples include `test_ffn_exit.py`, `test_reduction_fusion.py`, and `test_append_stages.py`. A declaration-only assertion cannot establish that a new parallel path moves and reduces the correct values. diff --git a/python/sglang/srt/layers/activation.py b/python/sglang/srt/layers/activation.py index 7c1bf82fed..375dcde960 100644 --- a/python/sglang/srt/layers/activation.py +++ b/python/sglang/srt/layers/activation.py @@ -338,8 +338,8 @@ class XIELU(BaseFusedOp): self.register_buffer("eps", torch.tensor(eps, dtype=dtype)) self.with_vector_loads = with_vector_loads # Temporary until xIELU CUDA fully implemented - self._beta_scalar = float(self.beta.detach().cpu().float().item()) - self._eps_scalar = float(self.eps.detach().cpu().float().item()) + self._beta_scalar = torch.tensor(beta, dtype=dtype, device="cpu").item() + self._eps_scalar = torch.tensor(eps, dtype=dtype, device="cpu").item() self._xielu_cuda_obj = None try: diff --git a/python/sglang/srt/layers/hyperconnection.py b/python/sglang/srt/layers/hyperconnection.py index cac970b0e1..51671019d5 100644 --- a/python/sglang/srt/layers/hyperconnection.py +++ b/python/sglang/srt/layers/hyperconnection.py @@ -139,14 +139,12 @@ class GatedResidual(HyperConnectionBase): self.hidden_size * self.hc_count, self.config.hc_lowrank, bias=False, - device=torch.get_device_module().current_device(), dtype=config.params_dtype, ) self.input_mix_weight_up = nn.Linear( self.config.hc_lowrank, self.hc_count * self.hidden_size, bias=False, - device=torch.get_device_module().current_device(), dtype=config.params_dtype, ) lowrank = self.config.hc_lowrank @@ -166,7 +164,6 @@ class GatedResidual(HyperConnectionBase): self.hidden_size * self.hc_count, self.hc_count, bias=False, - device=torch.get_device_module().current_device(), dtype=config.params_dtype, ) # hc_combine rejects other shapes; device and dtype are checked per call. diff --git a/python/sglang/srt/layers/layer_boundary/__init__.py b/python/sglang/srt/layers/layer_boundary/__init__.py index dffaa6ea30..c2b6b3e74c 100644 --- a/python/sglang/srt/layers/layer_boundary/__init__.py +++ b/python/sglang/srt/layers/layer_boundary/__init__.py @@ -40,11 +40,10 @@ from sglang.srt.layers.layer_boundary.contracts import ( ) from sglang.srt.layers.layer_boundary.exit import ExitDecision, FfnExit, MixerExit from sglang.srt.layers.layer_boundary.factories import ( + append_stages, declare_attn, declare_ffn, - make_attn_stage, - make_ffn_stage, - make_stages, + layer_stack, ) from sglang.srt.layers.layer_boundary.layout import ( Layout, @@ -85,9 +84,8 @@ from sglang.srt.layers.layer_boundary.residual.mhc import ( __all__ = [ "declare_attn", "declare_ffn", - "make_attn_stage", - "make_ffn_stage", - "make_stages", + "append_stages", + "layer_stack", "PLAIN_ADD", "AttentionInputs", "StagePath", diff --git a/python/sglang/srt/layers/layer_boundary/construction.py b/python/sglang/srt/layers/layer_boundary/construction.py index ddcaf0a6e8..460d80d71c 100644 --- a/python/sglang/srt/layers/layer_boundary/construction.py +++ b/python/sglang/srt/layers/layer_boundary/construction.py @@ -145,8 +145,11 @@ class StagePlan: finishes_directly: Attention can publish its output with finish instead of an exit scope and output transport. qkv_latent_func: Optional hook for prepared attention input. - fusions: Optional backend provider of ordered consumer fusion candidates - and the producer deferral policies (see make_attn_stage). + fusions: Optional backend provider. Consumer side: ordered + attention_input(plan) and ffn_input(plan) candidates. Producer side + (FFN exit): can_defer_finalize(plan, batch), called on every exit, + and can_defer_all_reduce(plan, batch), called when LoRA or TP1 + shared experts are enabled. The plan owns static paths, not per-forward tensors or a neighbour's norm. Runtime residual state belongs to the ForwardBatch's ResidualStream. diff --git a/python/sglang/srt/layers/layer_boundary/factories.py b/python/sglang/srt/layers/layer_boundary/factories.py index d842c10770..73c08eeda6 100644 --- a/python/sglang/srt/layers/layer_boundary/factories.py +++ b/python/sglang/srt/layers/layer_boundary/factories.py @@ -2,6 +2,9 @@ from __future__ import annotations +import contextlib +import os +import sys from dataclasses import dataclass, replace from types import MappingProxyType from typing import Mapping, Optional @@ -48,6 +51,7 @@ from sglang.srt.layers.layer_boundary.residual.add_norm import ( PLAIN_ADD, REPLACE_AT_EXIT, ) +from sglang.srt.layers.layer_boundary.stage import StageBoundary from sglang.srt.layers.moe import is_moe_input_scattered_across_dp_ranks from sglang.srt.runtime_context import get_exec, get_parallel @@ -194,9 +198,10 @@ class StageDeclaration: update: Producer operation that writes compute output into the residual. sparse: Whether this FFN is a MoE; used to resolve input/output rows. Expert routing and all-to-all remain inside the MoE computation. - terminal: Whether this stage ends the model's layer stack. Prevents - leaving work that requires a following layer; a finalize handoff - may still reach the terminal norm when the fusion provider allows it. + terminal: Whether this stage ends the model's layer stack, as the + stack records it. Prevents leaving work that requires a following + layer; a finalize handoff may still reach the terminal norm when + the fusion provider allows it. output_transform: Optional operation on the contribution before the residual update. An FFN's exit runs it under an explicit reduction-order contract; an attention's is run by the input of @@ -207,8 +212,8 @@ class StageDeclaration: dense_tp_size: Dense FFN compute width: None uses the configured width, 1 means local compute, and the full TP size means TP compute. exit_rows: Required FFN output rows at the layer or branch exit. - previous: Declaration whose output this stage consumes. It may be - reconstructed locally, including across pipeline ranks. + previous: Declaration whose output this stage consumes, as the stack + records it; across pipeline ranks it is built locally. prepared_from: Declaration whose already-read input a branch reuses. Mutually exclusive with previous; avoids a second update/read. @@ -263,11 +268,8 @@ class StageConnection: def declare_attn( *, - previous: Optional[StageDeclaration] = None, - prepared_from: Optional[StageDeclaration] = None, read=NORM_QUANT_READOUT, update=PLAIN_ADD, - terminal=False, reduction=ProducerReduction.ALWAYS_PARTIAL, gathers_attn_tp_input=True, output_transform=None, @@ -275,11 +277,8 @@ def declare_attn( """Declare attention or a mixer; construct its executable boundary later. Args: - previous: Producer declaration whose output this stage consumes. - prepared_from: Source of an already-read branch input, instead of previous. read: Input operation; defaults to normalization with quantization support. update: Operation that adds this stage's output to the residual. - terminal: Whether this stage ends the model's layer stack. reduction: ALWAYS_PARTIAL for an attention whose sum the next stage's input always completes; EXIT_SCOPED for a mixer whose exit decides. gathers_attn_tp_input: Whether compute gathers attention-TP input slices itself. @@ -300,9 +299,6 @@ def declare_attn( StageKind.ATTENTION, read, update, - previous=previous, - prepared_from=prepared_from, - terminal=terminal, output_transform=output_transform, reduction=reduction, gathers_attn_tp_input=gathers_attn_tp_input, @@ -311,12 +307,9 @@ def declare_attn( def declare_ffn( *, - previous: Optional[StageDeclaration] = None, - prepared_from: Optional[StageDeclaration] = None, sparse=False, read=NORM_READOUT, update=PLAIN_ADD, - terminal=False, output_transform=None, next_layer_sparse=False, dense_tp_size=None, @@ -325,12 +318,9 @@ def declare_ffn( """Declare a dense or MoE FFN independently of its compute module. Args: - previous: Producer declaration whose output this stage consumes. - prepared_from: Source of an already-read branch input, instead of previous. sparse: Whether the FFN is a MoE; EP dispatch stays inside compute. read: Operation deriving FFN input from the residual. update: Operation writing FFN output into the residual. - terminal: Whether this stage ends the model's layer stack. output_transform: Optional contribution transform before residual update. next_layer_sparse: Whether the next decoder layer's FFN is sparse; used only to derive the TBO exit rows when exit_rows is not supplied. @@ -347,9 +337,6 @@ def declare_ffn( read, update, sparse=sparse, - previous=previous, - prepared_from=prepared_from, - terminal=terminal, output_transform=output_transform, dense_tp_size=dense_tp_size, exit_rows=exit_rows or tbo_exit_rows(sparse, next_layer_sparse), @@ -569,167 +556,364 @@ def _incoming(stage): ) -def _connections(stage, following=None): - if following is not None and following.previous != stage: - raise ValueError("the following declaration must consume this stage's output") - incoming = _incoming(stage) - outgoing = _connect( - stage, - following, - residual_from=incoming, - ) - return incoming, outgoing +class _Append: + """One append_stages call: its validated stages, the boundaries it + returned, which the stack fills in when it closes, and where it was made.""" + + __slots__ = ("declarations", "bindings", "prepared_from", "boundaries", "origin") + + def __init__(self, declarations, bindings, prepared_from, boundaries, origin): + self.declarations = declarations + self.bindings = bindings + self.prepared_from = prepared_from + self.boundaries = boundaries + self.origin = origin -def make_attn_stage( - *, - declaration, - norm, - following: Optional[StageDeclaration] = None, - qkv_latent_func=None, - fusions=None, -): - """Resolve one attention/mixer's boundaries and bind its input norm. +class _LayerStack: + """A layer stack under construction: what was appended, in order, and how + to reach the layers other pipeline ranks hold on either side of it.""" + + __slots__ = ("appends", "previous_layers", "next_layers") + + def __init__(self, previous_layers=(), next_layers=()): + self.appends = [] + self.previous_layers = previous_layers + self.next_layers = next_layers + + +# The stack being built; layer_stack saves and restores an outer one. +_stack: Optional[_LayerStack] = None + + +@contextlib.contextmanager +def layer_stack(*, previous_layers=(), next_layers=()): + """Open a layer stack that append_stages extends in order. + + Every stage binds when the stack closes, once its producer and its + consumer are both known: the stage appended before and after it. The last + stage ends the model's layer stack unless a later layer declares a stage. Args: - declaration: Attention StageDeclaration, including its input source. - norm: This consumer's normalization module, never its neighbour's norm. - following: Local consumer declaration whose previous is declaration. - None leaves a layer or stack exit for an independently bound reader. - qkv_latent_func: Optional attention input hook, invoked after preparation - and movement onto the compute input rows. - fusions: Optional backend provider. Consumer side: ordered - attention_input(plan) and ffn_input(plan) candidates. Producer side - (FFN exit): can_defer_finalize(plan, batch), called on every exit, - and can_defer_all_reduce(plan, batch), called when LoRA or TP1 shared - experts are enabled. - - Returns: - A StageBoundary with precomputed paths for supported batch variants. + previous_layers: Callables that build, nearest first, the layers before + this stack that another pipeline rank holds. Called only if this + stack appended stages, after its own layers are built, until one of + them declares a stage: its last stage is the producer of this + stack's first. What they build is discarded. + next_layers: Likewise for the layers after this stack, whose first + declared stage is the consumer of this stack's last. """ - if declaration.kind is not StageKind.ATTENTION: - raise TypeError("make_attn_stage requires an attention declaration") - incoming, outgoing = _connections(declaration, following) - return _bind_stage( + global _stack + outer = _stack + stack = _LayerStack(previous_layers, next_layers) + _stack = stack + try: + yield stack + if stack.appends: + _bind_stack( + stack.appends, + previous=_neighbour_stage(stack.previous_layers, last=True), + following=_neighbour_stage(stack.next_layers, last=False), + ) + finally: + _stack = outer + + +def _neighbour_stage(build_layers, *, last): + """The stage a neighbouring layer declares next to this stack: the last + one of the nearest layer before it, or the first of the nearest after. + Branches are side paths, so they never stand next to the stack.""" + global _stack + for build_layer in build_layers: + outer = _stack + _stack = _LayerStack() + try: + build_layer() + appends = [a for a in _stack.appends if a.prepared_from is None] + finally: + _stack = outer + if appends: + return appends[-1].declarations[-1] if last else appends[0].declarations[0] + return None + + +def _require_stack() -> _LayerStack: + if _stack is None: + raise RuntimeError( + "append_stages needs an open layer stack; build the layers inside " + "make_layers or layer_stack" + ) + return _stack + + +def _note_origin(error: Exception, origin: str) -> None: + add_note = getattr(error, "add_note", None) + if add_note is not None: + add_note(f"while binding the stages appended at {origin}") + + +class _PendingStage: + """The last stage of a linear append. Its consumer is the next append's + first stage, or the stack exit, so it binds only once that is known.""" + + __slots__ = ( + "boundary", + "declaration", + "norm", + "options", + "predecessor", + "predecessor_incoming", + "predecessor_boundary", + "origin", + ) + + def __init__( + self, + boundary, declaration, norm, - incoming, - outgoing, - qkv_latent_func=qkv_latent_func, - fusions=fusions, - ) + options, + predecessor, + predecessor_incoming, + predecessor_boundary, + origin, + ): + self.boundary = boundary + self.declaration = declaration + self.norm = norm + self.options = options + # The stage before it within the same append, or None when it is the + # append's only stage and takes its input from the stack. + self.predecessor = predecessor + self.predecessor_incoming = predecessor_incoming + self.predecessor_boundary = predecessor_boundary + self.origin = origin -def make_ffn_stage( - *, - declaration, - norm, - following: Optional[StageDeclaration] = None, - fusions=None, -): - """Resolve one FFN's boundaries and bind its input norm. +class _Chain: + """The stage a following append extends, and the one stage still waiting + for its consumer, while the stack binds its appends in order.""" - Args: - declaration: FFN StageDeclaration, including its input source. - norm: This FFN's input normalization module. - following: Local consumer declaration whose previous is declaration; - None leaves a layer or stack exit for an independently bound reader. - fusions: Optional backend fusion provider, as in make_attn_stage. + __slots__ = ("previous", "pending") - Returns: - A StageBoundary. Expert routing and all-to-all stay inside compute. + def __init__(self, previous): + self.previous = _detached(previous) + self.pending = None + + +def _detached(declaration): + """The declaration a following append extends, without its own history. + + A stage's incoming edge reads its producer's own incoming edge only when + the producer is an attention that always leaves its sum; the copy has no + history, so that lookback stops at the producer. """ - if declaration.kind is not StageKind.FFN: - raise TypeError("make_ffn_stage requires an FFN declaration") - incoming, outgoing = _connections(declaration, following) - return _bind_stage( - declaration, - norm, - incoming, - outgoing, - fusions=fusions, - ) + if declaration is None: + return None + return replace(declaration, previous=None, prepared_from=None) -def make_stages( - *stages, previous=None, prepared_from=None, following=None, terminal=False -): - """Bind a local linear sequence of any positive number of stages. +def _bind_stack(appends, *, previous, following): + """Bind every appended stage, in order, and fill in the boundaries each + append returned.""" + chain = _Chain(previous) + # A returned declaration's boundary as bound, for the branches that read it. + sources = {} + bound = [] + for append in appends: + prepared_from = append.prepared_from + if prepared_from is not None: + source = sources.get(id(prepared_from)) + if source is None: + raise ValueError( + "prepared_from must be the declaration of a stage appended " + f"earlier to the same layer stack (at {append.origin})" + ) + prepared_from = source.declaration + boundaries = _extend(chain, append, prepared_from) + for returned, boundary in zip(append.boundaries, boundaries): + sources[id(returned.declaration)] = boundary + bound.append(boundaries) + # The last stage's consumer is the next rank's first stage, if any; + # without one it ends the model's layer stack. + _bind_pending(chain, consumer=following, terminal=following is None) + for append, boundaries in zip(appends, bound): + for returned, boundary in zip(append.boundaries, boundaries): + returned.plan = boundary.plan + returned.declaration = boundary.declaration - Args: - *stages: Items of (declaration, norm) or (declaration, norm, options). - Declarations must have no source or terminal flag. Options are - constructor keywords: fusions, and qkv_latent_func for attention. - previous: External producer declaration consumed by the first stage. - prepared_from: Already-read input declaration reused by the first stage - of a branch; mutually exclusive with previous. - following: External consumer declaration after the last local stage. - None denotes a layer or stack exit with an independently bound read. - terminal: Marks only the final stage as the end of the model's stack. - Returns: - A tuple of independent StageBoundary objects in declaration order. - Sources are connected on copied declarations; caller inputs are unchanged. - No sequence object or runtime routing is retained. - """ - if not stages: - raise ValueError("make_stages needs at least one stage") - if previous is not None and prepared_from is not None: - raise ValueError("choose a previous output or a prepared branch input") - declarations = [] - bindings = [] - for index, item in enumerate(stages): +def _bind_pending(chain: _Chain, *, consumer, terminal): + pending = chain.pending + if pending is None: + return + chain.pending = None + try: + declaration = replace(pending.declaration, terminal=terminal) + if pending.predecessor is None: + incoming = _incoming(declaration) + else: + incoming = _connect( + pending.predecessor, + declaration, + residual_from=pending.predecessor_incoming, + ) + outgoing = _connect(declaration, consumer, residual_from=incoming) + bound = _bind_stage( + declaration, pending.norm, incoming, outgoing, **pending.options + ) + except Exception as error: + _note_origin(error, pending.origin) + raise + pending.boundary.plan = bound.plan + pending.boundary.declaration = declaration + if pending.predecessor_boundary is not None: + _carry_capture(pending.predecessor_boundary, pending.boundary) + + +def _carry_capture(producer, consumer): + """Let an attention that always leaves its sum preserve the residual its + FFN's entry needs for capture. Only between stages of one append.""" + if not ( + producer.kind is StageKind.ATTENTION + and producer.declaration.reduction is ProducerReduction.ALWAYS_PARTIAL + and consumer.kind is StageKind.FFN + ): + return + for variant, steps in producer.plan.paths.items(): + next_steps = consumer.plan.paths[variant] + predicate = next_steps.entry.preserves_residual + if predicate is not None: + producer.plan.paths[variant] = msgspec.structs.replace( + steps, + entry=msgspec.structs.replace( + steps.entry, capture_preserves_residual=predicate + ), + ) + + +_STAGE_OPTIONS = { + StageKind.ATTENTION: frozenset({"qkv_latent_func", "fusions"}), + StageKind.FFN: frozenset({"fusions"}), +} + + +def _stage_items(stages): + declarations, bindings = [], [] + for item in stages: if len(item) not in (2, 3): raise ValueError("a stage needs (declaration, norm[, options])") declaration, norm = item[:2] if not isinstance(declaration, StageDeclaration): - raise TypeError("make_stages requires stage declarations") + raise TypeError("append_stages requires stage declarations") if declaration.previous is not None or declaration.prepared_from is not None: - raise ValueError("pass input sources to make_stages, not its declarations") + raise ValueError("a declaration takes its source from the layer stack") if declaration.terminal: - raise ValueError("pass terminal to make_stages, not its declarations") - declaration = replace( - declaration, - previous=previous if index == 0 else declarations[-1], - prepared_from=prepared_from if index == 0 else None, - terminal=terminal and index == len(stages) - 1, - ) + raise ValueError("the layer stack marks the terminal stage") + options = dict(item[2] if len(item) == 3 else {}) + unexpected = options.keys() - _STAGE_OPTIONS[declaration.kind] + if unexpected: + raise TypeError( + f"unsupported {declaration.kind.name} stage options: " + f"{sorted(unexpected)}" + ) declarations.append(declaration) - bindings.append((norm, item[2] if len(item) == 3 else {})) - external_consumer = following - boundaries = [] - incoming = _incoming(declarations[0]) - for index, (declaration, (norm, options)) in enumerate(zip(declarations, bindings)): - following = ( - declarations[index + 1] - if index + 1 < len(declarations) - else external_consumer - ) - outgoing = _connect(declaration, following, residual_from=incoming) - options = dict(options) - if declaration.kind is StageKind.FFN: - unexpected = options.keys() - { - "fusions", - } - if unexpected: - raise TypeError(f"unsupported FFN options: {sorted(unexpected)}") - boundaries.append(_bind_stage(declaration, norm, incoming, outgoing, **options)) - incoming = outgoing - for producer, consumer in zip(boundaries, boundaries[1:]): - if not ( - producer.kind is StageKind.ATTENTION - and producer.declaration.reduction is ProducerReduction.ALWAYS_PARTIAL - and consumer.kind is StageKind.FFN - ): - continue - for variant, steps in producer.plan.paths.items(): - next_steps = consumer.plan.paths[variant] - predicate = next_steps.entry.preserves_residual - if predicate is not None: - producer.plan.paths[variant] = msgspec.structs.replace( - steps, - entry=msgspec.structs.replace( - steps.entry, capture_preserves_residual=predicate + bindings.append((norm, options)) + return declarations, bindings + + +def _origin(stack: _LayerStack) -> str: + """Where the current append_stages call was made, and its place in the stack.""" + frame = sys._getframe(2) + return ( + f"{os.path.basename(frame.f_code.co_filename)}:{frame.f_lineno} " + f"(append {len(stack.appends)} of its layer stack)" + ) + + +def append_stages(*stages, prepared_from=None): + """Append a local linear sequence of stages to the open layer stack. + + Args: + *stages: Items of (declaration, norm) or (declaration, norm, options). + Declarations carry no source or terminal flag. Options are + constructor keywords: fusions, and qkv_latent_func for attention. + prepared_from: The ``declaration`` of a boundary an earlier call + returned on this stack: the already-read stage this sequence + branches from. A branch is a side path the model merges back + explicitly: it neither extends the stack nor waits for a consumer, + so each of its stages binds with only the stages of the branch. + + Returns: + A tuple of StageBoundary objects in declaration order. Their + declarations are usable at once and gain their place in the stack when + it closes, which is also when their plans are bound. + """ + if not stages: + raise ValueError("append_stages needs at least one stage") + stack = _require_stack() + declarations, bindings = _stage_items(stages) + declarations = [replace(d) for d in declarations] + boundaries = tuple(StageBoundary(None, declaration=d) for d in declarations) + stack.appends.append( + _Append(declarations, bindings, prepared_from, boundaries, _origin(stack)) + ) + return boundaries + + +def _extend(chain, append, prepared_from): + """Chain one append onto the stack and bind what can be bound.""" + declarations, bindings = append.declarations, append.bindings + branch = prepared_from is not None + if not branch: + # This append's first stage is the consumer the pending stage waited for. + _bind_pending(chain, consumer=declarations[0], terminal=False) + try: + chained = [] + for index, declaration in enumerate(declarations): + chained.append( + replace( + declaration, + previous=( + chained[-1] if index else (None if branch else chain.previous) ), + prepared_from=prepared_from if index == 0 else None, ) + ) + boundaries = [] + last = len(chained) - 1 + # incoming feeds the stage being bound; previous_incoming fed the one + # before. + previous_incoming, incoming = None, _incoming(chained[0]) + for index, (declaration, (norm, options)) in enumerate(zip(chained, bindings)): + if index == last and not branch: + boundaries.append(StageBoundary(None, declaration=declaration)) + chain.pending = _PendingStage( + boundaries[-1], + declaration, + norm, + options, + predecessor=chained[index - 1] if index else None, + predecessor_incoming=previous_incoming, + predecessor_boundary=boundaries[-2] if index else None, + origin=append.origin, + ) + break + following = chained[index + 1] if index < last else None + outgoing = _connect(declaration, following, residual_from=incoming) + boundaries.append( + _bind_stage(declaration, norm, incoming, outgoing, **options) + ) + previous_incoming, incoming = incoming, outgoing + except Exception as error: + _note_origin(error, append.origin) + raise + if branch: + for producer, consumer in zip(boundaries, boundaries[1:]): + _carry_capture(producer, consumer) + else: + # The pair that ends on the pending stage is carried when it binds. + for producer, consumer in zip(boundaries[:-2], boundaries[1:-1]): + _carry_capture(producer, consumer) + chain.previous = _detached(chained[-1]) return tuple(boundaries) diff --git a/python/sglang/srt/layers/moe/fused_moe_triton/layer.py b/python/sglang/srt/layers/moe/fused_moe_triton/layer.py index 312ce58095..da18b9ac18 100644 --- a/python/sglang/srt/layers/moe/fused_moe_triton/layer.py +++ b/python/sglang/srt/layers/moe/fused_moe_triton/layer.py @@ -87,6 +87,7 @@ from sglang.srt.utils import ( is_npu, round_up, ) +from sglang.srt.utils.common import is_building_neighbour_layer from sglang.srt.utils.custom_op import register_custom_op _is_hip = is_hip() @@ -163,6 +164,8 @@ def create_moe_dispatcher( moe_runner_config: MoeRunnerConfig, quant_method: FusedMoEMethodBase, ) -> BaseDispatcher: + if is_building_neighbour_layer(): + return StandardDispatcher(moe_runner_config) a2a_backend = get_moe_a2a_backend() if a2a_backend.is_none() and is_npu(): return AscendTPDispatcher(moe_runner_config) diff --git a/python/sglang/srt/layers/moe/kt_ep_wrapper.py b/python/sglang/srt/layers/moe/kt_ep_wrapper.py index 36c15cf139..563546840c 100644 --- a/python/sglang/srt/layers/moe/kt_ep_wrapper.py +++ b/python/sglang/srt/layers/moe/kt_ep_wrapper.py @@ -20,6 +20,7 @@ from sglang.srt.runtime_context import ( get_schedule, ) from sglang.srt.utils import get_compiler_backend +from sglang.srt.utils.common import is_building_neighbour_layer if TYPE_CHECKING: from sglang.srt.layers.moe import MoeRunnerConfig @@ -215,7 +216,7 @@ class KTEPWrapperMethod(FusedMoEMethodBase): # 2. Initialize KT wrapper for CPU experts # CPU experts: num_gpu_experts to num_experts-1 - if self.tp_rank == 0: + if self.tp_rank == 0 and not is_building_neighbour_layer(): self.wrapper = KTMoEWrapper( layer_idx=self.kt_config.layer_idx, num_experts=num_experts, diff --git a/python/sglang/srt/models/apertus.py b/python/sglang/srt/models/apertus.py index be1a6b73e7..0414fb5adf 100644 --- a/python/sglang/srt/models/apertus.py +++ b/python/sglang/srt/models/apertus.py @@ -29,9 +29,9 @@ from transformers import ApertusConfig from sglang.srt.layers.activation import XIELU from sglang.srt.layers.aux_hidden_states import AuxHiddenStateList from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -265,14 +265,12 @@ class ApertusDecoderLayer(nn.Module): self.feedforward_layernorm = RMSNorm( config.hidden_size, eps=config.rms_norm_eps ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.attention_layernorm), ( declare_ffn(sparse=False, next_layer_sparse=False), self.feedforward_layernorm, ), - previous=declare_ffn() if layer_id != 0 else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( diff --git a/python/sglang/srt/models/arcee.py b/python/sglang/srt/models/arcee.py index f166732f02..9ee234e013 100644 --- a/python/sglang/srt/models/arcee.py +++ b/python/sglang/srt/models/arcee.py @@ -23,9 +23,9 @@ from transformers import LlamaConfig from sglang.srt.layers.activation import get_act_fn from sglang.srt.layers.aux_hidden_states import AuxHiddenStateList from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -241,14 +241,12 @@ class ArceeDecoderLayer(nn.Module): self.post_attention_layernorm = RMSNorm( config.hidden_size, eps=config.rms_norm_eps ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn(sparse=False, next_layer_sparse=False), self.post_attention_layernorm, ), - previous=declare_ffn() if layer_id != 0 else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( diff --git a/python/sglang/srt/models/bailing_moe.py b/python/sglang/srt/models/bailing_moe.py index 016f35ff51..ceda0c2a3a 100644 --- a/python/sglang/srt/models/bailing_moe.py +++ b/python/sglang/srt/models/bailing_moe.py @@ -39,10 +39,10 @@ from sglang.srt.layers.dp_attention import ( is_dp_attention_enabled, ) from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, is_dense_ffn_fully_dp, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -664,9 +664,6 @@ class BailingMoEBlock(nn.Module): self.is_layer_sparse = self._is_layer_sparse( config, layer_id=layer_id, is_nextn=is_nextn ) - is_previous_layer_sparse = self._is_layer_sparse( - config, layer_id=layer_id - 1, is_nextn=False - ) is_next_layer_sparse = self._is_layer_sparse( config, layer_id=layer_id + 1, is_nextn=False ) @@ -696,7 +693,7 @@ class BailingMoEBlock(nn.Module): self.post_attention_layernorm = RMSNorm(hidden_size, eps=config.rms_norm_eps) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn( @@ -705,12 +702,6 @@ class BailingMoEBlock(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=is_previous_layer_sparse, next_layer_sparse=self.is_layer_sparse - ) - if layer_id != 0 - else None, - terminal=layer_id == (1 if is_nextn else config.num_hidden_layers) - 1, ) def _is_layer_sparse( diff --git a/python/sglang/srt/models/bailing_moe_linear.py b/python/sglang/srt/models/bailing_moe_linear.py index 6cbe96cbdd..ef27368c24 100644 --- a/python/sglang/srt/models/bailing_moe_linear.py +++ b/python/sglang/srt/models/bailing_moe_linear.py @@ -19,9 +19,9 @@ from sglang.srt.layers.dp_attention import ( is_dp_attention_enabled, ) from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -783,7 +783,6 @@ class BailingMoELinearDecoderLayer(nn.Module): self.expert_num = config.num_experts self.hidden_size = config.hidden_size is_moe_layer = self._is_layer_sparse(config, self.layer_id, is_nextn=is_nextn) - is_previous_moe_layer = self._is_layer_sparse(config, self.layer_id - 1) is_next_layer_moe_layer = self._is_layer_sparse(config, self.layer_id + 1) if self.expert_num == 1: self.mlp = BailingMLP( @@ -821,7 +820,7 @@ class BailingMoELinearDecoderLayer(nn.Module): if config.attention_type == 1 and self.use_mla else None ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( ( declare_attn(), self.input_layernorm, @@ -834,12 +833,6 @@ class BailingMoELinearDecoderLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=is_previous_moe_layer, next_layer_sparse=is_moe_layer - ) - if layer_id != 0 - else None, - terminal=layer_id == (1 if is_nextn else config.num_hidden_layers) - 1, ) def _is_layer_sparse( diff --git a/python/sglang/srt/models/bailing_moe_nextn.py b/python/sglang/srt/models/bailing_moe_nextn.py index fa9d35b9ea..d85bf9d0cd 100644 --- a/python/sglang/srt/models/bailing_moe_nextn.py +++ b/python/sglang/srt/models/bailing_moe_nextn.py @@ -27,6 +27,7 @@ from torch import nn from transformers import PretrainedConfig from sglang.srt.layers.dp_attention import is_dp_attention_enabled +from sglang.srt.layers.layer_boundary import layer_stack from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm from sglang.srt.layers.linear import ReplicatedLinear @@ -109,27 +110,30 @@ class BailingMoEModelNextN(nn.Module): self.is_hybrid = ( hasattr(config, "model_type") and config.model_type == "bailing_hybrid" ) - if self.is_hybrid: - config.attention_type = 1 - decoder_layer_cls = BailingMoELinearDecoderLayer - decoder_kwargs = { - "quant_config": quant_config, - "layer_id": 0, - "is_nextn": True, - "prefix": add_prefix(f"layers.{config.num_hidden_layers}", prefix), - } - if _is_bailing_moe_v3_config(config): - decoder_layer_cls = BailingMoeV3DecoderLayer - decoder_kwargs["num_fused_shared_experts"] = num_fused_shared_experts - self.decoder = decoder_layer_cls(config, **decoder_kwargs) - else: - self.decoder = BailingMoEBlock( - config, - 0, - quant_config=quant_config, - prefix=add_prefix("decoder", prefix), - is_nextn=True, - ) + with layer_stack(): + if self.is_hybrid: + config.attention_type = 1 + decoder_layer_cls = BailingMoELinearDecoderLayer + decoder_kwargs = { + "quant_config": quant_config, + "layer_id": 0, + "is_nextn": True, + "prefix": add_prefix(f"layers.{config.num_hidden_layers}", prefix), + } + if _is_bailing_moe_v3_config(config): + decoder_layer_cls = BailingMoeV3DecoderLayer + decoder_kwargs["num_fused_shared_experts"] = ( + num_fused_shared_experts + ) + self.decoder = decoder_layer_cls(config, **decoder_kwargs) + else: + self.decoder = BailingMoEBlock( + config, + 0, + quant_config=quant_config, + prefix=add_prefix("decoder", prefix), + is_nextn=True, + ) self.shared_head = nn.Module() self.final_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) diff --git a/python/sglang/srt/models/bailing_moe_v3.py b/python/sglang/srt/models/bailing_moe_v3.py index d5b3ad1f87..d7506e5e0a 100644 --- a/python/sglang/srt/models/bailing_moe_v3.py +++ b/python/sglang/srt/models/bailing_moe_v3.py @@ -27,10 +27,10 @@ from sglang.srt.layers.activation import SiluAndMul from sglang.srt.layers.aux_hidden_states import AuxHiddenStateList from sglang.srt.layers.dp_attention import is_dp_attention_enabled from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, is_dense_ffn_fully_dp, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -1085,9 +1085,6 @@ class BailingMoELinearDecoderLayer(nn.Module): and (self.layer_id >= config.first_k_dense_replace) ) self.is_layer_sparse = is_moe_layer - is_previous_moe_layer = not (self.expert_num == 1) and ( - self.layer_id - 1 >= config.first_k_dense_replace - ) is_next_layer_sparse = not (self.expert_num == 1) and ( self.layer_id + 1 >= config.first_k_dense_replace ) @@ -1132,7 +1129,7 @@ class BailingMoELinearDecoderLayer(nn.Module): self.input_layernorm = RMSNorm(self.hidden_size, eps=rms_norm_eps) self.post_attention_layernorm = RMSNorm(self.hidden_size, eps=rms_norm_eps) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( ( declare_attn(), self.input_layernorm, @@ -1149,12 +1146,6 @@ class BailingMoELinearDecoderLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=is_previous_moe_layer, next_layer_sparse=is_moe_layer - ) - if layer_id != 0 - else None, - terminal=layer_id == (1 if is_nextn else config.num_hidden_layers) - 1, ) @torch.inference_mode() diff --git a/python/sglang/srt/models/deepseek_nextn.py b/python/sglang/srt/models/deepseek_nextn.py index 6a71ee2c6f..46b7f708e4 100644 --- a/python/sglang/srt/models/deepseek_nextn.py +++ b/python/sglang/srt/models/deepseek_nextn.py @@ -28,6 +28,7 @@ from sglang.kernels.ops.layernorm.fused_eh_norm import fused_eh_norm from sglang.srt.environ import envs from sglang.srt.eplb.expert_distribution import get_global_expert_distribution_recorder from sglang.srt.layers.attention.index_topk_share import IndexTopKShareState +from sglang.srt.layers.layer_boundary import layer_stack from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm from sglang.srt.layers.linear import ReplicatedLinear @@ -121,16 +122,17 @@ class DeepseekModelNextN(nn.Module): layer_name = "layers." + str(config.num_hidden_layers) self.quant_config = quant_config - self.decoder = DeepseekV2DecoderLayer( - config, - 0, - quant_config=quant_config, - moe_quant_config_override=moe_quant_config_override, - is_nextn=True, - prefix=add_prefix(layer_name, prefix), - alt_stream=self.alt_stream, - skip_rope=config.qk_rope_head_dim == 0, - ) + with layer_stack(): + self.decoder = DeepseekV2DecoderLayer( + config, + 0, + quant_config=quant_config, + moe_quant_config_override=moe_quant_config_override, + is_nextn=True, + prefix=add_prefix(layer_name, prefix), + alt_stream=self.alt_stream, + skip_rope=config.qk_rope_head_dim == 0, + ) self.shared_head = nn.Module() self.shared_head.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) diff --git a/python/sglang/srt/models/deepseek_v2.py b/python/sglang/srt/models/deepseek_v2.py index 557beb85e6..737037f2b8 100644 --- a/python/sglang/srt/models/deepseek_v2.py +++ b/python/sglang/srt/models/deepseek_v2.py @@ -76,11 +76,11 @@ from sglang.srt.layers.dcp.planner import ( ) from sglang.srt.layers.flashinfer_comm_fusion import uses_cutedsl_ar_fusion from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, get_attn_tp_context, is_dense_ffn_fully_dp, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import access as residual_access from sglang.srt.layers.layer_boundary.residual import batch as residual_batch @@ -2665,14 +2665,8 @@ class DeepseekV2DecoderLayer(nn.Module): ) self.is_layer_sparse = self._is_layer_sparse(layer_id, is_nextn=is_nextn) - is_previous_layer_sparse = self._is_layer_sparse(layer_id - 1, is_nextn=False) is_next_layer_sparse = self._is_layer_sparse(layer_id + 1, is_nextn=False) - self._stage_enters_stack = (layer_id) == 0 - self._stage_terminal = (layer_id) == ( - 1 if is_nextn else config.num_hidden_layers - ) - 1 - self._stage_previous_sparse = is_previous_layer_sparse self._stage_next_sparse = is_next_layer_sparse if self.is_layer_sparse: @@ -2740,7 +2734,7 @@ class DeepseekV2DecoderLayer(nn.Module): from sglang.srt.layers.layer_boundary.fusions.cutedsl import CuteDSLFusion fusions = CuteDSLFusion() - attn_boundary, ffn_boundary = make_stages( + attn_boundary, ffn_boundary = append_stages( ( declare_attn(output_transform=attn_output), input_layernorm, @@ -2755,13 +2749,6 @@ class DeepseekV2DecoderLayer(nn.Module): post_attention_layernorm, {"fusions": fusions}, ), - previous=declare_ffn( - sparse=self._stage_previous_sparse, - next_layer_sparse=self.is_layer_sparse, - ) - if not self._stage_enters_stack - else None, - terminal=self._stage_terminal, ) return attn_boundary, ffn_boundary diff --git a/python/sglang/srt/models/dots3_common/modeling.py b/python/sglang/srt/models/dots3_common/modeling.py index 4c3540b81b..2488ce139f 100644 --- a/python/sglang/srt/models/dots3_common/modeling.py +++ b/python/sglang/srt/models/dots3_common/modeling.py @@ -56,10 +56,10 @@ from sglang.srt.layers.activation import SiluAndMul from sglang.srt.layers.attention.dsa.dsa_indexer import Indexer from sglang.srt.layers.dp_attention import is_dp_attention_enabled from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, is_dense_ffn_fully_dp, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -1384,7 +1384,6 @@ class Dots3DecoderLayer(nn.Module): alt_stream=alt_stream, ) self.is_layer_sparse = self._is_layer_sparse(layer_id, is_nextn=is_nextn) - is_previous_layer_sparse = self._is_layer_sparse(layer_id - 1, is_nextn=False) is_next_layer_sparse = self._is_layer_sparse(layer_id + 1, is_nextn=False) if self.is_layer_sparse: @@ -1417,7 +1416,7 @@ class Dots3DecoderLayer(nn.Module): config.hidden_size, eps=config.rms_norm_eps ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn( @@ -1426,12 +1425,6 @@ class Dots3DecoderLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=is_previous_layer_sparse, next_layer_sparse=self.is_layer_sparse - ) - if layer_id != 0 - else None, - terminal=layer_id == (1 if is_nextn else config.num_hidden_layers) - 1, ) def _is_layer_sparse(self, layer_id: int, is_nextn: bool) -> bool: diff --git a/python/sglang/srt/models/dots3_common/nextn.py b/python/sglang/srt/models/dots3_common/nextn.py index 073f9fc758..ea0281e751 100644 --- a/python/sglang/srt/models/dots3_common/nextn.py +++ b/python/sglang/srt/models/dots3_common/nextn.py @@ -9,6 +9,7 @@ from transformers import PretrainedConfig from sglang.srt.eplb.expert_distribution import get_global_expert_distribution_recorder from sglang.srt.layers.dp_attention import is_dp_attention_enabled +from sglang.srt.layers.layer_boundary import layer_stack from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm from sglang.srt.layers.linear import ReplicatedLinear @@ -48,13 +49,14 @@ class Dots3MTPHead(nn.Module): quant_config=quant_config, prefix=add_prefix("eh_proj", prefix), ) - self.decoder = Dots3DecoderLayer( - config, - layer_id=0, - quant_config=quant_config, - is_nextn=True, - prefix=add_prefix("decoder", prefix), - ) + with layer_stack(): + self.decoder = Dots3DecoderLayer( + config, + layer_id=0, + quant_config=quant_config, + is_nextn=True, + prefix=add_prefix("decoder", prefix), + ) self.shared_head = nn.Module() self.shared_head.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) diff --git a/python/sglang/srt/models/ernie45_moe_vl.py b/python/sglang/srt/models/ernie45_moe_vl.py index 17763aa703..562f0d357a 100644 --- a/python/sglang/srt/models/ernie45_moe_vl.py +++ b/python/sglang/srt/models/ernie45_moe_vl.py @@ -23,9 +23,9 @@ from torch import nn from transformers import PretrainedConfig from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -438,19 +438,12 @@ class Ernie4_5_VLMoeDecoderLayer(nn.Module): next_sparse = layer_id + 1 < config.num_hidden_layers and _is_moe_layer( config, layer_id + 1 ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn(sparse=sparse, next_layer_sparse=next_sparse), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=_is_moe_layer(config, layer_id - 1), - next_layer_sparse=sparse, - ) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( diff --git a/python/sglang/srt/models/exaone4.py b/python/sglang/srt/models/exaone4.py index 230c01d750..c8c583b212 100644 --- a/python/sglang/srt/models/exaone4.py +++ b/python/sglang/srt/models/exaone4.py @@ -7,9 +7,9 @@ from transformers import Exaone4Config from sglang.srt.layers.activation import SiluAndMul from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layer_boundary.residual.post_norm import ( @@ -275,7 +275,7 @@ class Exaone4DecoderLayer(nn.Module): # Post-LN: each stage reads the residual as it is, and its output is # normalized before it is added. The layer writes the FFN's itself. ffn_update = PostNormAdd(self.post_feedforward_layernorm, applied_at_exit=True) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( ( declare_attn( read=PLAIN_READOUT, @@ -292,8 +292,6 @@ class Exaone4DecoderLayer(nn.Module): ), None, ), - previous=declare_ffn(update=ffn_update) if layer_id != 0 else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( diff --git a/python/sglang/srt/models/exaone_moe.py b/python/sglang/srt/models/exaone_moe.py index 912904d484..6b618d3e4c 100755 --- a/python/sglang/srt/models/exaone_moe.py +++ b/python/sglang/srt/models/exaone_moe.py @@ -31,9 +31,9 @@ from sglang.srt.layers.activation import SiluAndMul from sglang.srt.layers.aux_hidden_states import AuxHiddenStateList from sglang.srt.layers.dp_attention import is_dp_attention_enabled from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -491,7 +491,7 @@ class ExaoneMoEDecoderLayer(nn.Module): ) is_moe_layer = config.is_moe_layer num_layers = config.num_hidden_layers - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn( @@ -501,13 +501,6 @@ class ExaoneMoEDecoderLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=is_moe_layer[layer_id - 1], - next_layer_sparse=is_moe_layer[layer_id], - ) - if layer_id != 0 - else None, - terminal=layer_id == num_layers - 1, ) def forward( diff --git a/python/sglang/srt/models/falcon_h1.py b/python/sglang/srt/models/falcon_h1.py index 03c5b67037..8d2d70c096 100644 --- a/python/sglang/srt/models/falcon_h1.py +++ b/python/sglang/srt/models/falcon_h1.py @@ -15,9 +15,9 @@ from sglang.srt.layers.dp_attention import ( is_dp_attention_enabled, ) from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.output import OutputTransform from sglang.srt.layers.layer_boundary.residual import batch as residual_batch @@ -203,7 +203,6 @@ class FalconH1HybridAttentionDecoderLayer(nn.Module): # FalconH1 all layers are dense and have no nextn now self.is_layer_sparse = False - is_previous_layer_sparse = False is_next_layer_sparse = False self.feed_forward = FalconH1MLP( @@ -223,7 +222,7 @@ class FalconH1HybridAttentionDecoderLayer(nn.Module): self.q_norm = RMSNorm(self.head_dim, eps=config.rms_norm_eps) self.k_norm = RMSNorm(self.head_dim, eps=config.rms_norm_eps) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn( @@ -235,12 +234,6 @@ class FalconH1HybridAttentionDecoderLayer(nn.Module): ), self.pre_ff_layernorm, ), - previous=declare_ffn( - sparse=is_previous_layer_sparse, next_layer_sparse=self.is_layer_sparse - ) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) self.alt_stream = alt_stream diff --git a/python/sglang/srt/models/gemma4_causal.py b/python/sglang/srt/models/gemma4_causal.py index 38c46f43ad..2a2b7afe08 100644 --- a/python/sglang/srt/models/gemma4_causal.py +++ b/python/sglang/srt/models/gemma4_causal.py @@ -36,9 +36,9 @@ from sglang.kernels.ops.moe.gemma4_routing import ( from sglang.srt.layers.aux_hidden_states import AuxHiddenStateList from sglang.srt.layers.layer_boundary import ( SumGroup, + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.output import OutputTransform from sglang.srt.layers.layer_boundary.residual import batch as residual_batch @@ -666,7 +666,7 @@ class Gemma4DecoderLayer(nn.Module): self.register_buffer("layer_scalar", torch.ones(1), persistent=True) self.has_ple = self.hidden_size_per_layer_input > 0 self.prefix = prefix - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( ( declare_attn( output_transform=OutputTransform(self.post_attention_layernorm) @@ -681,14 +681,6 @@ class Gemma4DecoderLayer(nn.Module): ), self.pre_feedforward_layernorm, ), - previous=declare_ffn( - sparse=self.enable_moe_block, - next_layer_sparse=self.enable_moe_block, - update=REPLACE_AT_EXIT, - ) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( diff --git a/python/sglang/srt/models/gigachat35_mtp.py b/python/sglang/srt/models/gigachat35_mtp.py index 3942183396..25c6e0da75 100644 --- a/python/sglang/srt/models/gigachat35_mtp.py +++ b/python/sglang/srt/models/gigachat35_mtp.py @@ -18,6 +18,7 @@ from typing import Iterable, Optional import torch from torch import nn +from sglang.srt.layers.layer_boundary import layer_stack from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.logits_processor import LogitsProcessor from sglang.srt.layers.quantization.base_config import QuantizationConfig @@ -64,14 +65,15 @@ class GigaChat35ModelNextN(nn.Module): self.alt_stream = torch.cuda.Stream() if torch.cuda.is_available() else None - self.decoder = GigaChat35DecoderLayer( - config=config, - layer_id=0, - quant_config=quant_config, - prefix=add_prefix("decoder", prefix), - alt_stream=self.alt_stream, - is_nextn=True, - ) + with layer_stack(): + self.decoder = GigaChat35DecoderLayer( + config=config, + layer_id=0, + quant_config=quant_config, + prefix=add_prefix("decoder", prefix), + alt_stream=self.alt_stream, + is_nextn=True, + ) self.shared_head = nn.Module() self.shared_head.norm = build_norm(config, config.hidden_size) diff --git a/python/sglang/srt/models/glm4.py b/python/sglang/srt/models/glm4.py index e705e46e67..15b4d9775e 100644 --- a/python/sglang/srt/models/glm4.py +++ b/python/sglang/srt/models/glm4.py @@ -27,9 +27,9 @@ from sglang.srt.layers.activation import SiluAndMul from sglang.srt.layers.aux_hidden_states import AuxHiddenStateList from sglang.srt.layers.dp_attention import is_dp_attention_enabled from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.output import OutputTransform from sglang.srt.layers.layer_boundary.residual import batch as residual_batch @@ -263,7 +263,7 @@ class Glm4DecoderLayer(nn.Module): config.hidden_size, eps=config.rms_norm_eps ) self.post_mlp_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( ( declare_attn( output_transform=OutputTransform(self.post_self_attn_layernorm) @@ -278,8 +278,6 @@ class Glm4DecoderLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn() if layer_id != 0 else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( diff --git a/python/sglang/srt/models/glm4_moe.py b/python/sglang/srt/models/glm4_moe.py index f39c23c4bc..f8ca48d745 100644 --- a/python/sglang/srt/models/glm4_moe.py +++ b/python/sglang/srt/models/glm4_moe.py @@ -44,10 +44,10 @@ from sglang.srt.layers.dp_attention import ( is_dp_attention_enabled, ) from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, is_dense_ffn_fully_dp, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -741,7 +741,6 @@ class Glm4MoeDecoderLayer(nn.Module): ) self.is_layer_sparse = self._is_layer_sparse(layer_id, is_nextn=is_nextn) - is_previous_layer_sparse = self._is_layer_sparse(layer_id - 1, is_nextn=False) is_next_layer_sparse = self._is_layer_sparse(layer_id + 1, is_nextn=False) if self.is_layer_sparse: @@ -773,7 +772,7 @@ class Glm4MoeDecoderLayer(nn.Module): config.hidden_size, eps=config.rms_norm_eps ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn( @@ -782,12 +781,6 @@ class Glm4MoeDecoderLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=is_previous_layer_sparse, next_layer_sparse=self.is_layer_sparse - ) - if layer_id != 0 - else None, - terminal=layer_id == (1 if is_nextn else config.num_hidden_layers) - 1, ) # Detect if QKV uses aiter FP8 per-token quant so we can fuse diff --git a/python/sglang/srt/models/glm4_moe_lite.py b/python/sglang/srt/models/glm4_moe_lite.py index 27d7ef3e4d..30dca657f7 100644 --- a/python/sglang/srt/models/glm4_moe_lite.py +++ b/python/sglang/srt/models/glm4_moe_lite.py @@ -38,11 +38,11 @@ from sglang.srt.layers.dp_attention import ( is_dp_attention_enabled, ) from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, get_attn_tp_context, is_dense_ffn_fully_dp, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -448,7 +448,6 @@ class Glm4MoeLiteDecoderLayer(nn.Module): ) self.is_layer_sparse = self._is_layer_sparse(layer_id, is_nextn=is_nextn) - is_previous_layer_sparse = self._is_layer_sparse(layer_id - 1, is_nextn=False) is_next_layer_sparse = self._is_layer_sparse(layer_id + 1, is_nextn=False) if self.is_layer_sparse: @@ -483,7 +482,7 @@ class Glm4MoeLiteDecoderLayer(nn.Module): self._gfx95_quant_format = self._detect_gfx95_quant_format() - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( ( declare_attn(), self.input_layernorm, @@ -496,12 +495,6 @@ class Glm4MoeLiteDecoderLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=is_previous_layer_sparse, next_layer_sparse=self.is_layer_sparse - ) - if layer_id != 0 - else None, - terminal=layer_id == (1 if is_nextn else config.num_hidden_layers) - 1, ) def _detect_gfx95_quant_format(self) -> str: diff --git a/python/sglang/srt/models/glm4_moe_lite_nextn.py b/python/sglang/srt/models/glm4_moe_lite_nextn.py index e0baf64600..e63c6e7fc8 100644 --- a/python/sglang/srt/models/glm4_moe_lite_nextn.py +++ b/python/sglang/srt/models/glm4_moe_lite_nextn.py @@ -23,6 +23,7 @@ from transformers import PretrainedConfig from sglang.srt.eplb.expert_distribution import get_global_expert_distribution_recorder from sglang.srt.layers.dp_attention import is_dp_attention_enabled +from sglang.srt.layers.layer_boundary import layer_stack from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm from sglang.srt.layers.logits_processor import LogitsProcessor @@ -71,13 +72,14 @@ class Glm4MoeLiteModelNextN(nn.Module): self.eh_proj = nn.Linear(2 * config.hidden_size, config.hidden_size, bias=False) - self.decoder = Glm4MoeLiteDecoderLayer( - config, - 0, - quant_config=quant_config, - is_nextn=True, - prefix=add_prefix("decoder", prefix), - ) + with layer_stack(): + self.decoder = Glm4MoeLiteDecoderLayer( + config, + 0, + quant_config=quant_config, + is_nextn=True, + prefix=add_prefix("decoder", prefix), + ) self.shared_head = nn.Module() self.shared_head.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) diff --git a/python/sglang/srt/models/glm4_moe_nextn.py b/python/sglang/srt/models/glm4_moe_nextn.py index 83d438e067..d35d2a772c 100644 --- a/python/sglang/srt/models/glm4_moe_nextn.py +++ b/python/sglang/srt/models/glm4_moe_nextn.py @@ -23,6 +23,7 @@ from transformers import PretrainedConfig from sglang.srt.eplb.expert_distribution import get_global_expert_distribution_recorder from sglang.srt.layers.dp_attention import is_dp_attention_enabled +from sglang.srt.layers.layer_boundary import layer_stack from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm from sglang.srt.layers.logits_processor import LogitsProcessor @@ -67,13 +68,14 @@ class Glm4MoeModelNextN(nn.Module): self.eh_proj = nn.Linear(2 * config.hidden_size, config.hidden_size, bias=False) - self.decoder = Glm4MoeDecoderLayer( - config, - 0, - quant_config=quant_config, - is_nextn=True, - prefix=add_prefix("decoder", prefix), - ) + with layer_stack(): + self.decoder = Glm4MoeDecoderLayer( + config, + 0, + quant_config=quant_config, + is_nextn=True, + prefix=add_prefix("decoder", prefix), + ) self.shared_head = nn.Module() self.shared_head.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) diff --git a/python/sglang/srt/models/glm5_next.py b/python/sglang/srt/models/glm5_next.py index ed58952849..f9f919100b 100644 --- a/python/sglang/srt/models/glm5_next.py +++ b/python/sglang/srt/models/glm5_next.py @@ -29,11 +29,11 @@ from sglang.srt.layers.conv import Conv2dLayer from sglang.srt.layers.layer_boundary import ( PLAIN_RESIDUAL_OPS, MHCState, + append_stages, declare_attn, declare_ffn, get_attn_tp_context, is_dense_ffn_fully_dp, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import access as residual_access from sglang.srt.layers.layer_boundary.residual import batch as residual_batch @@ -723,7 +723,6 @@ class Glm5NextDecoderLayer(nn.Module): ) self.is_layer_sparse = self._is_layer_sparse(layer_id, is_nextn=is_nextn) - is_previous_layer_sparse = self._is_layer_sparse(layer_id - 1, is_nextn=False) is_next_layer_sparse = self._is_layer_sparse(layer_id + 1, is_nextn=False) if self.is_layer_sparse: @@ -793,7 +792,7 @@ class Glm5NextDecoderLayer(nn.Module): ), is_last_layer=terminal, ).residual_ops() - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( ( declare_attn(read=residual.attn_readout, update=residual.attn_update), self.input_layernorm, @@ -812,14 +811,6 @@ class Glm5NextDecoderLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=is_previous_layer_sparse, - next_layer_sparse=self.is_layer_sparse, - update=residual.ffn_update, - ) - if layer_id != 0 - else None, - terminal=terminal, ) def _hc_pre( diff --git a/python/sglang/srt/models/glm_image_vl.py b/python/sglang/srt/models/glm_image_vl.py index 6b12e961b6..051144e761 100644 --- a/python/sglang/srt/models/glm_image_vl.py +++ b/python/sglang/srt/models/glm_image_vl.py @@ -35,9 +35,10 @@ from sglang.srt.layers.attention.vision import ( ) from sglang.srt.layers.dp_attention import is_dp_attention_enabled from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, + layer_stack, ) from sglang.srt.layers.layer_boundary.output import OutputTransform from sglang.srt.layers.layer_boundary.residual import batch as residual_batch @@ -907,7 +908,7 @@ class GlmImageTextDecoderLayer(nn.Module): config.hidden_size, eps=config.rms_norm_eps ) self.post_mlp_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( ( declare_attn( read=UNFUSED_NORM_READOUT, @@ -924,8 +925,6 @@ class GlmImageTextDecoderLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn() if layer_id != 0 else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( @@ -971,17 +970,18 @@ class GlmImageTextModel(nn.Module): prefix=add_prefix("embed_tokens", prefix), ) - self.layers = nn.ModuleList( - [ - GlmImageTextDecoderLayer( - layer_id=i, - config=config, - quant_config=self.quant_config, - prefix=add_prefix(f"layers.{i}", getattr(config, "prefix", "")), - ) - for i in range(config.num_hidden_layers) - ] - ) + with layer_stack(): + self.layers = nn.ModuleList( + [ + GlmImageTextDecoderLayer( + layer_id=i, + config=config, + quant_config=self.quant_config, + prefix=add_prefix(f"layers.{i}", getattr(config, "prefix", "")), + ) + for i in range(config.num_hidden_layers) + ] + ) self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) def forward( diff --git a/python/sglang/srt/models/glm_ocr_nextn.py b/python/sglang/srt/models/glm_ocr_nextn.py index a44ccb51ed..78f16e77d7 100644 --- a/python/sglang/srt/models/glm_ocr_nextn.py +++ b/python/sglang/srt/models/glm_ocr_nextn.py @@ -23,6 +23,7 @@ from transformers import PretrainedConfig from sglang.srt.eplb.expert_distribution import get_global_expert_distribution_recorder from sglang.srt.layers.dp_attention import is_dp_attention_enabled +from sglang.srt.layers.layer_boundary import layer_stack from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm from sglang.srt.layers.logits_processor import LogitsProcessor @@ -70,12 +71,13 @@ class GlmOcrModelNextN(nn.Module): self.eh_proj = nn.Linear(2 * config.hidden_size, config.hidden_size, bias=False) - self.decoder = Glm4DecoderLayer( - config, - 0, - quant_config=quant_config, - prefix=add_prefix("decoder", prefix), - ) + with layer_stack(): + self.decoder = Glm4DecoderLayer( + config, + 0, + quant_config=quant_config, + prefix=add_prefix("decoder", prefix), + ) self.shared_head = nn.Module() self.shared_head.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) diff --git a/python/sglang/srt/models/gpt_oss.py b/python/sglang/srt/models/gpt_oss.py index 62d37b61ed..335c537fbb 100644 --- a/python/sglang/srt/models/gpt_oss.py +++ b/python/sglang/srt/models/gpt_oss.py @@ -34,9 +34,9 @@ from sglang.srt.layers.dp_attention import ( is_dp_attention_enabled, ) from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -560,7 +560,6 @@ class GptOssDecoderLayer(nn.Module): # GptOss all layers are sparse self.is_layer_sparse = True - is_previous_layer_sparse = True is_next_layer_sparse = True if self.is_layer_sparse: @@ -591,7 +590,7 @@ class GptOssDecoderLayer(nn.Module): x_pad_to_multiple=post_attn_pad_multiple, ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn( @@ -600,12 +599,6 @@ class GptOssDecoderLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=is_previous_layer_sparse, next_layer_sparse=self.is_layer_sparse - ) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( diff --git a/python/sglang/srt/models/granitemoehybrid.py b/python/sglang/srt/models/granitemoehybrid.py index 21db5163e7..42f2706835 100644 --- a/python/sglang/srt/models/granitemoehybrid.py +++ b/python/sglang/srt/models/granitemoehybrid.py @@ -11,9 +11,9 @@ from sglang.srt.layers.attention.hybrid_linear_attn_backend import ( from sglang.srt.layers.attention.mamba.mamba import MambaMixer2 from sglang.srt.layers.aux_hidden_states import AuxHiddenStateList from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.output import OutputTransform from sglang.srt.layers.layer_boundary.residual import batch as residual_batch @@ -40,13 +40,13 @@ from sglang.srt.utils import make_pp_layers from .granitemoe import GraniteMoeMoE, GraniteMoeSharedMLP -def _make_stages(layer, config, layer_idx: int): +def _make_stages(layer): """Both kinds of layer scale their mixer and FFN outputs by residual_multiplier, add them in the activation dtype and normalize the sum as a separate step.""" scale = OutputTransform(layer._scale_output) sparse = layer.block_sparse_moe is not None - return make_stages( + return append_stages( ( declare_attn(read=UNFUSED_NORM_READOUT, output_transform=scale), layer.input_layernorm, @@ -60,12 +60,6 @@ def _make_stages(layer, config, layer_idx: int): ), layer.post_attention_layernorm, ), - previous=( - declare_ffn(sparse=sparse, next_layer_sparse=sparse) - if layer_idx != 0 - else None - ), - terminal=layer_idx == config.num_hidden_layers - 1, ) @@ -124,7 +118,7 @@ class GraniteMoeHybridMambaDecoderLayer(nn.Module): self.post_attention_layernorm = RMSNorm( config.hidden_size, eps=config.rms_norm_eps ) - self.attn_boundary, self.ffn_boundary = _make_stages(self, config, layer_idx) + self.attn_boundary, self.ffn_boundary = _make_stages(self) def _scale_output(self, hidden_states: torch.Tensor) -> torch.Tensor: return hidden_states * self.residual_multiplier @@ -315,7 +309,7 @@ class GraniteMoeHybridAttentionDecoderLayer(nn.Module): self.post_attention_layernorm = RMSNorm( config.hidden_size, eps=config.rms_norm_eps ) - self.attn_boundary, self.ffn_boundary = _make_stages(self, config, layer_idx) + self.attn_boundary, self.ffn_boundary = _make_stages(self) def _scale_output(self, hidden_states: torch.Tensor) -> torch.Tensor: return hidden_states * self.residual_multiplier diff --git a/python/sglang/srt/models/hunyuan_v4.py b/python/sglang/srt/models/hunyuan_v4.py index 3ff4f3e840..2fff1e3ff6 100644 --- a/python/sglang/srt/models/hunyuan_v4.py +++ b/python/sglang/srt/models/hunyuan_v4.py @@ -9,10 +9,11 @@ from transformers import PretrainedConfig from sglang.srt.layers.attention.index_topk_share import IndexTopKShareState from sglang.srt.layers.layer_boundary import ( IHCState, + append_stages, declare_attn, declare_ffn, get_attn_tp_context, - make_stages, + layer_stack, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -583,8 +584,8 @@ def _build_stages( layer_types = config.mlp_layer_types is_sparse = layer_types[layer_id] != "dense" - next_id, previous_id = layer_id + 1, layer_id - 1 - return make_stages( + next_id = layer_id + 1 + return append_stages( ( declare_attn(read=residual.attn_readout, update=residual.attn_update), input_layernorm, @@ -601,16 +602,6 @@ def _build_stages( ), post_attention_layernorm, ), - previous=( - declare_ffn( - sparse=layer_types[previous_id] != "dense", - next_layer_sparse=is_sparse, - update=residual.ffn_update, - ) - if layer_id != 0 - else None - ), - terminal=layer_id == config.num_hidden_layers - 1, ) @@ -703,18 +694,19 @@ class HYV4Model(nn.Module): **get_embedding_tp_kwargs(), ) self.alt_stream = get_stream("alt") if is_cuda() else None - self.layers = nn.ModuleList( - [ - HYV4DecoderLayer( - config, - i, - quant_config, - f"{prefix}.layers.{i}", - self.alt_stream, - ) - for i in range(config.num_hidden_layers) - ] - ) + with layer_stack(): + self.layers = nn.ModuleList( + [ + HYV4DecoderLayer( + config, + i, + quant_config, + f"{prefix}.layers.{i}", + self.alt_stream, + ) + for i in range(config.num_hidden_layers) + ] + ) self.hc_head = HYV4HCHeadLayer(config, f"{prefix}.hc_head") self.norm = RMSNorm(config.hidden_size, config.rms_norm_eps) # The head mixes the streams down and fuses the final norm into one kernel. diff --git a/python/sglang/srt/models/hunyuan_v4_nextn.py b/python/sglang/srt/models/hunyuan_v4_nextn.py index b5a9801d27..5a9ab63b92 100644 --- a/python/sglang/srt/models/hunyuan_v4_nextn.py +++ b/python/sglang/srt/models/hunyuan_v4_nextn.py @@ -6,10 +6,11 @@ from torch import nn from sglang.srt.layers.attention.index_topk_share import IndexTopKShareState from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, get_attn_tp_context, - make_stages, + layer_stack, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -71,14 +72,13 @@ class HYV4MTPDecoderLayer(nn.Module): ) if hasattr(self.mlp, "shared_experts"): self.mlp.shared_experts.swiglu_limit = None - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( ( declare_attn(), self.input_layernorm, {"qkv_latent_func": self.self_attn.prepare_qkv_latent}, ), (declare_ffn(sparse=True), self.post_attention_layernorm), - terminal=True, ) def forward( @@ -124,12 +124,13 @@ class HYV4ModelNextN(nn.Module): self.hnorm = RMSNorm(config.hidden_size, config.rms_norm_eps) self.eh_proj = nn.Linear(2 * config.hidden_size, config.hidden_size, bias=False) self.alt_stream = get_stream("alt") if is_cuda() else None - self.decoder = HYV4MTPDecoderLayer( - config, - quant_config, - f"{prefix}.decoder", - self.alt_stream, - ) + with layer_stack(): + self.decoder = HYV4MTPDecoderLayer( + config, + quant_config, + f"{prefix}.decoder", + self.alt_stream, + ) self.shared_head = nn.Module() self.shared_head.norm = RMSNorm(config.hidden_size, config.rms_norm_eps) diff --git a/python/sglang/srt/models/interns2_mobius.py b/python/sglang/srt/models/interns2_mobius.py index 81fb63ce3b..7a44063015 100644 --- a/python/sglang/srt/models/interns2_mobius.py +++ b/python/sglang/srt/models/interns2_mobius.py @@ -16,9 +16,9 @@ from sglang.srt.layers.aux_hidden_states import ( ) from sglang.srt.layers.dp_attention import is_dp_attention_enabled from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layer_boundary.residual.add_norm import ( @@ -485,7 +485,7 @@ class InternS2MobiusLinearDecoderLayer(_InternS2MobiusDecoderMixin, nn.Module): config.hidden_size, eps=config.rms_norm_eps ) accepts_fp8_input = _linear_accepts_fp8_tuple(self.linear_attn.in_proj_qkvz) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( ( declare_attn( read=NormQuantReadout( @@ -501,10 +501,6 @@ class InternS2MobiusLinearDecoderLayer(_InternS2MobiusDecoderMixin, nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn(sparse=True, next_layer_sparse=True) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( @@ -617,7 +613,7 @@ class InternS2MobiusAttentionDecoderLayer( self.q_norm = GemmaRMSNorm(self.head_dim, eps=config.rms_norm_eps) self.k_norm = GemmaRMSNorm(self.head_dim, eps=config.rms_norm_eps) accepts_fp8_input = _linear_accepts_fp8_tuple(self.qkv_proj) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( ( declare_attn( read=NormQuantReadout( @@ -633,10 +629,6 @@ class InternS2MobiusAttentionDecoderLayer( ), self.post_attention_layernorm, ), - previous=declare_ffn(sparse=True, next_layer_sparse=True) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) self.alt_stream = alt_stream diff --git a/python/sglang/srt/models/iquest_q1.py b/python/sglang/srt/models/iquest_q1.py index ad3cc08e59..8bc8edff6f 100644 --- a/python/sglang/srt/models/iquest_q1.py +++ b/python/sglang/srt/models/iquest_q1.py @@ -11,9 +11,9 @@ from triton.language.extra import libdevice from sglang.srt.layers.activation import SiluAndMul from sglang.srt.layers.dp_attention import reject_attn_tp_shard_with_tp_reduce from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.output import OutputTransform from sglang.srt.layers.layer_boundary.residual import batch as residual_batch @@ -496,7 +496,7 @@ class IQuestQ1DecoderLayer(nn.Module): # Intentional: layer 0 adds the raw input and an FFN output norm, later layers # add the normalized input without one; the MTP draft follows layer 0. moe = _is_moe_layer(config, layer_id) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( ( declare_attn( read=UNFUSED_NORM_READOUT @@ -515,12 +515,6 @@ class IQuestQ1DecoderLayer(nn.Module): ), self.feed_forward_norm, ), - previous=declare_ffn( - sparse=_is_moe_layer(config, layer_id - 1), next_layer_sparse=moe - ) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def _attn_output(self, attn_output: torch.Tensor) -> torch.Tensor: diff --git a/python/sglang/srt/models/kimi_linear.py b/python/sglang/srt/models/kimi_linear.py index 39691bdbea..dc68a6f689 100644 --- a/python/sglang/srt/models/kimi_linear.py +++ b/python/sglang/srt/models/kimi_linear.py @@ -16,9 +16,9 @@ from sglang.srt.distributed import ( from sglang.srt.eplb.expert_distribution import get_global_expert_distribution_recorder from sglang.srt.layers.dcp.planner import prepare_decode_context_parallel_metadata from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -618,19 +618,12 @@ class KimiDecoderLayer(nn.Module): next_sparse = layer_idx + 1 < config.num_hidden_layers and _is_sparse_layer( config, layer_idx + 1 ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn(sparse=sparse, next_layer_sparse=next_sparse), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=_is_sparse_layer(config, layer_idx - 1), - next_layer_sparse=sparse, - ) - if layer_idx != 0 - else None, - terminal=layer_idx == config.num_hidden_layers - 1, ) def forward( diff --git a/python/sglang/srt/models/laguna.py b/python/sglang/srt/models/laguna.py index 5c38e95453..450e44a680 100644 --- a/python/sglang/srt/models/laguna.py +++ b/python/sglang/srt/models/laguna.py @@ -24,9 +24,9 @@ from sglang.srt.layers.dp_attention import ( is_dp_attention_enabled, ) from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -425,7 +425,6 @@ class LagunaDecoderLayer(nn.Module): mlp_types = config.mlp_layer_types self.is_layer_sparse = mlp_types[layer_id] == "sparse" - is_previous_layer_sparse = layer_id > 0 and mlp_types[layer_id - 1] == "sparse" is_next_layer_sparse = ( layer_id + 1 < config.num_hidden_layers and mlp_types[layer_id + 1] == "sparse" @@ -453,7 +452,7 @@ class LagunaDecoderLayer(nn.Module): config.hidden_size, eps=config.rms_norm_eps ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn( @@ -462,12 +461,6 @@ class LagunaDecoderLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=is_previous_layer_sparse, next_layer_sparse=self.is_layer_sparse - ) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( diff --git a/python/sglang/srt/models/llada2.py b/python/sglang/srt/models/llada2.py index 641921f034..00fdd47dce 100644 --- a/python/sglang/srt/models/llada2.py +++ b/python/sglang/srt/models/llada2.py @@ -35,10 +35,10 @@ from sglang.srt.layers.dp_attention import ( is_dp_attention_enabled, ) from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, is_dense_ffn_fully_dp, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -603,7 +603,6 @@ class LLaDA2MoeBlock(nn.Module): self.layer_id = layer_id self.is_layer_sparse = self._is_layer_sparse(config, layer_id=layer_id) - is_previous_layer_sparse = self._is_layer_sparse(config, layer_id=layer_id - 1) is_next_layer_sparse = self._is_layer_sparse(config, layer_id=layer_id + 1) if self.is_layer_sparse: @@ -631,7 +630,7 @@ class LLaDA2MoeBlock(nn.Module): self.post_attention_layernorm = RMSNorm(hidden_size, eps=config.rms_norm_eps) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn( @@ -640,12 +639,6 @@ class LLaDA2MoeBlock(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=is_previous_layer_sparse, next_layer_sparse=self.is_layer_sparse - ) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def _is_layer_sparse(self, config: PretrainedConfig, layer_id: int) -> bool: diff --git a/python/sglang/srt/models/llama.py b/python/sglang/srt/models/llama.py index 5ab4729efe..be0ff07b25 100644 --- a/python/sglang/srt/models/llama.py +++ b/python/sglang/srt/models/llama.py @@ -31,9 +31,9 @@ from sglang.srt.distributed import ( from sglang.srt.layers.activation import SiluAndMul from sglang.srt.layers.aux_hidden_states import AuxHiddenStateList from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layer_boundary.residual.add_norm import NORM_QUANT_READOUT @@ -361,14 +361,12 @@ class LlamaDecoderLayer(nn.Module): return self.input_layernorm.fuse_input_quant(self.self_attn.qkv_proj) self.post_attention_layernorm.fuse_input_quant(self.mlp.gate_up_proj) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(read=self._attn_readout(layer_id)), self.input_layernorm), ( declare_ffn(sparse=False, next_layer_sparse=False), self.post_attention_layernorm, ), - previous=declare_ffn() if layer_id != 0 else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def _attn_readout(self, layer_id: int): diff --git a/python/sglang/srt/models/llama4.py b/python/sglang/srt/models/llama4.py index 6549923f60..7320b7e19c 100644 --- a/python/sglang/srt/models/llama4.py +++ b/python/sglang/srt/models/llama4.py @@ -30,9 +30,9 @@ from sglang.srt.layers.dp_attention import ( is_dp_attention_enabled, ) from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -398,7 +398,6 @@ class Llama4DecoderLayer(nn.Module): ) self.config = config is_moe_layer = self._is_moe_layer(layer_id) - is_previous_moe_layer = self._is_moe_layer(layer_id - 1) is_next_moe_layer = self._is_moe_layer(layer_id + 1) if is_moe_layer: @@ -422,7 +421,7 @@ class Llama4DecoderLayer(nn.Module): config.hidden_size, eps=config.rms_norm_eps ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn( @@ -431,12 +430,6 @@ class Llama4DecoderLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=is_previous_moe_layer, next_layer_sparse=is_moe_layer - ) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def _is_moe_layer(self, layer_id: int) -> bool: @@ -499,7 +492,10 @@ class Llama4Model(nn.Module): self.layers = make_layers( config.num_hidden_layers, lambda idx, prefix: Llama4DecoderLayer( - config=config, layer_id=idx, quant_config=quant_config, prefix=prefix + config=config, + layer_id=idx, + quant_config=quant_config, + prefix=prefix, ), prefix=add_prefix("layers", prefix), ) diff --git a/python/sglang/srt/models/llama_eagle.py b/python/sglang/srt/models/llama_eagle.py index 400b431f11..6b4b7eca3e 100644 --- a/python/sglang/srt/models/llama_eagle.py +++ b/python/sglang/srt/models/llama_eagle.py @@ -25,6 +25,7 @@ import torch from torch import nn from transformers import LlamaConfig +from sglang.srt.layers.layer_boundary import layer_stack from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layer_boundary.residual.post_norm import PLAIN_READOUT from sglang.srt.layers.logits_processor import LogitsProcessor @@ -74,17 +75,18 @@ class LlamaModel(nn.Module): config.hidden_size, prefix=add_prefix("embed_tokens", prefix), ) - self.layers = nn.ModuleList( - [ - LlamaDecoderLayer( - config, - i, - quant_config=quant_config, - prefix=add_prefix(f"layers.{i}", prefix), - ) - for i in range(config.num_hidden_layers) - ] - ) + with layer_stack(): + self.layers = nn.ModuleList( + [ + LlamaDecoderLayer( + config, + i, + quant_config=quant_config, + prefix=add_prefix(f"layers.{i}", prefix), + ) + for i in range(config.num_hidden_layers) + ] + ) self.fc = torch.nn.Linear(config.hidden_size * 2, config.hidden_size) def forward( diff --git a/python/sglang/srt/models/llama_eagle3.py b/python/sglang/srt/models/llama_eagle3.py index 4244ed1853..243d0749e9 100644 --- a/python/sglang/srt/models/llama_eagle3.py +++ b/python/sglang/srt/models/llama_eagle3.py @@ -27,6 +27,7 @@ import torch from torch import nn from transformers import LlamaConfig +from sglang.srt.layers.layer_boundary import layer_stack from sglang.srt.layers.layernorm import RMSNorm from sglang.srt.layers.linear import QKVParallelLinear from sglang.srt.layers.logits_processor import LogitsProcessor @@ -193,12 +194,13 @@ class LlamaModel(nn.Module): else: self.fc_norm = None - self.layers = nn.ModuleList( - [ - LlamaDecoderLayer(config, i, quant_config, prefix) - for i in range(config.num_hidden_layers) - ] - ) + with layer_stack(): + self.layers = nn.ModuleList( + [ + LlamaDecoderLayer(config, i, quant_config, prefix) + for i in range(config.num_hidden_layers) + ] + ) self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) self.norm_output = getattr(config, "norm_output", False) diff --git a/python/sglang/srt/models/longcat_flash.py b/python/sglang/srt/models/longcat_flash.py index b385738767..193cfffcc4 100644 --- a/python/sglang/srt/models/longcat_flash.py +++ b/python/sglang/srt/models/longcat_flash.py @@ -52,9 +52,10 @@ from sglang.srt.layers.dp_attention import ( is_dp_attention_enabled, ) from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, + layer_stack, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -412,7 +413,7 @@ class LongcatFlashDecoderLayer(nn.Module): prefix=add_prefix("mlp", prefix), ) - self.attn_boundary, self.moe_boundary = make_stages( + self.attn_boundary, self.moe_boundary = append_stages( ( declare_attn(), self.input_layernorm[0], @@ -422,13 +423,9 @@ class LongcatFlashDecoderLayer(nn.Module): declare_ffn(sparse=True, next_layer_sparse=True), self.post_attention_layernorm[0], ), - previous=declare_ffn(sparse=True, next_layer_sparse=True) - if self.layer_id != 0 - else None, - terminal=self.layer_id == config.num_hidden_layers - 1, ) self.first_ffn_boundary, self.second_attn_boundary, self.second_ffn_boundary = ( - make_stages( + append_stages( ( declare_ffn(), self.post_attention_layernorm[0], @@ -562,18 +559,19 @@ class LongcatFlashModel(nn.Module): ) self.alt_stream = get_stream("alt") - self.layers = nn.ModuleList( - [ - LongcatFlashDecoderLayer( - config, - layer_id, - quant_config=quant_config, - prefix=add_prefix(f"layers.{layer_id}", prefix), - alt_stream=self.alt_stream, - ) - for layer_id in range(config.num_hidden_layers) - ] - ) + with layer_stack(): + self.layers = nn.ModuleList( + [ + LongcatFlashDecoderLayer( + config, + layer_id, + quant_config=quant_config, + prefix=add_prefix(f"layers.{layer_id}", prefix), + alt_stream=self.alt_stream, + ) + for layer_id in range(config.num_hidden_layers) + ] + ) self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) self.layers_to_capture = [] diff --git a/python/sglang/srt/models/longcat_flash_nextn.py b/python/sglang/srt/models/longcat_flash_nextn.py index 112d28fbe5..4bc7ea86f9 100644 --- a/python/sglang/srt/models/longcat_flash_nextn.py +++ b/python/sglang/srt/models/longcat_flash_nextn.py @@ -45,9 +45,10 @@ from sglang.srt.layers.dp_attention import ( is_dp_attention_enabled, ) from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, + layer_stack, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -162,14 +163,12 @@ class LongcatFlashDenseDecoderLayer(nn.Module): config.hidden_size, eps=config.rms_norm_eps ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn(sparse=False, next_layer_sparse=False), self.post_attention_layernorm, ), - previous=declare_ffn() if self.layer_id != 0 else None, - terminal=self.layer_id == config.num_hidden_layers - 1, ) def forward( @@ -224,9 +223,10 @@ class LongcatFlashModelNextN(nn.Module): quant_config=quant_config, prefix=add_prefix("eh_proj", ""), ) - self.decoder = LongcatFlashDenseDecoderLayer( - config, 0, quant_config=quant_config, alt_stream=self.alt_stream - ) + with layer_stack(): + self.decoder = LongcatFlashDenseDecoderLayer( + config, 0, quant_config=quant_config, alt_stream=self.alt_stream + ) self.final_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) diff --git a/python/sglang/srt/models/mellum.py b/python/sglang/srt/models/mellum.py index ee1c076e79..2a87134ab2 100644 --- a/python/sglang/srt/models/mellum.py +++ b/python/sglang/srt/models/mellum.py @@ -26,9 +26,9 @@ from torch import nn from transformers import PretrainedConfig from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layernorm import RMSNorm from sglang.srt.layers.linear import QKVParallelLinear, RowParallelLinear @@ -439,13 +439,12 @@ class MellumDecoderLayer(Qwen3MoeDecoderLayer): reduce_results=False, ) - is_previous_layer_sparse = _is_sparse(layer_id - 1) is_next_layer_sparse = _is_sparse(layer_id + 1) self.input_layernorm = RMSNorm(cfg.hidden_size, eps=rms_norm_eps) self.post_attention_layernorm = RMSNorm(cfg.hidden_size, eps=rms_norm_eps) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn( @@ -454,12 +453,6 @@ class MellumDecoderLayer(Qwen3MoeDecoderLayer): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=is_previous_layer_sparse, next_layer_sparse=self.is_layer_sparse - ) - if layer_id != 0 - else None, - terminal=layer_id == cfg.num_hidden_layers - 1, ) diff --git a/python/sglang/srt/models/mimo_mtp.py b/python/sglang/srt/models/mimo_mtp.py index 8391b58699..eeedc32f26 100644 --- a/python/sglang/srt/models/mimo_mtp.py +++ b/python/sglang/srt/models/mimo_mtp.py @@ -8,6 +8,7 @@ import torch from torch import nn from transformers import PretrainedConfig +from sglang.srt.layers.layer_boundary import layer_stack from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm from sglang.srt.layers.logits_processor import LogitsProcessor @@ -40,9 +41,10 @@ class MiMoMultiTokenPredictorLayer(nn.Module): self.input_proj = nn.Linear( config.hidden_size * 2, config.hidden_size, bias=False ) - self.mtp_block = Qwen2DecoderLayer( - config=config, quant_config=quant_config, prefix=prefix - ) + with layer_stack(): + self.mtp_block = Qwen2DecoderLayer( + config=config, quant_config=quant_config, prefix=prefix + ) self.final_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) def forward( diff --git a/python/sglang/srt/models/mimo_v2.py b/python/sglang/srt/models/mimo_v2.py index 47e650259b..5aa378cace 100644 --- a/python/sglang/srt/models/mimo_v2.py +++ b/python/sglang/srt/models/mimo_v2.py @@ -36,10 +36,10 @@ from sglang.srt.layers.dp_attention import ( is_dp_attention_enabled, ) from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, is_dense_ffn_fully_dp, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -826,7 +826,6 @@ class MiMoV2DecoderLayer(nn.Module): ) self.is_layer_sparse = self.is_moe_layer(layer_id) - is_previous_layer_sparse = self.is_moe_layer(layer_id - 1) is_next_layer_sparse = self.is_moe_layer(layer_id + 1) if self.is_layer_sparse: @@ -857,7 +856,7 @@ class MiMoV2DecoderLayer(nn.Module): config.hidden_size, eps=config.layernorm_epsilon ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn( @@ -866,12 +865,6 @@ class MiMoV2DecoderLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=is_previous_layer_sparse, next_layer_sparse=self.is_layer_sparse - ) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( diff --git a/python/sglang/srt/models/mimo_v2_nextn.py b/python/sglang/srt/models/mimo_v2_nextn.py index a6a76a2c48..75a043fad9 100644 --- a/python/sglang/srt/models/mimo_v2_nextn.py +++ b/python/sglang/srt/models/mimo_v2_nextn.py @@ -25,10 +25,11 @@ from sglang.srt.layers.dp_attention import ( is_dp_attention_enabled, ) from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, is_dense_ffn_fully_dp, - make_stages, + layer_stack, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -98,7 +99,6 @@ class MiMoV2MTPLayer(nn.Module): prefix=add_prefix("self_attn", prefix), ) self.is_layer_sparse = False - is_previous_layer_sparse = True is_next_layer_sparse = False if is_dense_ffn_fully_dp(): @@ -120,7 +120,7 @@ class MiMoV2MTPLayer(nn.Module): config.hidden_size, eps=config.layernorm_epsilon ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn( @@ -129,12 +129,6 @@ class MiMoV2MTPLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=is_previous_layer_sparse, next_layer_sparse=self.is_layer_sparse - ) - if layer_id != 0 - else None, - terminal=layer_id == 1 - 1, ) def forward( @@ -185,12 +179,13 @@ class MiMoV2ModelNextN(nn.Module): self.eh_proj = nn.Linear(2 * config.hidden_size, config.hidden_size, bias=False) - self.mtp_block = MiMoV2MTPLayer( - config, - 0, - quant_config=quant_config, - prefix=add_prefix("decoder", prefix), - ) + with layer_stack(): + self.mtp_block = MiMoV2MTPLayer( + config, + 0, + quant_config=quant_config, + prefix=add_prefix("decoder", prefix), + ) self.final_layernorm = RMSNorm(config.hidden_size, eps=config.layernorm_epsilon) def forward( diff --git a/python/sglang/srt/models/minimax_m2.py b/python/sglang/srt/models/minimax_m2.py index b25d7f89d1..ff9d3d7b41 100644 --- a/python/sglang/srt/models/minimax_m2.py +++ b/python/sglang/srt/models/minimax_m2.py @@ -39,9 +39,9 @@ from sglang.srt.layers.dp_attention import ( is_dp_attention_enabled, ) from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -99,6 +99,7 @@ from sglang.srt.utils import ( is_xpu, make_pp_layers, ) +from sglang.srt.utils.common import is_building_neighbour_layer from sglang.srt.utils.custom_op import register_custom_op from sglang.srt.utils.hf_transformers_utils import get_rope_config @@ -408,7 +409,12 @@ class MiniMaxM2QKRMSNorm: use_fused_norm = get_bool_env_var("SGLANG_USE_FUSED_PARALLEL_QKNORM") self._forward_impl = self._forward_naive - if self._world_size > 1 and _is_cuda and use_fused_norm: + if ( + self._world_size > 1 + and _is_cuda + and use_fused_norm + and not is_building_neighbour_layer() + ): occupancy = get_fused_parallel_qknorm_max_occupancy( q_norm.weight.dtype, self._world_size, @@ -867,10 +873,9 @@ class MiniMaxM2DecoderLayer(nn.Module): config.hidden_size, eps=getattr(config, "rms_norm_eps", 1e-6) ) - is_previous_layer_sparse = True is_next_layer_sparse = True - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn( @@ -879,12 +884,6 @@ class MiniMaxM2DecoderLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=is_previous_layer_sparse, next_layer_sparse=self.is_layer_sparse - ) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( diff --git a/python/sglang/srt/models/minimax_m3.py b/python/sglang/srt/models/minimax_m3.py index 3711a6730a..1f40a8481d 100644 --- a/python/sglang/srt/models/minimax_m3.py +++ b/python/sglang/srt/models/minimax_m3.py @@ -40,10 +40,10 @@ from sglang.srt.layers.dp_attention import ( is_dp_attention_enabled, ) from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, is_dense_ffn_fully_dp, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import GemmaRMSNorm, RMSNorm @@ -1517,10 +1517,9 @@ class MiniMaxM3DecoderLayer(nn.Module): return True return moe_layer_freq[lid] != 0 - is_previous_layer_sparse = _is_layer_sparse(layer_id - 1) is_next_layer_sparse = _is_layer_sparse(layer_id + 1) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn( @@ -1529,12 +1528,6 @@ class MiniMaxM3DecoderLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=is_previous_layer_sparse, next_layer_sparse=self.is_layer_sparse - ) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( diff --git a/python/sglang/srt/models/mistral_eagle.py b/python/sglang/srt/models/mistral_eagle.py index f7dcf9dbe9..42bd625a95 100644 --- a/python/sglang/srt/models/mistral_eagle.py +++ b/python/sglang/srt/models/mistral_eagle.py @@ -40,6 +40,7 @@ import torch from torch import nn from transformers import PretrainedConfig +from sglang.srt.layers.layer_boundary import layer_stack from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm from sglang.srt.layers.linear import RowParallelLinear @@ -75,17 +76,18 @@ class MistralEagleModel(nn.Module): config.hidden_size, prefix=add_prefix("embed_tokens", prefix), ) - self.layers = nn.ModuleList( - [ - LlamaDecoderLayer( - config=config, - layer_id=i, - prefix=add_prefix(f"layers.{i}", prefix), - quant_config=quant_config, - ) - for i in range(config.num_hidden_layers) - ] - ) + with layer_stack(): + self.layers = nn.ModuleList( + [ + LlamaDecoderLayer( + config=config, + layer_id=i, + prefix=add_prefix(f"layers.{i}", prefix), + quant_config=quant_config, + ) + for i in range(config.num_hidden_layers) + ] + ) self.start_layer = 0 self.end_layer = config.num_hidden_layers self.fc = RowParallelLinear( diff --git a/python/sglang/srt/models/mixtral.py b/python/sglang/srt/models/mixtral.py index 271c7634e3..2ef687ebc1 100644 --- a/python/sglang/srt/models/mixtral.py +++ b/python/sglang/srt/models/mixtral.py @@ -26,9 +26,9 @@ from torch import nn from transformers import MixtralConfig from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -235,16 +235,12 @@ class MixtralDecoderLayer(nn.Module): self.post_attention_layernorm = RMSNorm( config.hidden_size, eps=config.rms_norm_eps ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn(sparse=True, next_layer_sparse=True), self.post_attention_layernorm, ), - previous=declare_ffn(sparse=True, next_layer_sparse=True) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( diff --git a/python/sglang/srt/models/mllama.py b/python/sglang/srt/models/mllama.py index b3b4dd82d2..3171b97fac 100644 --- a/python/sglang/srt/models/mllama.py +++ b/python/sglang/srt/models/mllama.py @@ -22,6 +22,7 @@ from transformers.models.mllama.modeling_mllama import ( from sglang.srt.layers.activation import get_act_fn from sglang.srt.layers.attention.vision import VisionAttention +from sglang.srt.layers.layer_boundary import layer_stack from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm from sglang.srt.layers.linear import ( @@ -678,26 +679,27 @@ class MllamaTextModel(nn.Module): self.cross_attention_layers = config.cross_attention_layers layers = [] - for layer_id in range(config.num_hidden_layers): - if layer_id in self.cross_attention_layers: - layers.append( - MllamaCrossAttentionDecoderLayer( - config, - layer_id, - quant_config=quant_config, - prefix=add_prefix(f"layers.{layer_id}", prefix), + with layer_stack(): + for layer_id in range(config.num_hidden_layers): + if layer_id in self.cross_attention_layers: + layers.append( + MllamaCrossAttentionDecoderLayer( + config, + layer_id, + quant_config=quant_config, + prefix=add_prefix(f"layers.{layer_id}", prefix), + ) ) - ) - else: - # TODO: force LlamaDecoderLayer to config.attention_bias=False - layers.append( - LlamaDecoderLayer( - config, - quant_config=quant_config, - layer_id=layer_id, - prefix=add_prefix(f"layers.{layer_id}", prefix), + else: + # TODO: force LlamaDecoderLayer to config.attention_bias=False + layers.append( + LlamaDecoderLayer( + config, + quant_config=quant_config, + layer_id=layer_id, + prefix=add_prefix(f"layers.{layer_id}", prefix), + ) ) - ) self.layers = nn.ModuleList(layers) self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) diff --git a/python/sglang/srt/models/moss_vl.py b/python/sglang/srt/models/moss_vl.py index 56c89c889a..94dd9d75b4 100644 --- a/python/sglang/srt/models/moss_vl.py +++ b/python/sglang/srt/models/moss_vl.py @@ -21,9 +21,10 @@ from sglang.srt.layers.attention.vision import ( ) from sglang.srt.layers.conv import Conv3dLayer from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, + layer_stack, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -1017,14 +1018,12 @@ class MossVLSelfAttentionDecoderLayer(nn.Module): config.hidden_size, eps=config.rms_norm_eps, **norm_kwargs ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn(sparse=False, next_layer_sparse=False), self.post_attention_layernorm, ), - previous=declare_ffn() if layer_id != 0 else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( @@ -1070,25 +1069,26 @@ class MossVLTextModel(nn.Module): self.cross_attention_layers = config.cross_attention_layers layers = [] - for layer_id in range(config.num_hidden_layers): - if layer_id in self.cross_attention_layers: - layers.append( - MossVLCrossAttentionDecoderLayer( - config, - layer_id, - quant_config=quant_config, - prefix=add_prefix(f"layers.{layer_id}", prefix), + with layer_stack(): + for layer_id in range(config.num_hidden_layers): + if layer_id in self.cross_attention_layers: + layers.append( + MossVLCrossAttentionDecoderLayer( + config, + layer_id, + quant_config=quant_config, + prefix=add_prefix(f"layers.{layer_id}", prefix), + ) ) - ) - else: - layers.append( - MossVLSelfAttentionDecoderLayer( - config, - layer_id, - quant_config=quant_config, - prefix=add_prefix(f"layers.{layer_id}", prefix), + else: + layers.append( + MossVLSelfAttentionDecoderLayer( + config, + layer_id, + quant_config=quant_config, + prefix=add_prefix(f"layers.{layer_id}", prefix), + ) ) - ) self.layers = nn.ModuleList(layers) self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) diff --git a/python/sglang/srt/models/nanbeige.py b/python/sglang/srt/models/nanbeige.py index f8e522da5a..dc2244a323 100644 --- a/python/sglang/srt/models/nanbeige.py +++ b/python/sglang/srt/models/nanbeige.py @@ -9,9 +9,9 @@ from sglang.srt.layers.activation import SiluAndMul from sglang.srt.layers.aux_hidden_states import AuxHiddenStateList from sglang.srt.layers.dp_attention import is_dp_attention_enabled from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -231,14 +231,12 @@ class NanbeigeDecoderLayer(nn.Module): ) # Each loop runs the physical layers as one stack: the model folds the # residual back in and re-enters at layer 0. - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn(sparse=False, next_layer_sparse=False), self.post_attention_layernorm, ), - previous=declare_ffn() if layer_id != 0 else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( diff --git a/python/sglang/srt/models/nemotron_h_mtp.py b/python/sglang/srt/models/nemotron_h_mtp.py index 14a77ac252..c050745e0a 100644 --- a/python/sglang/srt/models/nemotron_h_mtp.py +++ b/python/sglang/srt/models/nemotron_h_mtp.py @@ -19,6 +19,7 @@ from torch import nn from sglang.srt.configs import NemotronHConfig from sglang.srt.layers.dp_attention import is_dp_attention_enabled +from sglang.srt.layers.layer_boundary import layer_stack from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm from sglang.srt.layers.linear import ColumnParallelLinear @@ -239,34 +240,37 @@ class NemotronHMultiTokenPredictor(nn.Module): # Total number of physical layers = num_steps * pattern_len total_layers = self.num_mtp_layers * self.pattern_len - for i in range(total_layers): - step_rel_idx = i % self.pattern_len + with layer_stack(): + for i in range(total_layers): + step_rel_idx = i % self.pattern_len - char = self.pattern_str[step_rel_idx] + char = self.pattern_str[step_rel_idx] - is_start_of_step = step_rel_idx == 0 - is_end_of_step = step_rel_idx == self.pattern_len - 1 + is_start_of_step = step_rel_idx == 0 + is_end_of_step = step_rel_idx == self.pattern_len - 1 - layer_prefix = f"{prefix}.layers.{i}" + layer_prefix = f"{prefix}.layers.{i}" - common_kwargs = dict( - config=config, - layer_idx=i, - quant_config=quant_config, - prefix=layer_prefix, - has_start_projections=is_start_of_step, - has_end_norm=is_end_of_step, - ) - - if char == "*": - self.layers[str(i)] = NemotronHMTPAttentionDecoderLayer(**common_kwargs) - elif char == "E": - self.layers[str(i)] = NemotronHMTPMoEDecoderLayer(**common_kwargs) - else: - raise NotImplementedError( - f"Pattern char '{char}' in {self.pattern_str} not implemented" + common_kwargs = dict( + config=config, + layer_idx=i, + quant_config=quant_config, + prefix=layer_prefix, + has_start_projections=is_start_of_step, + has_end_norm=is_end_of_step, ) + if char == "*": + self.layers[str(i)] = NemotronHMTPAttentionDecoderLayer( + **common_kwargs + ) + elif char == "E": + self.layers[str(i)] = NemotronHMTPMoEDecoderLayer(**common_kwargs) + else: + raise NotImplementedError( + f"Pattern char '{char}' in {self.pattern_str} not implemented" + ) + def get_input_embeddings(self, input_ids: torch.Tensor) -> torch.Tensor: assert self.embed_tokens is not None, ( "embed_tokens not initialized - must be shared from target model" diff --git a/python/sglang/srt/models/nemotron_h_utils.py b/python/sglang/srt/models/nemotron_h_utils.py index afed7add02..677a0182d6 100644 --- a/python/sglang/srt/models/nemotron_h_utils.py +++ b/python/sglang/srt/models/nemotron_h_utils.py @@ -4,9 +4,9 @@ from sglang.srt.configs.nemotron_h import ATTENTION, MAMBA, MOE from sglang.srt.layers.layer_boundary import ( ExitRows, ProducerReduction, + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual.add_norm import ( NormQuantReadout, @@ -42,16 +42,10 @@ def make_stage_boundary(layer_norm: RMSNorm, *, pattern: str, layer_idx: int): if get_parallel().attn_cp_size > 1 or layernorm_sp.layernorm_sp_enabled(): raise NotImplementedError("a Nemotron stage with attention CP or LayerNorm SP") - previous = _declaration(pattern, layer_idx - 1) if layer_idx > 0 else None - (boundary,) = make_stages( + (boundary,) = append_stages( ( _declaration(pattern, layer_idx), layer_norm, ), - previous=previous, - following=_declaration(pattern, layer_idx + 1) - if layer_idx + 1 < len(pattern) - else None, - terminal=layer_idx == len(pattern) - 1, ) return boundary diff --git a/python/sglang/srt/models/nemotron_nas.py b/python/sglang/srt/models/nemotron_nas.py index de20b5766c..54a6688734 100644 --- a/python/sglang/srt/models/nemotron_nas.py +++ b/python/sglang/srt/models/nemotron_nas.py @@ -24,9 +24,9 @@ from torch import nn from transformers import LlamaConfig from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -63,21 +63,6 @@ def _find_multiple(n: int, k: int) -> int: return n + k - (n % k) -def _previous_stage(block_configs, layer_idx: int): - """The declaration of the last attention or FFN before this layer, or None - when this layer's first stage starts the layer stack.""" - for block in reversed(block_configs[:layer_idx]): - if not block.ffn.no_op: - return declare_ffn() - if not block.attention.no_op: - return declare_attn() - return None - - -def _has_stage(block) -> bool: - return not (block.attention.no_op and block.ffn.no_op) - - class DeciLMDecoderLayer(nn.Module): def __init__( self, @@ -162,12 +147,8 @@ class DeciLMDecoderLayer(nn.Module): ) self.entry_boundary = None if stages: - boundaries = make_stages( + boundaries = append_stages( *stages, - previous=_previous_stage(config.block_configs, layer_idx), - terminal=not any( - _has_stage(block) for block in config.block_configs[layer_idx + 1 :] - ), ) self.entry_boundary = boundaries[0] if not self._is_no_op_attention: diff --git a/python/sglang/srt/models/qwen2.py b/python/sglang/srt/models/qwen2.py index a5a1c128ce..545b6618df 100644 --- a/python/sglang/srt/models/qwen2.py +++ b/python/sglang/srt/models/qwen2.py @@ -29,9 +29,9 @@ from sglang.srt.layers.activation import SiluAndMul from sglang.srt.layers.aux_hidden_states import AuxHiddenStateList from sglang.srt.layers.dp_attention import is_dp_attention_enabled from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layer_boundary.residual.add_norm import NORM_QUANT_READOUT @@ -292,14 +292,12 @@ class Qwen2DecoderLayer(nn.Module): ) self.input_layernorm.fuse_input_quant(self.self_attn.qkv_proj) self.post_attention_layernorm.fuse_input_quant(self.mlp.gate_up_proj) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(read=self._attn_readout(layer_id)), self.input_layernorm), ( declare_ffn(sparse=False, next_layer_sparse=False), self.post_attention_layernorm, ), - previous=declare_ffn() if layer_id != 0 else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def _attn_readout(self, layer_id: int): diff --git a/python/sglang/srt/models/qwen2_eagle.py b/python/sglang/srt/models/qwen2_eagle.py index 51375ebea0..195816fb02 100644 --- a/python/sglang/srt/models/qwen2_eagle.py +++ b/python/sglang/srt/models/qwen2_eagle.py @@ -24,6 +24,7 @@ from typing import Iterable, Optional, Tuple import torch from torch import nn +from sglang.srt.layers.layer_boundary import layer_stack from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layer_boundary.residual.post_norm import PLAIN_READOUT from sglang.srt.layers.logits_processor import LogitsProcessor @@ -75,17 +76,18 @@ class Qwen2Model(nn.Module): config.hidden_size, prefix=add_prefix("embed_tokens", prefix), ) - self.layers = nn.ModuleList( - [ - Qwen2DecoderLayer( - config, - i, - quant_config=quant_config, - prefix=add_prefix(f"layers.{i}", prefix), - ) - for i in range(config.num_hidden_layers) - ] - ) + with layer_stack(): + self.layers = nn.ModuleList( + [ + Qwen2DecoderLayer( + config, + i, + quant_config=quant_config, + prefix=add_prefix(f"layers.{i}", prefix), + ) + for i in range(config.num_hidden_layers) + ] + ) self.fc = torch.nn.Linear(config.hidden_size * 2, config.hidden_size) def forward( diff --git a/python/sglang/srt/models/qwen2_moe.py b/python/sglang/srt/models/qwen2_moe.py index 9e60ee3216..dbc5250375 100644 --- a/python/sglang/srt/models/qwen2_moe.py +++ b/python/sglang/srt/models/qwen2_moe.py @@ -49,9 +49,9 @@ from sglang.srt.layers.dp_attention import ( ) from sglang.srt.layers.flashinfer_comm_fusion import uses_cutedsl_ar_fusion from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -1027,7 +1027,6 @@ class Qwen2MoeDecoderLayer(nn.Module): # Qwen2MoE all layers are sparse and have no nextn now self.is_layer_sparse = True - is_previous_layer_sparse = True is_next_layer_sparse = True if self.is_layer_sparse: @@ -1052,7 +1051,7 @@ class Qwen2MoeDecoderLayer(nn.Module): self.post_attention_layernorm = RMSNorm( config.hidden_size, eps=config.rms_norm_eps ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn( @@ -1061,12 +1060,6 @@ class Qwen2MoeDecoderLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=is_previous_layer_sparse, next_layer_sparse=self.is_layer_sparse - ) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( diff --git a/python/sglang/srt/models/qwen3.py b/python/sglang/srt/models/qwen3.py index 7e1ac5b2f3..1d9752e25a 100644 --- a/python/sglang/srt/models/qwen3.py +++ b/python/sglang/srt/models/qwen3.py @@ -7,9 +7,9 @@ from torch import nn from sglang.srt.layers.aux_hidden_states import AuxHiddenStateList from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -375,14 +375,12 @@ class Qwen3DecoderLayer(nn.Module): config.hidden_size, eps=config.rms_norm_eps, **norm_kwargs ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn(sparse=False, next_layer_sparse=False), self.post_attention_layernorm, ), - previous=declare_ffn() if layer_id != 0 else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( diff --git a/python/sglang/srt/models/qwen3_5.py b/python/sglang/srt/models/qwen3_5.py index a26ed8edf7..215f057960 100644 --- a/python/sglang/srt/models/qwen3_5.py +++ b/python/sglang/srt/models/qwen3_5.py @@ -54,9 +54,9 @@ from sglang.srt.layers.dp_attention import ( ) from sglang.srt.layers.flashinfer_comm_fusion import uses_cutedsl_ar_fusion from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) # Layers - Attention @@ -447,7 +447,6 @@ class Qwen3_5GatedDeltaNet(nn.Module): eps=self.layer_norm_epsilon, group_size=None, norm_before_gate=True, - device=torch.get_device_module().current_device(), dtype=torch.get_default_dtype(), **( {"activation": self.output_gate_type} @@ -1078,7 +1077,6 @@ class Qwen3_5LinearDecoderLayer(nn.Module): reduce_results=False, ) is_layer_sparse = True - is_previous_layer_sparse = True is_next_layer_sparse = True elif config.model_type == "qwen3_5_text": self.mlp = Qwen2MoeMLP( @@ -1092,7 +1090,6 @@ class Qwen3_5LinearDecoderLayer(nn.Module): ) _maybe_enable_silu_fp4_quant_fusion(self.mlp) is_layer_sparse = False - is_previous_layer_sparse = False is_next_layer_sparse = False else: raise ValueError(f"Invalid model type: {config.model_type}") @@ -1108,7 +1105,7 @@ class Qwen3_5LinearDecoderLayer(nn.Module): boundary_fusions = _layer_fusions(config, is_nextn) # A subclass that brings its own residual builds the stages itself. if build_stages: - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( ( declare_attn( read=NormQuantReadout( @@ -1130,12 +1127,6 @@ class Qwen3_5LinearDecoderLayer(nn.Module): self.post_attention_layernorm, {"fusions": boundary_fusions}, ), - previous=declare_ffn( - sparse=is_previous_layer_sparse, next_layer_sparse=is_layer_sparse - ) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( @@ -1285,7 +1276,6 @@ class Qwen3_5AttentionDecoderLayer(nn.Module): reduce_results=False, ) is_layer_sparse = False - is_previous_layer_sparse = False is_next_layer_sparse = False elif config.model_type in _QWEN3_5_MOE_TEXT_MODEL_TYPES: self.mlp = Qwen2MoeSparseMoeBlock( @@ -1304,7 +1294,6 @@ class Qwen3_5AttentionDecoderLayer(nn.Module): reduce_results=False, ) is_layer_sparse = True - is_previous_layer_sparse = True is_next_layer_sparse = True else: raise ValueError(f"Invalid model type: {config.model_type}") @@ -1323,7 +1312,7 @@ class Qwen3_5AttentionDecoderLayer(nn.Module): boundary_fusions = _layer_fusions(config, is_nextn) # A subclass that brings its own residual builds the stages itself. if build_stages: - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( ( declare_attn( read=NormQuantReadout( @@ -1343,12 +1332,6 @@ class Qwen3_5AttentionDecoderLayer(nn.Module): self.post_attention_layernorm, {"fusions": boundary_fusions}, ), - previous=declare_ffn( - sparse=is_previous_layer_sparse, next_layer_sparse=is_layer_sparse - ) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) self.alt_stream = alt_stream diff --git a/python/sglang/srt/models/qwen3_moe.py b/python/sglang/srt/models/qwen3_moe.py index dff2427627..0f9aa9bb39 100644 --- a/python/sglang/srt/models/qwen3_moe.py +++ b/python/sglang/srt/models/qwen3_moe.py @@ -31,9 +31,9 @@ from sglang.srt.eplb.expert_location import ModelConfigForExpertLocation from sglang.srt.eplb.expert_location_dispatch import ExpertLocationDispatchInfo from sglang.srt.layers.aux_hidden_states import AuxHiddenStateAccumulator from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -858,7 +858,6 @@ class Qwen3MoeDecoderLayer(nn.Module): # Qwen3MoE all layers are sparse and have no nextn now self.is_layer_sparse = True - is_previous_layer_sparse = True is_next_layer_sparse = True if self.is_layer_sparse: @@ -882,7 +881,7 @@ class Qwen3MoeDecoderLayer(nn.Module): config.hidden_size, eps=config.rms_norm_eps ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn( @@ -891,12 +890,6 @@ class Qwen3MoeDecoderLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=is_previous_layer_sparse, next_layer_sparse=self.is_layer_sparse - ) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( diff --git a/python/sglang/srt/models/qwen3_next.py b/python/sglang/srt/models/qwen3_next.py index 44b685d45c..463858940e 100644 --- a/python/sglang/srt/models/qwen3_next.py +++ b/python/sglang/srt/models/qwen3_next.py @@ -23,9 +23,9 @@ from sglang.srt.layers.dp_attention import ( is_dp_attention_enabled, ) from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import GemmaRMSNorm @@ -509,7 +509,6 @@ class Qwen3HybridLinearDecoderLayer(nn.Module): # Qwen3Next all layers are sparse and have no nextn now self.is_layer_sparse = True - is_previous_layer_sparse = True is_next_layer_sparse = True self.layer_id = layer_id @@ -538,7 +537,7 @@ class Qwen3HybridLinearDecoderLayer(nn.Module): self.post_attention_layernorm = GemmaRMSNorm( config.hidden_size, eps=config.rms_norm_eps ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn( @@ -547,12 +546,6 @@ class Qwen3HybridLinearDecoderLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=is_previous_layer_sparse, next_layer_sparse=self.is_layer_sparse - ) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( @@ -676,7 +669,6 @@ class Qwen3HybridAttentionDecoderLayer(nn.Module): # Qwen3Next all layers are sparse and have no nextn now self.is_layer_sparse = True - is_previous_layer_sparse = True is_next_layer_sparse = True if self.is_layer_sparse: @@ -708,7 +700,7 @@ class Qwen3HybridAttentionDecoderLayer(nn.Module): self.q_norm = GemmaRMSNorm(self.head_dim, eps=config.rms_norm_eps) self.k_norm = GemmaRMSNorm(self.head_dim, eps=config.rms_norm_eps) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn( @@ -717,12 +709,6 @@ class Qwen3HybridAttentionDecoderLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=is_previous_layer_sparse, next_layer_sparse=self.is_layer_sparse - ) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) self.alt_stream = alt_stream diff --git a/python/sglang/srt/models/qwen4_exp.py b/python/sglang/srt/models/qwen4_exp.py index e8dd479358..24792dfc81 100644 --- a/python/sglang/srt/models/qwen4_exp.py +++ b/python/sglang/srt/models/qwen4_exp.py @@ -39,10 +39,10 @@ from sglang.srt.layers.hyperconnection import ( from sglang.srt.layers.layer_boundary import ( ExitRows, GatedResidualState, + append_stages, declare_attn, declare_ffn, get_attn_tp_context, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.linear import ReplicatedLinear @@ -88,6 +88,7 @@ from sglang.srt.models.qwen4_exp_ple_table import ( ) from sglang.srt.runtime_context import get_parallel from sglang.srt.utils import get_bool_env_var, is_hip, logger +from sglang.srt.utils.common import is_building_neighbour_layer _use_aiter = get_bool_env_var("SGLANG_USE_AITER") and is_hip() @@ -534,6 +535,13 @@ class Qwen4ExpPLEGroupedNorm(nn.Module): return (x_norm * weight).to(compute_dtype) +def _offloads_ple(config) -> bool: + """Whether this layer keeps its PLE table in host memory. A pipeline + neighbour layer is never loaded or run, so it keeps the table on the meta + device and allocates no host memory or stream for it.""" + return bool(config.ple_offload_embedding) and not is_building_neighbour_layer() + + class Qwen4ExpNGramEmbedding(nn.Module): _MASK64 = (1 << 64) - 1 _SPLITMIX_GAMMA = 0x9E3779B97F4A7C15 @@ -605,7 +613,7 @@ class Qwen4ExpNGramEmbedding(nn.Module): and not self.use_attn_tp_ngram ) ngram_prefix = f"{prefix}.ngram_embedding" if prefix else "ngram_embedding" - offload_embedding = bool(config.ple_offload_embedding) + offload_embedding = _offloads_ple(config) # Offload only needs this embedding's metadata: build it on meta so the # shard is never allocated on the device. with torch.device("meta") if offload_embedding else nullcontext(): @@ -1088,9 +1096,7 @@ class Qwen4ExpPLELayer(nn.Module): bias=False, ) nn.init.zeros_(self.conv1d.weight) - self._prefetch_stream = ( - torch.cuda.Stream() if config.ple_offload_embedding else None - ) + self._prefetch_stream = torch.cuda.Stream() if _offloads_ple(config) else None self._graph_prefetch_buffers = {} self._eager_prefetch_buffer = None self._prefetch_state = None @@ -1441,7 +1447,7 @@ def _build_qwen4_exp_stages(residual, *, sparse, layer_id, config): the writes are gated injections, so neither stage offers a fused add-and-norm candidate. """ - return make_stages( + return append_stages( ( declare_attn(read=residual.attn_readout, update=residual.attn_update), None, @@ -1456,17 +1462,6 @@ def _build_qwen4_exp_stages(residual, *, sparse, layer_id, config): ), None, ), - previous=( - declare_ffn( - sparse=sparse, - next_layer_sparse=sparse, - update=residual.ffn_update, - exit_rows=_ffn_exit_rows(layer_id - 1, config), - ) - if layer_id != 0 - else None - ), - terminal=layer_id == config.num_hidden_layers - 1, ) diff --git a/python/sglang/srt/models/sarvam_moe.py b/python/sglang/srt/models/sarvam_moe.py index 7f2c871999..8b470d7558 100644 --- a/python/sglang/srt/models/sarvam_moe.py +++ b/python/sglang/srt/models/sarvam_moe.py @@ -20,10 +20,10 @@ from sglang.srt.layers.dp_attention import ( is_dp_attention_enabled, ) from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, is_dense_ffn_fully_dp, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -1002,12 +1002,6 @@ class SarvamMoEMLADecoderLayer(nn.Module): and layer_id >= first_k_dense and (layer_id - first_k_dense) % moe_layer_freq == 0 ) - is_previous_layer_sparse = ( - has_moe - and layer_id > 0 - and (layer_id - 1) >= first_k_dense - and (layer_id - 1 - first_k_dense) % moe_layer_freq == 0 - ) is_next_layer_sparse = ( has_moe and layer_id < config.num_hidden_layers - 1 @@ -1044,7 +1038,7 @@ class SarvamMoEMLADecoderLayer(nn.Module): config.hidden_size, eps=config.rms_norm_eps ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( ( declare_attn(), self.input_layernorm, @@ -1057,12 +1051,6 @@ class SarvamMoEMLADecoderLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=is_previous_layer_sparse, next_layer_sparse=self.is_layer_sparse - ) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( diff --git a/python/sglang/srt/models/sdar.py b/python/sglang/srt/models/sdar.py index 88f32d39f8..d96282736d 100644 --- a/python/sglang/srt/models/sdar.py +++ b/python/sglang/srt/models/sdar.py @@ -15,9 +15,9 @@ from sglang.srt.layers.dp_attention import ( is_dp_attention_enabled, ) from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -294,7 +294,7 @@ class SDARBlock(nn.Module): prefix=add_prefix("mlp", prefix), ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn( @@ -303,8 +303,6 @@ class SDARBlock(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn() if layer_id != 0 else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( diff --git a/python/sglang/srt/models/sdar_moe.py b/python/sglang/srt/models/sdar_moe.py index 43ca2b0d2d..b5372e246a 100644 --- a/python/sglang/srt/models/sdar_moe.py +++ b/python/sglang/srt/models/sdar_moe.py @@ -17,9 +17,9 @@ from sglang.srt.layers.dp_attention import ( is_dp_attention_enabled, ) from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -353,16 +353,12 @@ class SDARMoeBlock(nn.Module): prefix=add_prefix("mlp", prefix), ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn(sparse=True, next_layer_sparse=True), self.post_attention_layernorm, ), - previous=declare_ffn(sparse=True, next_layer_sparse=True) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( diff --git a/python/sglang/srt/models/spark2_5.py b/python/sglang/srt/models/spark2_5.py index 62c6296a85..f0a90aad8f 100644 --- a/python/sglang/srt/models/spark2_5.py +++ b/python/sglang/srt/models/spark2_5.py @@ -7,9 +7,9 @@ from torch import nn from sglang.srt.layers.activation import GeluAndMul from sglang.srt.layers.dp_attention import is_dp_attention_enabled from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -271,14 +271,12 @@ class Spark2_5DecoderLayer(nn.Module): self.post_attention_layernorm = RMSNorm( config.hidden_size, eps=config.rms_norm_eps ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn(sparse=False, next_layer_sparse=False), self.post_attention_layernorm, ), - previous=declare_ffn() if layer_id != 0 else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( diff --git a/python/sglang/srt/models/step3_vl.py b/python/sglang/srt/models/step3_vl.py index f8626c39bf..88711ed00f 100644 --- a/python/sglang/srt/models/step3_vl.py +++ b/python/sglang/srt/models/step3_vl.py @@ -25,9 +25,9 @@ from sglang.srt.layers.dp_attention import ( reject_attn_tp_shard_with_tp_reduce, ) from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -334,9 +334,6 @@ class Step3TextDecoderLayer(nn.Module): self.layer_id = layer_id self.is_layer_sparse = True if layer_id in moe_layers_idx else False - self.is_previous_layer_sparse = ( - True if layer_id - 1 in moe_layers_idx else False - ) self.is_next_layer_sparse = True if layer_id + 1 in moe_layers_idx else False if not self.is_layer_sparse: @@ -373,7 +370,7 @@ class Step3TextDecoderLayer(nn.Module): prefix=add_prefix("mlp", prefix), ) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn( @@ -382,13 +379,6 @@ class Step3TextDecoderLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=self.is_previous_layer_sparse, - next_layer_sparse=self.is_layer_sparse, - ) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def moe_mlp_forward(self, hidden_states): diff --git a/python/sglang/srt/models/step3p5.py b/python/sglang/srt/models/step3p5.py index 3195fa26ec..e5523a920f 100644 --- a/python/sglang/srt/models/step3p5.py +++ b/python/sglang/srt/models/step3p5.py @@ -11,9 +11,9 @@ from sglang.srt.layers.dp_attention import ( is_dp_attention_enabled, ) from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import GemmaRMSNorm @@ -409,7 +409,6 @@ class Step3p5DecoderLayer(nn.Module): quant_config: Optional[QuantizationConfig] = None, prefix: str = "", alt_stream: Optional[torch.cuda.Stream] = None, - is_nextn: bool = False, ) -> None: super().__init__() self.hidden_size = config.hidden_size @@ -426,7 +425,6 @@ class Step3p5DecoderLayer(nn.Module): self.num_attention_heads = config.num_attention_heads self.num_key_value_heads = config.num_attention_groups self.is_moe_layer = layer_id in moe_layers_set - self.is_previous_layer_sparse = (layer_id - 1) in moe_layers_set self.is_next_layer_sparse = (layer_id + 1) in moe_layers_set num_hidden_layers = config.num_hidden_layers @@ -520,9 +518,7 @@ class Step3p5DecoderLayer(nn.Module): config.hidden_size, eps=config.rms_norm_eps ) - # An MTP draft is a one-layer model; layer_id still indexes its config. - - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn( @@ -531,14 +527,6 @@ class Step3p5DecoderLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=self.is_previous_layer_sparse, - next_layer_sparse=self.is_moe_layer, - ) - if not (0 if is_nextn else layer_id) == 0 - else None, - terminal=(0 if is_nextn else layer_id) - == (1 if is_nextn else config.num_hidden_layers) - 1, ) self.layer_id = layer_id diff --git a/python/sglang/srt/models/step3p5_mtp.py b/python/sglang/srt/models/step3p5_mtp.py index 388ed66f8c..baddb80fa4 100644 --- a/python/sglang/srt/models/step3p5_mtp.py +++ b/python/sglang/srt/models/step3p5_mtp.py @@ -6,6 +6,7 @@ import torch import torch.nn as nn from transformers import PretrainedConfig +from sglang.srt.layers.layer_boundary import layer_stack from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import GemmaRMSNorm from sglang.srt.layers.logits_processor import LogitsProcessor @@ -79,12 +80,12 @@ class Step3p5AMultiTokenPredictor(nn.Module): self.hnorm = GemmaRMSNorm(config.hidden_size, config.rms_norm_eps) self.eh_proj = nn.Linear(config.hidden_size * 2, config.hidden_size, bias=False) self.shared_head = SharedHead(config=config, quant_config=quant_config) - self.mtp_block = Step3p5DecoderLayer( - config=config, - layer_id=layer_id, - prefix=f"{prefix}.mtp_block", - is_nextn=True, - ) + with layer_stack(): + self.mtp_block = Step3p5DecoderLayer( + config=config, + layer_id=layer_id, + prefix=f"{prefix}.mtp_block", + ) self.lm_head = self.shared_head.head def forward( diff --git a/python/sglang/srt/models/xllm.py b/python/sglang/srt/models/xllm.py index a74487809e..a75200a92f 100644 --- a/python/sglang/srt/models/xllm.py +++ b/python/sglang/srt/models/xllm.py @@ -41,10 +41,10 @@ from sglang.srt.eplb.expert_location_dispatch import ExpertLocationDispatchInfo from sglang.srt.layers.activation import SiluAndMul from sglang.srt.layers.dp_attention import is_dp_attention_enabled from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, is_dense_ffn_fully_dp, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layernorm import RMSNorm @@ -1495,7 +1495,6 @@ class XllmDecoderLayer(nn.Module): config.num_experts > 0 and (lid + 1) % decoder_sparse_step == 0 ) - is_previous_layer_sparse = _is_sparse(layer_id - 1) is_next_layer_sparse = _is_sparse(layer_id + 1) if self.is_layer_sparse: @@ -1523,7 +1522,7 @@ class XllmDecoderLayer(nn.Module): self.input_layernorm = _make_norm(config) self.post_attention_layernorm = _make_norm(config) - self.attn_boundary, self.ffn_boundary = make_stages( + self.attn_boundary, self.ffn_boundary = append_stages( (declare_attn(), self.input_layernorm), ( declare_ffn( @@ -1532,12 +1531,6 @@ class XllmDecoderLayer(nn.Module): ), self.post_attention_layernorm, ), - previous=declare_ffn( - sparse=is_previous_layer_sparse, next_layer_sparse=self.is_layer_sparse - ) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) def forward( diff --git a/python/sglang/srt/models/zaya.py b/python/sglang/srt/models/zaya.py index f62993e093..cf4597b0bc 100644 --- a/python/sglang/srt/models/zaya.py +++ b/python/sglang/srt/models/zaya.py @@ -53,9 +53,9 @@ from torch import nn from sglang.srt.configs.zaya import ZayaConfig from sglang.srt.layers.layer_boundary import ( + append_stages, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.layers.layer_boundary.residual import batch as residual_batch from sglang.srt.layers.layer_boundary.residual.add_norm import PLAIN_ADD @@ -1320,12 +1320,8 @@ class ZayaDecoderATTLayer(nn.Module): self.res_scale = ResidualScaling(config, layer_id) else: self.res_scale = None - (self.attn_boundary,) = make_stages( + (self.attn_boundary,) = append_stages( (declare_attn(read=_ResidualMergeRead(self.res_scale)), self.input_norm), - previous=declare_ffn(sparse=True, next_layer_sparse=True) - if layer_id != 0 - else None, - terminal=layer_id == config.num_hidden_layers - 1, ) self.entry_boundary = self.attn_boundary @@ -1375,7 +1371,7 @@ class ZayaDecoderMLPLayer(nn.Module): self.res_scale = ResidualScaling(config, layer_id) else: self.res_scale = None - (self.ffn_boundary,) = make_stages( + (self.ffn_boundary,) = append_stages( ( declare_ffn( sparse=True, @@ -1384,8 +1380,6 @@ class ZayaDecoderMLPLayer(nn.Module): ), self.input_norm, ), - previous=declare_attn(), - terminal=layer_id == config.num_hidden_layers - 1, ) self.entry_boundary = self.ffn_boundary diff --git a/python/sglang/srt/utils/common.py b/python/sglang/srt/utils/common.py index 9791ca5f2f..c950c0477a 100644 --- a/python/sglang/srt/utils/common.py +++ b/python/sglang/srt/utils/common.py @@ -1542,6 +1542,33 @@ def mark_end(name): time_infos[name].pretty_print() +# Set while a layer another pipeline stage holds is built, only for the stage +# boundaries it declares. +_building_neighbour_layer = False + + +def is_building_neighbour_layer() -> bool: + """Whether the layer under construction belongs to another pipeline stage + and is built here only to read the stage boundaries it declares. It is + built on the meta device and never loaded or run, so its constructor skips + the host and device resources a running layer needs: tables, streams, + engines and communicators.""" + return _building_neighbour_layer + + +@contextmanager +def building_neighbour_layer(): + """Build a pipeline neighbour layer: on the meta device, with + is_building_neighbour_layer() true.""" + global _building_neighbour_layer + outer, _building_neighbour_layer = _building_neighbour_layer, True + try: + with torch.device("meta"): + yield + finally: + _building_neighbour_layer = outer + + class LayerFn(Protocol): def __call__(self, idx: int, prefix: str) -> torch.nn.Module: ... @@ -1555,9 +1582,16 @@ def make_layers( return_tuple: bool = False, offloader_kwargs: Optional[Dict[str, Any]] = None, ) -> Tuple[torch.nn.Module, int, int]: - """Make a list of layers with the given layer function""" + """Make a list of layers with the given layer function. + + The local layers are built inside one layer stack, so layers that declare + stage boundaries connect in order without naming their neighbours. Across + a pipeline stage boundary the stack learns the neighbouring stage from the + layer itself, built again on the meta device. + """ # circular imports from sglang.srt.distributed import get_pp_indices + from sglang.srt.layers.layer_boundary.factories import layer_stack from sglang.srt.layers.utils import PPMissingLayer from sglang.srt.utils.offloader import get_offloader @@ -1571,20 +1605,30 @@ def make_layers( if pp_rank is not None and pp_size is not None else (0, num_hidden_layers) ) - modules = torch.nn.ModuleList( - [PPMissingLayer(return_tuple=return_tuple) for _ in range(start_layer)] - + get_offloader().wrap_modules( - ( - layer_fn(idx=idx, prefix=add_prefix(idx, prefix)) - for idx in range(start_layer, end_layer) - ), - **(offloader_kwargs or {}), + + def neighbour(idx): + return functools.partial( + _build_neighbour_layer, layer_fn, idx, add_prefix(idx, prefix) + ) + + with layer_stack( + previous_layers=[neighbour(idx) for idx in reversed(range(start_layer))], + next_layers=[neighbour(idx) for idx in range(end_layer, num_hidden_layers)], + ): + modules = torch.nn.ModuleList( + [PPMissingLayer(return_tuple=return_tuple) for _ in range(start_layer)] + + get_offloader().wrap_modules( + ( + layer_fn(idx=idx, prefix=add_prefix(idx, prefix)) + for idx in range(start_layer, end_layer) + ), + **(offloader_kwargs or {}), + ) + + [ + PPMissingLayer(return_tuple=return_tuple) + for _ in range(end_layer, num_hidden_layers) + ] ) - + [ - PPMissingLayer(return_tuple=return_tuple) - for _ in range(end_layer, num_hidden_layers) - ] - ) if pp_rank is None or pp_size is None: return modules return modules, start_layer, end_layer @@ -1613,6 +1657,21 @@ def make_pp_layers( ) +def _build_neighbour_layer(layer_fn: LayerFn, idx: int, prefix: str) -> None: + """Build a layer another pipeline stage holds, only for the stage + boundaries it declares (see building_neighbour_layer). RoPE modules it + adds to the shared cache are meta, so they are dropped again.""" + from sglang.srt.layers.rotary_embedding.factory import _ROPE_DICT + + cached = set(_ROPE_DICT) + try: + with building_neighbour_layer(): + layer_fn(idx=idx, prefix=prefix) + finally: + for key in set(_ROPE_DICT) - cached: + del _ROPE_DICT[key] + + def set_random_seed(seed: int) -> None: """Set the random seed for all libraries.""" random.seed(seed) diff --git a/python/sglang/test/boundary_fixtures.py b/python/sglang/test/boundary_fixtures.py index 680b1694ab..bf10734068 100644 --- a/python/sglang/test/boundary_fixtures.py +++ b/python/sglang/test/boundary_fixtures.py @@ -4,15 +4,36 @@ from types import SimpleNamespace from sglang.srt.layers.layer_boundary.construction import StagePlan from sglang.srt.layers.layer_boundary.factories import ( + append_stages, declare_attn, declare_ffn, - make_stages, + layer_stack, ) from sglang.srt.layers.layer_boundary.ops import keep_output from sglang.srt.layers.layer_boundary.residual.add_norm import PLAIN_RESIDUAL_OPS from sglang.srt.layers.layer_boundary.stage import StageBoundary +def build_stages(*stages, previous=None, terminal=False): + """Bind one sequence of stages in a layer stack of its own. + + Args: + *stages: Items of (declaration, norm) or (declaration, norm, options), + as for append_stages. + previous: Declaration of the stage before the first one, standing for + a layer the test does not build; None starts the stack. + terminal: Whether the last stage ends the model's layer stack; + otherwise the next layer's attention follows it. + """ + with layer_stack( + previous_layers=( + [lambda: append_stages((previous, None))] if previous is not None else [] + ), + next_layers=[] if terminal else [lambda: append_stages((declare_attn(), None))], + ): + return append_stages(*stages) + + def make_test_stages( *, attention_norm, @@ -42,7 +63,7 @@ def make_test_stages( output_transform=output, ) common = {key: options.pop(key) for key in ("fusions",) if key in options} - attn, ffn = make_stages( + attn, ffn = build_stages( ( attention, attention_norm, diff --git a/test/registered/unit/layer_boundary/test_append_stages.py b/test/registered/unit/layer_boundary/test_append_stages.py new file mode 100644 index 0000000000..c0b67bdb3c --- /dev/null +++ b/test/registered/unit/layer_boundary/test_append_stages.py @@ -0,0 +1,297 @@ +"""A layer stack connects the stages appended to it in order.""" + +import unittest +from unittest.mock import patch + +import test_declared_decoder_boundary as fixture +import torch +from torch import nn + +from sglang.srt.layers.layer_boundary import ( + ProducerReduction, + append_stages, + declare_attn, + declare_ffn, + layer_stack, +) +from sglang.srt.layers.rotary_embedding import factory as rope_factory +from sglang.srt.utils.common import is_building_neighbour_layer, make_layers +from sglang.test.ci.ci_register import register_cpu_ci +from sglang.test.test_utils import CustomTestCase + +register_cpu_ci(est_time=5, suite="base-a-test-cpu") + + +def layer(sparse=False): + """One decoder layer's stages: attention, then an FFN.""" + return append_stages( + (declare_attn(), fixture.Norm()), + (declare_ffn(sparse=sparse, next_layer_sparse=sparse), fixture.Norm()), + ) + + +def mixer(): + """A single-stage mixer whose exit depends on the stage after it.""" + return declare_attn( + reduction=ProducerReduction.EXIT_SCOPED, gathers_attn_tp_input=False + ) + + +class TestAppendStages(CustomTestCase): + def setUp(self): + self.planning = fixture.planning(fixture.parallel_of(attn_dp=1, attn_tp=2)) + self.planning.__enter__() + self.addCleanup(self.planning.__exit__, None, None, None) + + def test_only_the_stacks_last_stage_ends_it(self): + with layer_stack(): + stages = [s for _ in range(3) for s in layer()] + self.assertEqual([s.plan.terminal for s in stages], [False] * 5 + [True]) + self.assertEqual([s.plan.enters_stack for s in stages], [True] + [False] * 5) + + def test_a_one_layer_stack_starts_and_ends_on_that_layer(self): + # A NextN / MTP draft builds its single layer in a stack of its own. + with layer_stack(): + attention, ffn = layer() + self.assertTrue(attention.plan.enters_stack) + self.assertTrue(ffn.plan.terminal) + + def test_every_stage_binds_when_the_stack_closes(self): + with layer_stack(): + stages = [s for _ in range(3) for s in layer()] + # Declarations are usable at once; plans wait for the neighbours. + self.assertTrue(all(s.declaration is not None for s in stages)) + self.assertTrue(all(s.plan is None for s in stages)) + self.assertTrue(all(s.plan is not None for s in stages)) + + def test_appending_needs_an_open_stack(self): + with self.assertRaisesRegex(RuntimeError, "open layer stack"): + layer() + + def test_the_previous_layer_gives_the_first_stage_its_producer(self): + built = [] + + def previous_layer(): + built.append(None) + layer(sparse=True) + + with layer_stack(previous_layers=[previous_layer]): + first, _ = layer() + self.assertEqual(len(built), 1) + self.assertTrue(first.declaration.previous.sparse) + self.assertFalse(first.plan.enters_stack) + + def test_the_next_layer_gives_the_last_stage_its_consumer(self): + with layer_stack( + next_layers=[lambda: append_stages((declare_ffn(), fixture.Norm()))] + ): + (stage,) = append_stages((mixer(), fixture.Norm())) + self.assertFalse(stage.plan.terminal) + # It leaves its sum to the FFN that follows it, which completes it. + for edge in stage.plan.edges.values(): + self.assertTrue(edge.outgoing.produced.always_partial) + + def test_a_stack_without_stages_builds_no_neighbour(self): + def unexpected(): + self.fail("a neighbour was built for a stack without stages") + + with layer_stack(previous_layers=[unexpected], next_layers=[unexpected]): + pass + + def test_neighbours_are_built_after_the_stacks_own_layers(self): + order = [] + + def neighbour(): + order.append("neighbour") + layer() + + with layer_stack(previous_layers=[neighbour], next_layers=[neighbour]): + order.append("own") + layer() + self.assertEqual(order, ["own", "neighbour", "neighbour"]) + + def test_a_neighbour_without_stages_is_passed_over(self): + with layer_stack(previous_layers=[lambda: None, lambda: layer(sparse=True)]): + first, _ = layer() + self.assertTrue(first.declaration.previous.sparse) + with layer_stack(previous_layers=[lambda: None], next_layers=[lambda: None]): + first, last = layer() + # No earlier layer declares a stage: the stack starts here, and ends + # here when no later one does. + self.assertTrue(first.plan.enters_stack) + self.assertTrue(last.plan.terminal) + + def test_a_neighbour_is_read_without_joining_the_stack(self): + def previous_layer(): + # A layer with a dense FFN, then one with a MoE: the producer is + # the nearest stage, and neither joins this stack. + layer(sparse=False) + layer(sparse=True) + + with layer_stack(previous_layers=[previous_layer]): + stages = layer() + self.assertTrue(stages[0].declaration.previous.sparse) + self.assertIsNone(stages[0].declaration.previous.previous) + + def test_a_branch_neither_extends_the_stack_nor_waits(self): + with layer_stack(): + attention, moe = layer(sparse=True) + branch = append_stages( + (declare_ffn(), fixture.Norm()), + (declare_attn(), fixture.Norm()), + (declare_ffn(), fixture.Norm()), + prepared_from=moe.declaration, + ) + following = layer() + self.assertTrue(all(s.plan is not None for s in branch)) + # The branch reads the MoE's input; the next layer follows the main + # line's MoE, not the branch's dense FFN. + self.assertEqual(branch[0].declaration.prepared_from, moe.declaration) + self.assertTrue(following[0].declaration.previous.sparse) + + def test_a_nested_stack_leaves_the_outer_one_untouched(self): + with layer_stack(): + outer = layer() + with layer_stack(): + inner = layer() + later = layer() + self.assertTrue(inner[0].plan.enters_stack and inner[1].plan.terminal) + # The outer stack resumes from its own last stage. + self.assertFalse(outer[1].plan.terminal) + self.assertFalse(later[0].plan.enters_stack) + self.assertTrue(later[1].plan.terminal) + + def test_options_a_stage_does_not_take_are_rejected_when_appended(self): + with layer_stack(): + with self.assertRaisesRegex(TypeError, "ATTENTION stage options"): + append_stages((declare_attn(), fixture.Norm(), {"exit_rows": None})) + with self.assertRaisesRegex(TypeError, "FFN stage options"): + append_stages( + (declare_ffn(), fixture.Norm(), {"qkv_latent_func": None}) + ) + + def test_a_branch_reads_a_stage_of_its_own_stack(self): + with self.assertRaisesRegex(ValueError, "prepared_from"): + with layer_stack(): + layer() + append_stages( + (declare_ffn(), fixture.Norm()), prepared_from=declare_attn() + ) + + def test_a_branch_reads_the_stage_it_names_when_a_declaration_is_reused(self): + moe = declare_ffn(sparse=True, next_layer_sparse=True) + with layer_stack(): + _, first = append_stages( + (declare_attn(), fixture.Norm()), (moe, fixture.Norm()) + ) + _, second = append_stages( + (declare_attn(), fixture.Norm()), (moe, fixture.Norm()) + ) + (branch,) = append_stages( + (declare_ffn(), fixture.Norm()), prepared_from=first.declaration + ) + self.assertIsNot(first.declaration, second.declaration) + self.assertIs(branch.declaration.prepared_from, first.declaration) + + @unittest.skipUnless(hasattr(BaseException, "add_note"), "needs add_note") + def test_an_error_at_close_names_the_append_it_binds(self): + # The stack binds after the layers that appended to it have returned, + # so the traceback does not reach the layer; the note names it. + with self.assertRaises(ValueError) as caught: + with layer_stack(): + layer() + append_stages( + (declare_attn(), fixture.Norm()), + (declare_ffn(dense_tp_size=3), fixture.Norm()), + ) + layer() + (note,) = caught.exception.__notes__ + self.assertRegex( + note, r"test_append_stages\.py:\d+ \(append 1 of its layer stack\)" + ) + + +NUM_LAYERS = 4 + + +class TestMakeLayers(CustomTestCase): + """make_layers builds its layers in one stack, and on a pipeline stage + reads the stages the other stages' layers declare next to it.""" + + def setUp(self): + self.planning = fixture.planning(fixture.parallel_of(attn_dp=1, attn_tp=2)) + self.planning.__enter__() + self.addCleanup(self.planning.__exit__, None, None, None) + + def build(self, pp_rank=None, pp_size=None, *, declares=True): + """Build the layers; layer 1 has a MoE. Returns each layer_fn call as + (index, prefix, device, built as a neighbour) and the stages of the + layers built here.""" + calls, stages = [], {} + + def layer_fn(idx, prefix): + device = torch.empty(()).device.type + calls.append((idx, prefix, device, is_building_neighbour_layer())) + if declares: + stages.setdefault(idx, layer(sparse=idx == 1)) + return nn.Identity() + + make_layers( + NUM_LAYERS, layer_fn, pp_rank=pp_rank, pp_size=pp_size, prefix="layers" + ) + return calls, stages + + def test_the_layers_form_one_stack(self): + _, stages = self.build() + stages = [s for idx in range(NUM_LAYERS) for s in stages[idx]] + self.assertEqual([s.plan.terminal for s in stages], [False] * 7 + [True]) + self.assertEqual([s.plan.enters_stack for s in stages], [True] + [False] * 7) + + def test_a_pipeline_stage_builds_its_neighbours_on_meta_after_its_layers(self): + calls, _ = self.build(pp_rank=0, pp_size=2) + self.assertEqual( + calls, + [ + (0, "layers.0", "cpu", False), + (1, "layers.1", "cpu", False), + (2, "layers.2", "meta", True), + ], + ) + calls, _ = self.build(pp_rank=1, pp_size=2) + self.assertEqual( + calls, + [ + (2, "layers.2", "cpu", False), + (3, "layers.3", "cpu", False), + (1, "layers.1", "meta", True), + ], + ) + + def test_the_neighbours_connect_the_stack_across_pipeline_stages(self): + _, first = self.build(pp_rank=0, pp_size=2) + self.assertTrue(first[0][0].plan.enters_stack) + self.assertFalse(first[1][1].plan.terminal) + _, last = self.build(pp_rank=1, pp_size=2) + # Layer 2's attention follows layer 1's MoE, on the other stage. + self.assertTrue(last[2][0].declaration.previous.sparse) + self.assertFalse(last[2][0].plan.enters_stack) + self.assertTrue(last[3][1].plan.terminal) + + def test_layers_without_stages_build_no_neighbour(self): + calls, _ = self.build(pp_rank=1, pp_size=2, declares=False) + self.assertEqual([call[0] for call in calls], [2, 3]) + + def test_a_neighbour_s_rope_modules_leave_the_cache(self): + def layer_fn(idx, prefix): + rope_factory._ROPE_DICT[("test", idx)] = None + layer() + return nn.Identity() + + with patch.dict(rope_factory._ROPE_DICT): + make_layers(NUM_LAYERS, layer_fn, pp_rank=0, pp_size=2) + added = {key for key in rope_factory._ROPE_DICT if key[0] == "test"} + self.assertEqual(added, {("test", 0), ("test", 1)}) + + +if __name__ == "__main__": + unittest.main() diff --git a/test/registered/unit/layer_boundary/test_aux_hidden_states.py b/test/registered/unit/layer_boundary/test_aux_hidden_states.py index eeda55a440..f174945baf 100644 --- a/test/registered/unit/layer_boundary/test_aux_hidden_states.py +++ b/test/registered/unit/layer_boundary/test_aux_hidden_states.py @@ -185,9 +185,9 @@ class TestBoundCaptureOwnership(CustomTestCase): SumGroup, declare_attn, declare_ffn, - make_stages, ) from sglang.srt.model_executor.forward_batch_info import ForwardMode + from sglang.test.boundary_fixtures import build_stages from sglang.test.communicator_patch import patch_communicator parallel = fixture.parallel_of(attn_dp=1, attn_tp=2) @@ -244,7 +244,7 @@ class TestBoundCaptureOwnership(CustomTestCase): side_effect=fused_kernel, ), ): - attn, ffn = make_stages( + attn, ffn = build_stages( (declare_attn(), fixture.Norm()), (declare_ffn(), norm, {"fusions": custom_fusions}), previous=declare_ffn(), diff --git a/test/registered/unit/layer_boundary/test_boundary_output_contracts.py b/test/registered/unit/layer_boundary/test_boundary_output_contracts.py index 596c7ec365..63132d483d 100644 --- a/test/registered/unit/layer_boundary/test_boundary_output_contracts.py +++ b/test/registered/unit/layer_boundary/test_boundary_output_contracts.py @@ -13,9 +13,6 @@ from sglang.srt.layers.layer_boundary import ( declare_ffn, ) from sglang.srt.layers.layer_boundary import exit as exits -from sglang.srt.layers.layer_boundary import ( - make_stages, -) from sglang.srt.layers.layer_boundary import stage as stages from sglang.srt.layers.layer_boundary.contracts import BatchVariant from sglang.srt.layers.layer_boundary.fusions.cutedsl import CuteDSLFusion @@ -25,6 +22,7 @@ from sglang.srt.layers.layer_boundary.residual import batch from sglang.srt.layers.layer_boundary.residual.add_norm import REPLACE_AT_EXIT from sglang.srt.layers.layer_boundary.residual.stream import ResidualStream from sglang.srt.model_executor.forward_batch_info import ForwardMode +from sglang.test.boundary_fixtures import build_stages from sglang.test.ci.ci_register import register_cpu_ci from sglang.test.communicator_patch import patch_communicator @@ -59,7 +57,7 @@ class TestBoundaryIntegrations(unittest.TestCase): attn_dp=1, attn_tp=2, enable_attn_tp_input_scattered=True ) with fixture.planning(parallel): - attn, _ = make_stages( + attn, _ = build_stages( (declare_attn(), fixture.Norm()), (declare_ffn(), fixture.Norm()), previous=declare_ffn(), @@ -84,7 +82,7 @@ class TestBoundaryIntegrations(unittest.TestCase): patch_communicator("tp_reduce_scatter", reduce_scatter), ): # Rebind the path with the numerical collective stand-in. - attn, _ = make_stages( + attn, _ = build_stages( (declare_attn(), fixture.Norm()), (declare_ffn(), fixture.Norm()), previous=declare_ffn(), @@ -124,7 +122,7 @@ class TestBoundaryIntegrations(unittest.TestCase): return_value=parallel, ), ): - _, ffn = make_stages( + _, ffn = build_stages( (declare_attn(), fixture.Norm()), (declare_ffn(sparse=True), fixture.Norm(), {"fusions": fusion}), terminal=True, @@ -207,7 +205,7 @@ class TestBoundaryIntegrations(unittest.TestCase): fixture.planning(fixture.parallel_of(attn_dp=attn_dp, attn_tp=2)), ): for sparse in (False, True): - _, ffn = make_stages( + _, ffn = build_stages( (declare_attn(), fixture.Norm()), ( declare_ffn(sparse=sparse, update=REPLACE_AT_EXIT), @@ -227,7 +225,7 @@ class TestBoundaryIntegrations(unittest.TestCase): patch.object(stages, "sum_output", summed), ): for update in (PLAIN_ADD, REPLACE_AT_EXIT): - _, ffn = make_stages( + _, ffn = build_stages( (declare_attn(), fixture.Norm()), (declare_ffn(update=update), fixture.Norm()), ) diff --git a/test/registered/unit/layer_boundary/test_draft_layer_declarations.py b/test/registered/unit/layer_boundary/test_draft_layer_declarations.py new file mode 100644 index 0000000000..f71f252b68 --- /dev/null +++ b/test/registered/unit/layer_boundary/test_draft_layer_declarations.py @@ -0,0 +1,142 @@ +import importlib +import sys +import types +import unittest +from types import ModuleType +from unittest.mock import MagicMock, patch + +import torch + +from sglang.test.ci.ci_register import register_cpu_ci +from sglang.test.test_utils import CustomTestCase + +register_cpu_ci(est_time=15, suite="base-a-test-cpu") + +NUM_LAYERS = 4 + + +def import_model(name): + if importlib.util.find_spec("vllm") is not None: + return importlib.import_module(f"sglang.srt.models.{name}") + # CPU CI omits vLLM; the Bailing modules import an AWQ kernel from it. + vllm = ModuleType("vllm") + vllm.__path__ = [] + custom_ops = ModuleType("vllm._custom_ops") + custom_ops.awq_dequantize = MagicMock() + with patch.dict(sys.modules, {"vllm": vllm, "vllm._custom_ops": custom_ops}): + return importlib.import_module(f"sglang.srt.models.{name}") + + +class StubModule(torch.nn.Module): + def __getattr__(self, name): + try: + return super().__getattr__(name) + except AttributeError: + return MagicMock() + + +def stub(*args, **kwargs): + return StubModule() + + +PARALLEL = types.SimpleNamespace( + tp_size=1, + tp_rank=0, + attn_tp_size=1, + attn_tp_rank=0, + attn_dp_size=1, + attn_dp_rank=0, + moe_ep_size=1, + moe_tp_size=1, + moe_dp_size=1, +) + + +def bailing_hybrid_config(num_layers): + from sglang.srt.configs.bailing_hybrid import BailingHybridConfig + + return BailingHybridConfig(num_hidden_layers=num_layers, attention_type=1) + + +def bailing_moe_config(num_layers): + from sglang.srt.configs.bailing_moe_v2 import BailingMoeV2Config + + return BailingMoeV2Config(num_hidden_layers=num_layers) + + +# name: (module, decoder layer class, submodules to stub) +CASES = { + "bailing_moe_linear": ( + "bailing_moe_linear", + "BailingMoELinearDecoderLayer", + [ + "BailingMLP", + "BailingMoE", + "BailingMoEAttention", + "BailingMoELinearAttention", + "DsV3MLA", + ], + ), + "bailing_moe": ( + "bailing_moe", + "BailingMoEBlock", + ["BailingMoEAttention", "BailingMoEMLP", "BailingMoESparseMoeBlock"], + ), +} + + +def build_draft(case, config): + """Construct a NextN draft layer with stubbed compute; return its FFN + declaration and the compute submodules it built.""" + module_name, class_name, stubs = CASES[case] + module = import_model(module_name) + append_stages = MagicMock(return_value=(MagicMock(), MagicMock())) + built = [] + + def recording_stub(name): + def make(*args, **kwargs): + built.append(name) + return StubModule() + + return make + + from sglang.srt.layers.layer_boundary import declare_attn, declare_ffn + + patches = dict( + declare_attn=declare_attn, + declare_ffn=declare_ffn, + append_stages=append_stages, + ) + patches.update({name: recording_stub(name) for name in stubs}) + if hasattr(module, "get_parallel"): + patches["get_parallel"] = lambda: PARALLEL + with patch.multiple(module, **patches): + getattr(module, class_name)(config, layer_id=0, is_nextn=True) + append_stages.assert_called_once() + return append_stages.call_args.args[1][0], built + + +class TestDraftLayerDeclarations(CustomTestCase): + def test_bailing_draft_layer_builds_an_moe(self): + """The Bailing V2 NextN checkpoint holds expert weights, so the NextN + layer builds the sparse MoE block, also when the model's first layers + are dense.""" + config = bailing_moe_config(NUM_LAYERS) + config.first_k_dense_replace = 1 + ffn, built = build_draft("bailing_moe", config) + self.assertIn("BailingMoESparseMoeBlock", built) + self.assertNotIn("BailingMoEMLP", built) + self.assertTrue(ffn.sparse) + + def test_bailing_hybrid_draft_layer_is_planned_as_sparse(self): + """The Bailing hybrid NextN layer builds an MoE, so it declares a sparse + FFN, also when the model's first layers are dense.""" + config = bailing_hybrid_config(NUM_LAYERS) + config.first_k_dense_replace = 1 + ffn, built = build_draft("bailing_moe_linear", config) + self.assertIn("BailingMoE", built) + self.assertTrue(ffn.sparse) + + +if __name__ == "__main__": + unittest.main() diff --git a/test/registered/unit/layer_boundary/test_independent_stage_construction.py b/test/registered/unit/layer_boundary/test_independent_stage_construction.py index 0cb7d2764c..459ae93368 100644 --- a/test/registered/unit/layer_boundary/test_independent_stage_construction.py +++ b/test/registered/unit/layer_boundary/test_independent_stage_construction.py @@ -2,7 +2,6 @@ import itertools import unittest -from dataclasses import replace from types import SimpleNamespace from unittest.mock import patch @@ -12,21 +11,27 @@ import torch from sglang.srt.layers.layer_boundary import ( ProducerReduction, StageKind, + append_stages, declare_attn, declare_ffn, - make_attn_stage, - make_ffn_stage, - make_stages, + layer_stack, ) from sglang.srt.layers.layer_boundary.contracts import BatchVariant from sglang.srt.layers.layer_boundary.residual import batch from sglang.srt.model_executor.forward_batch_info import ForwardMode +from sglang.test.boundary_fixtures import build_stages from sglang.test.ci.ci_register import register_cpu_ci from sglang.test.test_utils import CustomTestCase register_cpu_ci(est_time=5, suite="base-a-test-cpu") +def next_layer(): + """The next layer's attention, which a stack that does not end the + model hands its output to.""" + append_stages((declare_attn(), fixture.Norm())) + + class TestIndependentStageConstruction(CustomTestCase): def test_active_variant_without_a_bound_path_never_falls_back(self): from sglang.srt.layers.layer_boundary import construction @@ -78,7 +83,7 @@ class TestIndependentStageConstruction(CustomTestCase): declare_attn() if kind is StageKind.ATTENTION else declare_ffn() for kind in kinds ] - boundaries = make_stages( + boundaries = build_stages( *((d, fixture.Norm()) for d in declarations), terminal=True ) self.assertEqual(len(boundaries), count) @@ -100,56 +105,40 @@ class TestIndependentStageConstruction(CustomTestCase): batch.snapshot(hidden, fb), torch.full((2, 4), float(3**count)) ) - def test_sequence_and_independent_factories_agree_on_terminal_placement(self): + def test_grouping_stages_into_appends_binds_the_same_plans(self): + # A stage binds once its producer and consumer are both known, so one + # append per stage binds what one append of the whole sequence does. for dp, tp, terminal in itertools.product((1, 2), (1, 2), (False, True)): for kinds in itertools.product(StageKind, repeat=3): with ( self.subTest(dp=dp, tp=tp, terminal=terminal, kinds=kinds), fixture.planning(fixture.parallel_of(attn_dp=dp, attn_tp=tp)), ): - declarations = [ - declare_attn() if kind is StageKind.ATTENTION else declare_ffn() - for kind in kinds - ] - sequence = make_stages( - *( - (declaration, fixture.Norm()) - for declaration in declarations - ), - terminal=terminal, - ) - linked = [] - for index, declaration in enumerate(declarations): - linked.append( - replace( - declaration, - previous=linked[-1] if linked else None, - terminal=terminal and index == len(declarations) - 1, + + def stages(): + return [ + ( + ( + declare_attn() + if kind is StageKind.ATTENTION + else declare_ffn() + ), + fixture.Norm(), ) - ) - for index, (declaration, combined) in enumerate( - zip(linked, sequence) - ): - make = ( - make_attn_stage - if declaration.kind is StageKind.ATTENTION - else make_ffn_stage - ) - independent = make( - declaration=declaration, - norm=fixture.Norm(), - following=linked[index + 1] - if index + 1 < len(linked) - else None, - ) - self.assertEqual(combined.plan.terminal, declaration.terminal) - self.assertEqual( - independent.plan.terminal, declaration.terminal - ) - self.assertEqual(combined.plan.edges, independent.plan.edges) + for kind in kinds + ] + + together = build_stages(*stages(), terminal=terminal) + with layer_stack(next_layers=[] if terminal else [next_layer]): + apart = [b for stage in stages() for b in append_stages(stage)] + for index, (combined, single) in enumerate(zip(together, apart)): + ends = terminal and index == len(kinds) - 1 + self.assertEqual(combined.plan.terminal, ends) + self.assertEqual(single.plan.terminal, ends) + self.assertEqual(combined.plan.edges, single.plan.edges) for variant, steps in combined.plan.paths.items(): self.assertEqual( - steps.output, independent.plan.paths[variant].output + steps.output, single.plan.paths[variant].output ) def test_sequence_branch_keeps_source_lineage(self): @@ -164,26 +153,32 @@ class TestIndependentStageConstruction(CustomTestCase): ) ), ): - _, source = make_stages( - (declare_attn(), fixture.Norm()), - (declare_ffn(sparse=True), fixture.Norm()), - previous=declare_ffn(sparse=True), + + def source_layer(): + return append_stages( + (declare_attn(), fixture.Norm()), + (declare_ffn(sparse=True), fixture.Norm()), + )[1] + + def previous_layer(): + append_stages((declare_ffn(sparse=True), fixture.Norm())) + + neighbours = dict( + previous_layers=[previous_layer], next_layers=[next_layer] ) - before = dict(source.plan.edges) - first, attn, last = make_stages( - ( - declare_ffn(), - fixture.Norm(), - ), - (declare_attn(), fixture.Norm()), - ( - declare_ffn(), - fixture.Norm(), - ), - prepared_from=source.declaration, - ) - self.assertEqual(source.plan.edges, before) - self.assertIs(first.declaration.prepared_from, source.declaration) + with layer_stack(**neighbours): + alone = source_layer() + with layer_stack(**neighbours): + source = source_layer() + first, attn, last = append_stages( + (declare_ffn(), fixture.Norm()), + (declare_attn(), fixture.Norm()), + (declare_ffn(), fixture.Norm()), + prepared_from=source.declaration, + ) + # The branch leaves the stage it reads from as it was. + self.assertEqual(source.plan.edges, alone.plan.edges) + self.assertEqual(first.declaration.prepared_from, source.declaration) self.assertIs(attn.declaration.previous, first.declaration) self.assertIs(last.declaration.previous, attn.declaration) for v, edge in first.plan.edges.items(): @@ -195,9 +190,9 @@ class TestIndependentStageConstruction(CustomTestCase): def test_an_ffn_without_a_fusion_object_may_defer_or_reduce_scatter(self): with fixture.planning(fixture.parallel_of(attn_dp=1, attn_tp=2)): - boundary = make_ffn_stage( - declaration=declare_ffn(previous=declare_attn(), sparse=True), - norm=fixture.Norm(), + _, boundary = build_stages( + (declare_attn(), fixture.Norm()), + (declare_ffn(sparse=True), fixture.Norm()), ) self.assertIsNone(boundary.plan.fusions) produced = boundary.plan.paths[BatchVariant.ORDINARY].output @@ -215,7 +210,7 @@ class TestIndependentStageConstruction(CustomTestCase): self.subTest(kind=following.kind), fixture.planning(fixture.parallel_of(attn_dp=1, attn_tp=2)), ): - mixer, consumer = make_stages( + mixer, consumer = build_stages( ( declare_attn( reduction=ProducerReduction.EXIT_SCOPED, diff --git a/test/registered/unit/layer_boundary/test_nemotron_h_aux_capture.py b/test/registered/unit/layer_boundary/test_nemotron_h_aux_capture.py index c93f256b81..6b10242aee 100644 --- a/test/registered/unit/layer_boundary/test_nemotron_h_aux_capture.py +++ b/test/registered/unit/layer_boundary/test_nemotron_h_aux_capture.py @@ -4,6 +4,7 @@ from types import SimpleNamespace import torch from torch import nn +from sglang.srt.layers.layer_boundary import layer_stack from sglang.srt.model_executor.forward_batch_info import ForwardMode from sglang.srt.models import nemotron_h as model from sglang.srt.runtime_context import get_context, get_flags, get_parallel @@ -48,16 +49,17 @@ def _build(pattern, tp, capture): instance.norm_f = _Norm() instance.layers_to_capture = set(range(len(pattern) + 1)) if capture else set() layers = [] - for i, kind in enumerate(pattern): - cls = model.ALL_DECODER_LAYER_TYPES[kind] - layer = cls.__new__(cls) - nn.Module.__init__(layer) - layer.norm = _Norm() - layer._init_stage_boundary(config, i) - layer.mixer = _Mixer(0.5 if kind in "M*" else 0.25, tp) - if kind == "M": - layer._forward_mamba = lambda h, batch, mixer=layer.mixer: mixer(h) - layers.append(layer) + with layer_stack(): + for i, kind in enumerate(pattern): + cls = model.ALL_DECODER_LAYER_TYPES[kind] + layer = cls.__new__(cls) + nn.Module.__init__(layer) + layer.norm = _Norm() + layer._init_stage_boundary(config, i) + layer.mixer = _Mixer(0.5 if kind in "M*" else 0.25, tp) + if kind == "M": + layer._forward_mamba = lambda h, batch, mixer=layer.mixer: mixer(h) + layers.append(layer) instance.layers = nn.ModuleList(layers) return instance diff --git a/test/registered/unit/layer_boundary/test_nemotron_h_mtp_reduction.py b/test/registered/unit/layer_boundary/test_nemotron_h_mtp_reduction.py index 1c00e1a28b..b48f80ffa7 100644 --- a/test/registered/unit/layer_boundary/test_nemotron_h_mtp_reduction.py +++ b/test/registered/unit/layer_boundary/test_nemotron_h_mtp_reduction.py @@ -5,11 +5,13 @@ from unittest.mock import Mock import torch from torch import nn +from sglang.srt.layers.layer_boundary import layer_stack from sglang.srt.layers.layer_boundary.layout import SumGroup from sglang.srt.layers.layer_boundary.residual.add_norm import PLAIN_ADD from sglang.srt.layers.layer_boundary.residual.stream import ResidualStream from sglang.srt.model_executor.forward_batch_info import ForwardMode from sglang.srt.models import nemotron_h_mtp +from sglang.srt.models.nemotron_h_utils import make_stage_boundary from sglang.srt.runtime_context import get_context, get_flags, get_parallel from sglang.test.ci.ci_register import register_cpu_ci from sglang.test.communicator_patch import patch_communicator @@ -82,9 +84,18 @@ class TestNemotronMTPReduction(CustomTestCase): # A MoE stand-in leaving its output's sum to the exit. layer.mixer = _Partial(tp) layer.norm = _Norm() - layer._init_stage_boundary( - SimpleNamespace(hybrid_override_pattern="*E"), 1 - ) + # The MoE layer follows the pattern's attention layer, + # built elsewhere: the stack reads it as a neighbour. + with layer_stack( + previous_layers=[ + lambda: make_stage_boundary( + _Norm(), pattern="*E", layer_idx=0 + ) + ] + ): + layer._init_stage_boundary( + SimpleNamespace(hybrid_override_pattern="*E"), 1 + ) partial = torch.tensor([[1.0, 2.0], [3.0, 4.0]]) residual = torch.tensor([[7.0, 3.0], [5.0, 9.0]]) expected = partial * tp + residual diff --git a/test/registered/unit/layer_boundary/test_nemotron_h_stages.py b/test/registered/unit/layer_boundary/test_nemotron_h_stages.py index 44dfbdb40a..6930d9ca81 100644 --- a/test/registered/unit/layer_boundary/test_nemotron_h_stages.py +++ b/test/registered/unit/layer_boundary/test_nemotron_h_stages.py @@ -28,7 +28,7 @@ register_cpu_ci(est_time=3, suite="base-a-test-cpu") def layer_stage(pattern, index): from sglang.srt.layers.layer_boundary.construction import BatchVariant - from sglang.srt.layers.layer_boundary.factories import _connections + from sglang.srt.layers.layer_boundary.factories import _connect, _incoming previous = utils._declaration(pattern, index - 1) if index else None declaration = replace( @@ -41,7 +41,8 @@ def layer_stage(pattern, index): if index + 1 < len(pattern) else None ) - incoming, outgoing = _connections(declaration, following) + incoming = _incoming(declaration) + outgoing = _connect(declaration, following, residual_from=incoming) return SimpleNamespace( kind=declaration.kind, edges=( diff --git a/test/registered/unit/layer_boundary/test_qwen4_exp_ple_rows.py b/test/registered/unit/layer_boundary/test_qwen4_exp_ple_rows.py index 7ed01177e1..b11419434a 100644 --- a/test/registered/unit/layer_boundary/test_qwen4_exp_ple_rows.py +++ b/test/registered/unit/layer_boundary/test_qwen4_exp_ple_rows.py @@ -5,6 +5,7 @@ from types import SimpleNamespace import test_declared_decoder_boundary as fixture +from sglang.srt.layers.layer_boundary import layer_stack from sglang.srt.layers.layer_boundary.contracts import BatchVariant from sglang.srt.layers.layer_boundary.layout import TokenAxis from sglang.srt.layers.layer_boundary.residual.gated import GatedResidualState @@ -38,7 +39,10 @@ class TestQwen4ExpPleRows(CustomTestCase): # rank's slice of the rows between layers; the PLE embedding is # computed for every row, so its layer must read the full rows. config = SimpleNamespace(num_hidden_layers=4, ple_layer_ids=[3]) - with fixture.planning(fixture.parallel_of(attn_dp=1, attn_tp=2), a2a=True): + with ( + fixture.planning(fixture.parallel_of(attn_dp=1, attn_tp=2), a2a=True), + layer_stack(), + ): layers = [ _build_qwen4_exp_stages( _residual_ops(), sparse=True, layer_id=layer_id, config=config diff --git a/test/registered/unit/layer_boundary/test_stage_boundaries_built_once.py b/test/registered/unit/layer_boundary/test_stage_boundaries_built_once.py index 9e9217bef3..4c188535db 100644 --- a/test/registered/unit/layer_boundary/test_stage_boundaries_built_once.py +++ b/test/registered/unit/layer_boundary/test_stage_boundaries_built_once.py @@ -10,7 +10,7 @@ from sglang.test.test_utils import CustomTestCase register_cpu_ci(est_time=5, suite="base-a-test-cpu") -_BUILDERS = {"make_stages", "make_attn_stage", "make_ffn_stage"} +_BUILDERS = {"append_stages"} def _called(node): @@ -146,7 +146,7 @@ class TestStageBoundariesBuiltOnce(CustomTestCase): " if build_stages:\n" " self.stages = self._build_stages()\n" " def _build_stages(self):\n" - " return make_stages((declare_attn(), norm), (declare_ffn(), norm))\n" + " return append_stages((declare_attn(), norm), (declare_ffn(), norm))\n" "class Twice(Base):\n" " def __init__(self):\n" " super().__init__()\n" diff --git a/test/registered/unit/layer_boundary/test_stage_layers_built_in_a_stack.py b/test/registered/unit/layer_boundary/test_stage_layers_built_in_a_stack.py new file mode 100644 index 0000000000..b86f54d8e2 --- /dev/null +++ b/test/registered/unit/layer_boundary/test_stage_layers_built_in_a_stack.py @@ -0,0 +1,302 @@ +"""Layers that append stages are built inside an open layer stack.""" + +import ast +import unittest +from pathlib import Path + +import sglang.srt.models +from sglang.test.ci.ci_register import register_cpu_ci +from sglang.test.test_utils import CustomTestCase + +register_cpu_ci(est_time=10, suite="base-a-test-cpu") + +_PACKAGE = ("sglang", "srt", "models") +# Calls that build layers inside a stack of their own. +_STACK_BUILDERS = ("make_layers", "make_pp_layers") + + +def _call_name(func): + return func.attr if isinstance(func, ast.Attribute) else getattr(func, "id", None) + + +def _calls(node): + return { + _call_name(call.func) for call in ast.walk(node) if isinstance(call, ast.Call) + } + + +class _Classes: + """The package's classes, resolved the way each file names them: its own + class of that name, else the class its import names. A name resolved + neither way stands for every class of that name, which fails closed.""" + + def __init__(self, trees, root): + self.by_name = {} + modules = {} + for path, tree in trees.items(): + parts = path.relative_to(root).with_suffix("").parts + if parts[-1] == "__init__": + parts = parts[:-1] + modules[".".join(_PACKAGE + parts)] = path + for node in ast.walk(tree): + if isinstance(node, ast.ClassDef): + self.by_name.setdefault(node.name, {})[path] = node + self.imports = {} + for path, tree in trees.items(): + package = path.relative_to(root).parent.parts + table = {} + for node in ast.walk(tree): + if not isinstance(node, ast.ImportFrom): + continue + module = node.module + if node.level: + up = node.level - 1 + base = ( + _PACKAGE + package[: len(package) - up] + if up <= len(package) + else None + ) + module = base and ".".join(base + ((module,) if module else ())) + for alias in node.names: + table[alias.asname or alias.name] = ( + modules.get(module), + alias.name, + ) + self.imports[path] = table + + def resolve(self, path, name): + defined = self.by_name.get(name, {}) + if path in defined: + return {(path, name)} + source, original = self.imports[path].get(name, (None, name)) + defined = self.by_name.get(original, {}) + if source in defined: + return {(source, original)} + return {(p, original) for p in defined} + + +def _stage_classes(trees, classes): + """Classes whose construction appends stages: a method reaches + append_stages, directly or through a module-level helper, or a base class + does. Derived from the code, so a new model joins without being listed.""" + helpers, functions = {"append_stages"}, {} + for tree in trees.values(): + for node in tree.body: + if isinstance(node, ast.FunctionDef) and not node.name.startswith("__"): + functions.setdefault(node.name, []).append(node) + changed = True + while changed: + changed = False + for name, nodes in functions.items(): + if name not in helpers and any(_calls(fn) & helpers for fn in nodes): + helpers.add(name) + changed = True + stage = set() + changed = True + while changed: + changed = False + for name, defs in classes.by_name.items(): + for path, node in defs.items(): + if (path, name) in stage: + continue + bases = set() + for base in node.bases: + bases |= classes.resolve(path, ast.unparse(base).split(".")[-1]) + if bases & stage or any( + isinstance(m, ast.FunctionDef) and _calls(m) & helpers + for m in node.body + ): + stage.add((path, name)) + changed = True + return stage + + +class _Holders: + """Which expressions can hold a stage class: the class itself, a + collection of them, and the names they are bound to. Within a file that is + a variable or attribute assigned one; across files it is a keyword or + parameter given one, since a class handed to a shared model is called + under the parameter's name in another file.""" + + def __init__(self, trees, classes, stage): + self.classes, self.stage = classes, stage + self.names = {path: set() for path in trees} + self.parameters = set() + changed = True + while changed: + changed = False + for path, tree in trees.items(): + for node in ast.walk(tree): + for target, value in self._bindings(node): + if value is None or not self.holds(path, value): + continue + names = self.parameters if target[0] else self.names[path] + if target[1] not in names: + names.add(target[1]) + changed = True + + @staticmethod + def _bindings(node): + """(is_parameter, name) targets and the values bound to them.""" + if isinstance(node, (ast.Assign, ast.AnnAssign)): + targets = node.targets if isinstance(node, ast.Assign) else [node.target] + for target in targets: + for t in target.elts if isinstance(target, ast.Tuple) else [target]: + if isinstance(t, (ast.Name, ast.Attribute)): + yield (False, _call_name(t)), node.value + elif isinstance(node, ast.keyword) and node.arg: + yield (True, node.arg), node.value + elif isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): + args = node.args + positional = args.posonlyargs + args.args + defaults = [None] * (len(positional) - len(args.defaults)) + args.defaults + for arg, default in zip( + positional + args.kwonlyargs, defaults + args.kw_defaults + ): + yield (True, arg.arg), default + + def holds(self, path, expr): + if isinstance(expr, ast.Name): + return ( + expr.id in self.names[path] + or expr.id in self.parameters + or bool(self.classes.resolve(path, expr.id) & self.stage) + ) + if isinstance(expr, ast.Attribute): + return expr.attr in self.names[path] + if isinstance(expr, (ast.List, ast.Tuple, ast.Set)): + return any(self.holds(path, e) for e in expr.elts) + if isinstance(expr, ast.Dict): + return any(self.holds(path, v) for v in expr.values) + if isinstance(expr, ast.Subscript): + return self.holds(path, expr.value) + if isinstance(expr, ast.IfExp): + return self.holds(path, expr.body) or self.holds(path, expr.orelse) + if isinstance(expr, ast.BoolOp): + return any(self.holds(path, v) for v in expr.values) + return False + + +def _opens_a_stack(node): + if isinstance(node, ast.With): + return any("layer_stack" in ast.unparse(i.context_expr) for i in node.items) + return isinstance(node, ast.Call) and _call_name(node.func) in _STACK_BUILDERS + + +def _unstacked_constructions(trees, root): + classes = _Classes(trees, root) + stage = _stage_classes(trees, classes) + holders = _Holders(trees, classes, stage) + found = [] + for path, tree in trees.items(): + parents = {c: n for n in ast.walk(tree) for c in ast.iter_child_nodes(n)} + stacked = [n for n in ast.walk(tree) if _opens_a_stack(n)] + # A named factory used inside a stack (e.g. handed to make_layers). + factories = { + x.id for s in stacked for x in ast.walk(s) if isinstance(x, ast.Name) + } + for node in ast.walk(tree): + if not (isinstance(node, ast.Call) and holders.holds(path, node.func)): + continue + chain, x = [], node + while x in parents: + x = parents[x] + chain.append(x) + if any(_opens_a_stack(c) for c in chain): + continue + enclosing = next( + (c for c in chain if isinstance(c, (ast.FunctionDef, ast.Lambda))), + None, + ) + if ( + isinstance(enclosing, ast.FunctionDef) + and enclosing.name != "__init__" + and enclosing.name in factories + ): + continue + found.append((str(path.relative_to(root)), node.lineno, node.func)) + return stage, [ + f"{p}:{line} {ast.unparse(func)}" + for p, line, func in sorted(found, key=lambda f: f[:2]) + ] + + +class TestStageLayersBuiltInAStack(CustomTestCase): + def test_every_stage_layer_is_built_inside_a_layer_stack(self): + # append_stages raises outside a stack, but only when that model is + # built: a draft or wrapper model nobody constructs in CI would fail + # only in production. This checks every construction site instead. + (root,) = map(Path, sglang.srt.models.__path__) + trees = {p: ast.parse(p.read_text()) for p in sorted(root.rglob("*.py"))} + stage, found = _unstacked_constructions(trees, root) + self.assertGreater(len(stage), 40) + self.assertEqual(found, []) + + def test_a_layer_built_outside_a_stack_is_reported(self): + draft = ( + "def make_stage_boundary(norm):\n" + " return append_stages((decl, norm))\n" + "class Layer:\n" + " def __init__(self):\n" + " self.boundary = make_stage_boundary(None)\n" + "class DraftLayer(Layer):\n" + " pass\n" + "class Draft:\n" + " def __init__(self):\n" + " self.decoder = DraftLayer()\n" + "class Stacked:\n" + " def __init__(self):\n" + " with layer_stack():\n" + " self.decoder = DraftLayer()\n" + ) + # A class of the same name that appends nothing, built by its own file. + other = ( + "class Layer:\n" + " def __init__(self):\n" + " pass\n" + "class Model:\n" + " def __init__(self):\n" + " self.layer = Layer()\n" + ) + # A class reached by a lookup, or handed to a shared model under a + # parameter's name. + lookup = ( + "from sglang.srt.models.draft import DraftLayer, Layer\n" + "LAYERS = {'draft': DraftLayer}\n" + "class Shared:\n" + " def __init__(self, layer_type=None):\n" + " self.layers = [layer_type() for _ in range(2)]\n" + "class Variant(Shared):\n" + " def __init__(self):\n" + " super().__init__(layer_type=Layer)\n" + "class Hybrid:\n" + " def __init__(self):\n" + " def get_layer(idx, prefix):\n" + " return LAYERS['draft']()\n" + " self.layers = make_layers(2, get_layer)\n" + " layer_class = LAYERS['draft']\n" + " self.extra = layer_class()\n" + ) + root = Path("models") + trees = { + root / "draft.py": ast.parse(draft), + root / "z.py": ast.parse(other), + root / "lookup.py": ast.parse(lookup), + } + stage, found = _unstacked_constructions(trees, root) + # Reached through a helper, inherited by a subclass, and not confused + # with a same-named class elsewhere. + self.assertEqual({name for _, name in stage}, {"Layer", "DraftLayer"}) + self.assertNotIn((root / "z.py", "Layer"), stage) + self.assertEqual( + found, + [ + "draft.py:10 DraftLayer", + "lookup.py:5 layer_type", + "lookup.py:15 layer_class", + ], + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/test/registered/unit/layer_boundary/test_stage_producers_leave_sums.py b/test/registered/unit/layer_boundary/test_stage_producers_leave_sums.py index c31c729488..4dd5ad0f03 100644 --- a/test/registered/unit/layer_boundary/test_stage_producers_leave_sums.py +++ b/test/registered/unit/layer_boundary/test_stage_producers_leave_sums.py @@ -34,7 +34,7 @@ def _is_sum(name) -> bool: _BUILDS_STAGES = ( - "make_stages(", + "append_stages(", "declare_attn(", "declare_ffn(", "_build_stages(", diff --git a/test/registered/unit/layer_boundary/test_terminal_stages.py b/test/registered/unit/layer_boundary/test_terminal_stages.py deleted file mode 100644 index b5115c850b..0000000000 --- a/test/registered/unit/layer_boundary/test_terminal_stages.py +++ /dev/null @@ -1,274 +0,0 @@ -import importlib -import sys -import types -import unittest -from types import ModuleType -from unittest.mock import MagicMock, patch - -import torch - -from sglang.test.ci.ci_register import register_cpu_ci -from sglang.test.test_utils import CustomTestCase - -register_cpu_ci(est_time=15, suite="base-a-test-cpu") - -NUM_LAYERS = 4 - - -def import_model(name): - if importlib.util.find_spec("vllm") is not None: - return importlib.import_module(f"sglang.srt.models.{name}") - # CPU CI omits vLLM; the Bailing modules import an AWQ kernel from it. - vllm = ModuleType("vllm") - vllm.__path__ = [] - custom_ops = ModuleType("vllm._custom_ops") - custom_ops.awq_dequantize = MagicMock() - with patch.dict(sys.modules, {"vllm": vllm, "vllm._custom_ops": custom_ops}): - return importlib.import_module(f"sglang.srt.models.{name}") - - -class StubModule(torch.nn.Module): - def __getattr__(self, name): - try: - return super().__getattr__(name) - except AttributeError: - return MagicMock() - - -def stub(*args, **kwargs): - return StubModule() - - -PARALLEL = types.SimpleNamespace( - tp_size=1, - tp_rank=0, - attn_tp_size=1, - attn_tp_rank=0, - attn_dp_size=1, - attn_dp_rank=0, - moe_ep_size=1, - moe_tp_size=1, - moe_dp_size=1, -) - - -def qwen3_next_config(num_layers): - from sglang.srt.configs.qwen3_next import Qwen3NextConfig - - return Qwen3NextConfig(num_hidden_layers=num_layers) - - -def bailing_hybrid_config(num_layers): - from sglang.srt.configs.bailing_hybrid import BailingHybridConfig - - return BailingHybridConfig(num_hidden_layers=num_layers, attention_type=1) - - -def bailing_moe_config(num_layers): - from sglang.srt.configs.bailing_moe_v2 import BailingMoeV2Config - - return BailingMoeV2Config(num_hidden_layers=num_layers) - - -def step3p5_config(num_layers): - from sglang.srt.configs.step3p5 import Step3p5Config - - # Per-layer lists also cover the MTP layer, whose id is num_layers. - return Step3p5Config( - num_hidden_layers=num_layers, - layer_types=["full_attention"] * (num_layers + 1), - moe_layers_enum=",".join(str(i) for i in range(1, num_layers + 1)), - swiglu_limits_shared=None, - rope_theta=[10000.0] * (num_layers + 1), - partial_rotary_factors=[1.0] * (num_layers + 1), - use_head_wise_attn_gate=False, - share_expert_dim=16, - ) - - -def minimax_m3_config(num_layers): - from transformers import PretrainedConfig - - return PretrainedConfig( - num_hidden_layers=num_layers, - hidden_size=16, - rms_norm_eps=1e-6, - dense_intermediate_size=32, - ) - - -# name: (module, decoder layer class, config factory, submodules to stub, -# (config layer count, layer id) of the draft model's layer or None) -CASES = { - "qwen3_next_linear": ( - "qwen3_next", - "Qwen3HybridLinearDecoderLayer", - qwen3_next_config, - ["Qwen3GatedDeltaNet", "Qwen2MoeSparseMoeBlock", "Qwen2MoeMLP"], - (1, 0), - ), - "qwen3_next_attention": ( - "qwen3_next", - "Qwen3HybridAttentionDecoderLayer", - qwen3_next_config, - [ - "Qwen2MoeSparseMoeBlock", - "Qwen2MoeMLP", - "QKVParallelLinear", - "RowParallelLinear", - "RadixAttention", - "get_rope", - ], - (1, 0), - ), - "bailing_moe_linear": ( - "bailing_moe_linear", - "BailingMoELinearDecoderLayer", - bailing_hybrid_config, - [ - "BailingMLP", - "BailingMoE", - "BailingMoEAttention", - "BailingMoELinearAttention", - "DsV3MLA", - ], - (NUM_LAYERS, 0), - ), - "bailing_moe": ( - "bailing_moe", - "BailingMoEBlock", - bailing_moe_config, - ["BailingMoEAttention", "BailingMoEMLP", "BailingMoESparseMoeBlock"], - (NUM_LAYERS, 0), - ), - "step3p5": ( - "step3p5", - "Step3p5DecoderLayer", - step3p5_config, - ["Step3p5Attention", "Step3p5MLP", "Step3p5MoEMLP"], - (NUM_LAYERS, NUM_LAYERS), - ), - "minimax_m3": ( - "minimax_m3", - "MiniMaxM3DecoderLayer", - minimax_m3_config, - ["MiniMaxM3Attention", "MiniMaxM3MLP", "MiniMaxM3MoE"], - None, - ), -} - - -def build(case, num_layers, layer_id, config=None, **kwargs): - """Construct a decoder with stubbed compute and inspect its stage declarations.""" - module_name, class_name, make_config, stubs, _ = CASES[case] - module = import_model(module_name) - communicator = MagicMock(return_value=(MagicMock(), MagicMock())) - built = [] - - def recording_stub(name): - def make(*args, **kwargs): - built.append(name) - return StubModule() - - return make - - from sglang.srt.layers.layer_boundary import declare_attn, declare_ffn - - patches = dict( - declare_attn=declare_attn, - declare_ffn=declare_ffn, - make_stages=communicator, - ) - patches.update({name: recording_stub(name) for name in stubs}) - if hasattr(module, "get_parallel"): - patches["get_parallel"] = lambda: PARALLEL - with patch.multiple(module, **patches): - getattr(module, class_name)( - config or make_config(num_layers), layer_id=layer_id, **kwargs - ) - communicator.assert_called_once() - declaration = communicator.call_args.args[1][0] - sequence = communicator.call_args.kwargs - return ( - {}, - { - "terminal": sequence["terminal"], - "enters_stack": sequence["previous"] is None, - "is_layer_sparse": declaration.sparse, - }, - built, - ) - - -def planned_as_last(case, num_layers, layer_id, **kwargs): - """Whether the layer's layout plan makes it the model's last layer, the one - fact the communicator reads the last layer from.""" - passed, planned, _ = build(case, num_layers, layer_id, **kwargs) - assert "is_last_layer" not in passed, "the plan is the only source" - return planned["terminal"] - - -class TestTerminalStages(CustomTestCase): - def test_only_the_last_layer_is_last(self): - """With all-reduce fusion on, only the model's last layer is marked last, - so its FFN all-reduce is never left to a next layer that does not exist.""" - for case in CASES: - for layer_id in range(NUM_LAYERS): - with self.subTest(case=case, layer_id=layer_id): - self.assertEqual( - planned_as_last(case, NUM_LAYERS, layer_id), - layer_id == NUM_LAYERS - 1, - ) - - def test_draft_model_layer_is_last(self): - """The single decoder layer of a NextN / MTP draft model is marked last, - whatever layer id and layer count it is built with.""" - for case, (*_, draft) in CASES.items(): - if draft is None: - continue - num_layers, layer_id = draft - with self.subTest(case=case): - self.assertTrue( - planned_as_last(case, num_layers, layer_id, is_nextn=True) - ) - - def test_draft_model_layer_is_planned_as_a_one_layer_model(self): - """The layout plan of a NextN / MTP draft layer treats it as the first and - the last layer, so it takes the model's input layout and returns the - model's output layout even when its MLP runs on SCATTERED tokens.""" - for case, (*_, draft) in CASES.items(): - if draft is None: - continue - num_layers, layer_id = draft - with self.subTest(case=case): - _, planned, _ = build(case, num_layers, layer_id, is_nextn=True) - self.assertTrue(planned["enters_stack"]) - self.assertTrue(planned["terminal"]) - - def test_bailing_draft_layer_builds_an_moe(self): - """The Bailing V2 NextN checkpoint holds expert weights, so the NextN - layer builds the sparse MoE block, also when the model's first layers - are dense.""" - config = bailing_moe_config(NUM_LAYERS) - config.first_k_dense_replace = 1 - _, planned, built = build( - "bailing_moe", NUM_LAYERS, 0, config=config, is_nextn=True - ) - self.assertIn("BailingMoESparseMoeBlock", built) - self.assertNotIn("BailingMoEMLP", built) - self.assertTrue(planned["is_layer_sparse"]) - - def test_bailing_hybrid_draft_layer_is_planned_as_sparse(self): - """The Bailing hybrid NextN layer builds an MoE, so its layout plan is - that of a sparse layer, also when the model's first layers are dense.""" - config = bailing_hybrid_config(NUM_LAYERS) - config.first_k_dense_replace = 1 - _, planned, built = build( - "bailing_moe_linear", NUM_LAYERS, 0, config=config, is_nextn=True - ) - self.assertIn("BailingMoE", built) - self.assertTrue(planned["is_layer_sparse"]) - - -if __name__ == "__main__": - unittest.main() diff --git a/test/registered/unit/layers/test_neighbour_layer_build.py b/test/registered/unit/layers/test_neighbour_layer_build.py new file mode 100644 index 0000000000..5605e17fac --- /dev/null +++ b/test/registered/unit/layers/test_neighbour_layer_build.py @@ -0,0 +1,76 @@ +"""A pipeline neighbour layer is built on the meta device and allocates nothing.""" + +import unittest +from types import SimpleNamespace +from unittest.mock import patch + +import torch + +from sglang.srt.layers.hyperconnection import GatedResidual, HyperConnectionConfig +from sglang.srt.layers.moe.fused_moe_triton import layer as fused_moe +from sglang.srt.models.qwen4_exp import _offloads_ple +from sglang.srt.utils.common import ( + building_neighbour_layer, + is_building_neighbour_layer, +) +from sglang.test.ci.ci_register import register_cpu_ci +from sglang.test.test_utils import CustomTestCase + +register_cpu_ci(est_time=5, suite="base-a-test-cpu") + + +class TestNeighbourLayerBuild(CustomTestCase): + def test_the_flag_holds_only_inside_the_build(self): + self.assertFalse(is_building_neighbour_layer()) + with building_neighbour_layer(): + self.assertTrue(is_building_neighbour_layer()) + self.assertEqual(torch.empty(1).device.type, "meta") + self.assertFalse(is_building_neighbour_layer()) + + def test_a_gated_residual_allocates_nothing(self): + config = HyperConnectionConfig(hc_count=4, hidden_size=64, hc_lowrank=16) + with building_neighbour_layer(): + module = GatedResidual(config, use_mix=True, use_combine=True) + tensors = [*module.parameters(), *module.buffers()] + self.assertTrue(tensors) + self.assertEqual({t.device.type for t in tensors}, {"meta"}) + + def test_a_neighbour_keeps_its_ple_table_off_the_host(self): + offloaded = SimpleNamespace(ple_offload_embedding=True) + self.assertTrue(_offloads_ple(offloaded)) + with building_neighbour_layer(): + self.assertFalse(_offloads_ple(offloaded)) + self.assertFalse(_offloads_ple(SimpleNamespace(ple_offload_embedding=False))) + + def test_a_neighbour_moe_sets_up_no_all_to_all(self): + class FlashinferA2A: + def is_flashinfer(self): + return True + + def __getattr__(self, name): + if name.startswith("is_"): + return lambda: False + raise AttributeError(name) + + def unexpected(**kwargs): + raise AssertionError("a neighbour layer built an all-to-all dispatcher") + + config = SimpleNamespace( + top_k=2, num_experts=4, num_local_experts=4, hidden_size=8 + ) + parallel = SimpleNamespace(tp_group=SimpleNamespace(device_group=None)) + standard = object() + with ( + patch.object(fused_moe, "get_moe_a2a_backend", FlashinferA2A), + patch.object(fused_moe, "get_parallel", lambda: parallel), + patch.object(fused_moe, "FlashinferDispatcher", unexpected), + patch.object(fused_moe, "StandardDispatcher", lambda config: standard), + ): + with building_neighbour_layer(): + self.assertIs(fused_moe.create_moe_dispatcher(config, None), standard) + with self.assertRaisesRegex(AssertionError, "all-to-all dispatcher"): + fused_moe.create_moe_dispatcher(config, None) + + +if __name__ == "__main__": + unittest.main() diff --git a/test/registered/unit/layers/test_xielu.py b/test/registered/unit/layers/test_xielu.py new file mode 100644 index 0000000000..99b1f0af9c --- /dev/null +++ b/test/registered/unit/layers/test_xielu.py @@ -0,0 +1,30 @@ +import unittest + +import torch + +from sglang.srt.layers.activation import XIELU +from sglang.test.ci.ci_register import register_cpu_ci +from sglang.test.test_utils import CustomTestCase + +register_cpu_ci(est_time=2, suite="base-a-test-cpu") + + +class TestXIELU(CustomTestCase): + def test_builds_on_the_meta_device_with_the_same_scalars(self): + # A layer built only for the stage boundaries it declares is built on + # the meta device, where nothing can be read back from a buffer. + for dtype in (torch.bfloat16, torch.float32): + with self.subTest(dtype=dtype): + on_cpu = XIELU(beta=0.3, eps=-1e-6, dtype=dtype) + with torch.device("meta"): + on_meta = XIELU(beta=0.3, eps=-1e-6, dtype=dtype) + self.assertEqual(on_meta.beta.device.type, "meta") + self.assertEqual(on_meta._beta_scalar, on_cpu._beta_scalar) + self.assertEqual(on_meta._eps_scalar, on_cpu._eps_scalar) + # Rounded through dtype, like the buffers they mirror. + self.assertEqual(on_cpu._beta_scalar, on_cpu.beta.float().item()) + self.assertEqual(on_cpu._eps_scalar, on_cpu.eps.float().item()) + + +if __name__ == "__main__": + unittest.main()