[Refactor] Construct independent decoder stage boundaries (#41552)

This commit is contained in:
Cheng Wan
2026-09-28 20:17:00 -07:00
committed by GitHub
parent acffb0d20d
commit 49b5e76e45
79 changed files with 5182 additions and 2318 deletions
@@ -1551,6 +1551,12 @@ Combining `--enable-response-store` with `--disaggregation-mode=prefill` or `dec
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The path of the draft model weights. This can be a local folder or a Hugging Face repo ID.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>speculative_boundary_reduction</code><br/>Server args field; no CLI flag</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Resolved FFN boundary reduction policy for the speculative draft. Argument processing derives this field from <code>boundary_reduction</code> and the draft model configuration; an independent value is overwritten by resolution. See <a href="#ffn-boundary-reduction">FFN boundary reduction</a>.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>rs+rsv</code> before resolution; model-dependent afterward</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>ar</code>, <code>rs</code>, <code>rsv</code>, <code>rs+rsv</code></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--speculative-draft-model-revision`</td>
@@ -2520,6 +2526,12 @@ Combining `--enable-response-store` with `--disaggregation-mode=prefill` or `dec
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool flag (set to enable)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--boundary-reduction</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Select the optional FFN output reduction strategy. Unsupported reduce-scatter paths fall back to all-reduce followed by token redistribution. See <a href="#ffn-boundary-reduction">FFN boundary reduction</a>.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>auto</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>auto</code>, <code>ar</code>, <code>rs</code>, <code>rsv</code>, <code>rs+rsv</code></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--disable-custom-all-reduce`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Disable the custom all-reduce kernel and fall back to NCCL.</td>
@@ -2848,6 +2860,26 @@ Combining `--enable-response-store` with `--disaggregation-mode=prefill` or `dec
</tbody>
</table>
### FFN boundary reduction
Use `--boundary-reduction` to choose which optional FFN output reductions may combine the sum with token redistribution. The selected policy applies to both dense and MoE FFN layers. It currently controls models using independently constructed stage boundaries; models still using `LayerCommunicator` retain their existing reduction policy.
| Value | Behavior |
| --- | --- |
| `auto` | Resolve the model default during server-argument processing, before model construction. |
| `ar` | Use all-reduce followed by token redistribution where needed. |
| `rs` | Permit fixed-size reduce-scatter when the layout and backend support it; otherwise use `ar`. |
| `rsv` | Permit variable-size reduce-scatter for eligible attention-DP paths; otherwise use `ar`. |
| `rs+rsv` | Permit both, trying the eligible variable-size path first, then fixed-size, then `ar`. |
`auto` selects `ar` for dense Qwen3 language backbones (including Qwen3-VL wrappers) and Moss-VL; `rsv` for BailingMoELinear/BailingMoeV2_5, LongCat Flash and Step3-VL; and `rs+rsv` otherwise. Qwen3-MoE, Qwen3-Next and Qwen3.5 are not matched by the dense Qwen3 rule and resolve to `rs+rsv`.
LongCat Flash and Moss-VL resolve these defaults, but still use the layer communicator at this revision and do not yet consume this option.
With `auto`, an independent speculative draft resolves its own model default; same-model MTP inherits the target policy. An explicit value applies to both target and draft. The result is recorded in the `speculative_boundary_reduction` server args field (`no_cli=True`); it has no separate CLI flag or independent override during resolution.
This option does not disable required attention or MoE collectives, including expert all-to-all, or select the all-reduce fusion backend. For example, `ar` can still use fused all-reduce plus residual normalization; use `--enforce-disable-flashinfer-allreduce-fusion` to disable that FlashInfer fusion.
## Dynamic batch tokenizer
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
@@ -0,0 +1,88 @@
"""Resolve collective preferences before constructing target and draft models."""
import json
from sglang.srt.arg_groups.model_override_base import model_config_of
_RSV_ARCHITECTURES = {
"BailingMoELinearForCausalLM",
"BailingMoeV2_5ForCausalLM",
"LongcatFlashForCausalLM",
"LongcatFlashForCausalLMNextN",
"Step3VLForConditionalGeneration",
}
_AR_ARCHITECTURES = {
"Qwen3ForCausalLM",
"Qwen3ForSequenceClassification",
"Qwen3Model",
"Qwen3ForRewardModel",
"MossVLForCausalLM",
"MossVLForConditionalGeneration",
}
def model_boundary_reduction(config):
"""One default for a model's dense and MoE layers, including its MTP."""
architectures = set(getattr(config, "architectures", None) or ())
if architectures & _RSV_ARCHITECTURES:
return "rsv"
if architectures & _AR_ARCHITECTURES:
return "ar"
# MiniCPM-V 4.5 constructs Qwen3 from its top-level config.
if (
getattr(config, "model_type", None) == "minicpmv"
and str(getattr(config, "version", "")) == "4.5"
):
return "ar"
# Wrappers carry their actual language backbone in the nested HF config.
# Match Qwen3 specifically: Qwen3-MoE/Next/3.5 fall through to rs+rsv.
if getattr(config, "model_type", None) in ("qwen3", "qwen3_vl_text"):
return "ar"
get_text = getattr(config, "get_text_config", None)
if get_text is not None:
text = get_text()
if text is not config:
return model_boundary_reduction(text)
for name in ("text_config", "llm_config", "language_config"):
text = getattr(config, name, None)
if text is not None and text is not config:
return model_boundary_reduction(text)
return "rs+rsv"
def resolve_boundary_reduction(view):
"""Resolve auto once, after speculative paths/revisions have been resolved."""
requested = view.boundary_reduction
dummy = view.model_path.lower() in ("none", "dummy")
if requested != "auto":
target = draft = requested
else:
target = draft = (
"rs+rsv"
if dummy
else model_boundary_reduction(model_config_of(view).hf_config)
)
draft_path = view.speculative_draft_model_path
if (
not dummy
and view.speculative_algorithm is not None
and draft_path is not None
and draft_path != view.model_path
):
from sglang.srt.utils.hf_transformers_utils import get_config
kwargs = {}
if view.decrypted_draft_config_file:
kwargs["_configuration_file"] = view.decrypted_draft_config_file
config = get_config(
draft_path,
trust_remote_code=view.trust_remote_code,
revision=view.speculative_draft_model_revision,
model_override_args=json.loads(view.json_model_override_args),
**kwargs,
)
draft = model_boundary_reduction(config)
return {
"boundary_reduction": target,
"speculative_boundary_reduction": draft,
}
@@ -250,6 +250,7 @@ POSITIONAL_FIELD_ORDER = (
"enable_torch_symm_mem",
"enable_scattered_sconv",
"pre_warm_nccl",
"boundary_reduction",
"enable_quant_communications",
"enable_flashinfer_allreduce_fusion",
"enforce_disable_flashinfer_allreduce_fusion",
@@ -620,6 +620,19 @@ class ExecComm(msgspec.Struct):
bool,
"Pre-warm NCCL/RCCL communicators during startup to reduce P99 TTFT cold-start latency. Default: enabled for AMD/HIP (RCCL), disabled for NVIDIA/CUDA (NCCL).",
] = False
boundary_reduction: A[
Literal["auto", "ar", "rs", "rsv", "rs+rsv"],
Arg(
help="Select FFN boundary reduction: ar uses all-reduce then token "
"redistribution; rs and rsv permit fixed-size and variable-size "
"reduce-scatter respectively; rs+rsv permits both (RSv first). "
"Unsupported paths fall back to ar. The option applies only to FFN "
"layers using stage boundaries, where auto resolves the model default. "
"Required attention and MoE collectives and "
"all-reduce fusion are unaffected.",
resolvable=True,
),
] = "auto"
enable_quant_communications: A[
Optional[bool],
"Enable INT8 quantization of TP communications (limited support).",
@@ -39,6 +39,10 @@ class Spec(msgspec.Struct):
aliases=["--speculative-draft-model"],
),
] = None
speculative_boundary_reduction: A[
Literal["ar", "rs", "rsv", "rs+rsv"],
Arg(no_cli=True, resolvable=True),
] = "rs+rsv"
speculative_draft_model_revision: A[
Optional[str],
"The specific draft model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version.",
+7
View File
@@ -92,6 +92,9 @@ def run_resolution_pipeline(server_args: Any) -> None:
run_hook(validate_prefill_cp_platform, server_args)
run_hook(handle_hardware_runtime_validation, server_args)
if cfg.model_path.lower() in ["none", "dummy"]:
from sglang.srt.arg_groups.boundary_reduction import resolve_boundary_reduction
run_post_process_pass(server_args, resolve_boundary_reduction)
return
from sglang.srt.arg_groups.model_path_hook import (
@@ -284,6 +287,10 @@ def run_resolution_pipeline(server_args: Any) -> None:
run_hook(handle_speculative_decoding, server_args)
from sglang.srt.arg_groups.boundary_reduction import resolve_boundary_reduction
run_post_process_pass(server_args, resolve_boundary_reduction)
# After the speculative hook so speculative_algorithm is final.
from sglang.srt.arg_groups.layernorm_sp_hook import handle_layernorm_sp
@@ -15,7 +15,6 @@ from sglang.srt.batch_overlap.operations import (
from sglang.srt.batch_overlap.operations_strategy import OperationsStrategy
from sglang.srt.layers import deep_gemm_wrapper
from sglang.srt.layers.communicator import (
CommunicateContext,
Layout,
tbo_split_moves,
)
@@ -1022,14 +1021,12 @@ def _model_forward_tbo_split_inputs(
zero_allocator: Optional[BumpAllocator],
layer_input_rows: Layout,
) -> List[Dict]:
context = CommunicateContext.init_new()
to_splitter, to_layer_input = tbo_split_moves(layer_input_rows)
hidden_states, residual = to_splitter(
hidden_states=hidden_states,
residual=residual,
forward_batch=forward_batch,
context=context,
)
inputs_arr = _model_forward_tbo_split_inputs_raw(
@@ -1045,7 +1042,6 @@ def _model_forward_tbo_split_inputs(
hidden_states=hidden_states,
residual=residual,
forward_batch=forward_batch,
context=context,
)
return dict(
hidden_states=hidden_states,
@@ -11,7 +11,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
"""Communication between a decoder layer's stages."""
"""Stage boundaries: layout transport, residual updates, reads, and fusion."""
from sglang.srt.layers.communicator.adapters.attention import (
AttentionInputs,
@@ -19,7 +19,6 @@ from sglang.srt.layers.communicator.adapters.attention import (
)
from sglang.srt.layers.communicator.boundary import (
BoundarySteps,
DecoderLayerSides,
EdgeDecl,
FusedMlpInput,
LayerStage,
@@ -28,25 +27,25 @@ from sglang.srt.layers.communicator.boundary import (
StageInput,
StageKind,
StageOutput,
decoder_layer_edges,
decoder_layer_sides,
input_scattered_layer_sides,
make_boundary,
make_output_boundary,
sequence_parallel_layer_sides,
stage_edges,
tbo_split_moves,
)
from sglang.srt.layers.communicator.layer import (
FfnCompletion,
FfnExit,
LayerCommunicator,
LayerFacts,
MHCLayerCommunicator,
MixerExit,
from sglang.srt.layers.communicator.construction import (
BatchVariant,
StageEdges,
make_stage,
)
from sglang.srt.layers.communicator.exit import FfnCompletion, FfnExit, MixerExit
from sglang.srt.layers.communicator.factories import (
declare_attn,
declare_ffn,
make_attn_stage,
make_ffn_stage,
make_stages,
)
from sglang.srt.layers.communicator.layout import (
CommunicateContext,
Layout,
SumGroup,
TokenAxis,
@@ -79,25 +78,28 @@ from sglang.srt.layers.communicator.residual.mhc import (
)
__all__ = [
"LayerCommunicator",
"MHCLayerCommunicator",
"LayerFacts",
"declare_attn",
"declare_ffn",
"make_attn_stage",
"make_ffn_stage",
"make_stages",
"ADD",
"AttentionInputs",
"BoundarySteps",
"CommunicateContext",
"CommunicateSimpleFn",
"CommunicateSummableTensorPairFn",
"DecoderLayerSides",
"EdgeDecl",
"FUSE_ALLREDUCE_MAX_BATCH_SIZE",
"FfnCompletion",
"FfnExit",
"FusedMlpInput",
"HandoffOutput",
"LayerCommunicator",
"LayerResidual",
"LayerFacts",
"LayerStage",
"Layout",
"MHCLayerCommunicator",
"MHCState",
"MixerExit",
"NORM_QUANT_READ",
@@ -111,20 +113,22 @@ __all__ = [
"SumGroup",
"TokenAxis",
"UnreducedOutput",
"decoder_layer_edges",
"decoder_layer_sides",
"enable_moe_dense_fully_dp",
"get_attn_tp_context",
"input_scattered_layer_sides",
"make_boundary",
"make_output_boundary",
"moe_cp_gathers_sparse_moe_input",
"move_rows",
"reduce_output",
"sequence_parallel_layer_sides",
"sparse_moe_gathers_over_moe_cp",
"stage_edges",
"tbo_split_moves",
"token_axis_sizes",
"tp_reduce_scatter",
]
from sglang.srt.layers.communicator.layer import (
LayerCommunicator,
LayerFacts,
MHCLayerCommunicator,
)
@@ -21,7 +21,6 @@ from typing import Callable
import torch
from sglang.srt.layers.communicator.layout import (
CommunicateContext,
enable_moe_dense_fully_dp,
)
from sglang.srt.layers.dp_attention import (
@@ -182,7 +181,6 @@ def _redistribute_from_attn_tp_shards(tensor: torch.Tensor) -> torch.Tensor:
return gathered
def _redistribute_to_attn_tp_shards(
tensor: torch.Tensor, context: CommunicateContext
) -> torch.Tensor:
return tensor.tensor_split(context.attn_tp_size)[context.attn_tp_rank]
def _redistribute_to_attn_tp_shards(tensor: torch.Tensor) -> torch.Tensor:
parallel = get_parallel()
return tensor.tensor_split(parallel.attn_tp_size)[parallel.attn_tp_rank]
@@ -0,0 +1,87 @@
# Copyright 2023-2024 SGLang Team
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
"""Move prepared branch inputs and merge complete branch contributions."""
from __future__ import annotations
from typing import Tuple
import torch
from sglang.srt.layers.communicator.ops import move_rows
from sglang.srt.layers.communicator.residual.stream import ResidualStream
from sglang.srt.model_executor.forward_batch_info import ForwardBatch
def branch_input(
self,
source: object,
hidden_states: torch.Tensor,
residual: torch.Tensor,
forward_batch: ForwardBatch,
) -> Tuple[torch.Tensor, torch.Tensor]:
"""The FFN input and residual that ``source``'s boundary read for its
own FFN, for this layer's FFN, which branches from the same input: moved
to the rows this FFN needs and its residual's rows."""
rows, residual_rows, _ = source._branch_rows(forward_batch)
to, residual_to, _ = self._branch_rows(forward_batch)
stream = residual if isinstance(residual, ResidualStream) else None
if stream is not None:
if stream.pending is not None:
raise RuntimeError("a branch must start from a prepared stage input")
residual = stream.residual
hidden_states = move_rows(hidden_states, rows, to, forward_batch)
residual = move_rows(residual, residual_rows, residual_to, forward_batch)
return (
hidden_states,
ResidualStream(residual) if stream is not None else residual,
)
def branch_output(
self, hidden_states: torch.Tensor, forward_batch: ForwardBatch
) -> torch.Tensor:
"""This layer's complete FFN output as a branch's contribution, which
adds to the layer's output without writing the residual: moved to the
rows the layer hands on."""
rows, _, to = self._branch_rows(forward_batch)
return move_rows(hidden_states, rows, to, forward_batch)
def merge_branch(
self,
contribution: torch.Tensor,
hidden_states: torch.Tensor,
residual: torch.Tensor,
source: object,
forward_batch: ForwardBatch,
) -> Tuple[torch.Tensor, torch.Tensor]:
"""A contribution from ``branch_output`` summed with what ``source``'s
layer hands on, ``hidden_states`` and ``residual``, moved to the rows
this layer hands on."""
_, _, rows = source._branch_rows(forward_batch)
_, _, to = self._branch_rows(forward_batch)
stream = residual if isinstance(residual, ResidualStream) else None
update = None
if stream is not None:
stream.check(hidden_states)
update = stream.pending.update
hidden_states, residual = stream.finish(hidden_states)
hidden_states = move_rows(hidden_states, rows, to, forward_batch)
residual = move_rows(residual, rows, to, forward_batch)
output = contribution + hidden_states
if stream is not None:
stream.write(residual)
return stream.leave(output, update), stream
return output, residual
@@ -0,0 +1,32 @@
# Copyright 2023-2024 SGLang Team
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
"""Publish the token rows consumed by the selected stage to LoRA kernels."""
from sglang.srt.layers.communicator.layout import TokenAxis
from sglang.srt.runtime_context import LoRABatchLayout, get_forward
def publish_attention(enabled):
if enabled:
get_forward().set("lora_batch_layout", LoRABatchLayout.DP_LOCAL)
def publish_ffn(enabled, input_rows):
if enabled:
get_forward().set(
"lora_batch_layout",
LoRABatchLayout.DP_LOCAL
if TokenAxis.ATTN_DP in input_rows.sharded
else LoRABatchLayout.TP_GLOBAL,
)
@@ -51,6 +51,7 @@ from sglang.srt.layers.communicator.ops import (
tp_reduce_scatter,
tp_slice,
)
from sglang.srt.layers.communicator.output import OutputTransform
from sglang.srt.layers.communicator.residual import (
LayerResidual,
StageRead,
@@ -102,6 +103,7 @@ class StageOutput(msgspec.Struct, frozen=True):
# How the producer's output is written into the residual. An arrival
# description has no producer object; its edge declares capabilities only.
update: Optional[StageUpdate] = ADD
transform: Optional[OutputTransform] = None
class StageDecl(msgspec.Struct, frozen=True):
@@ -481,6 +483,7 @@ class FusedMlpInput(msgspec.Struct, frozen=True):
# The group whose sum it completes.
completes: SumGroup
run: Callable[..., Optional[Tuple[torch.Tensor, torch.Tensor]]]
preserves_residual: Optional[Callable] = None
def tbo_split_moves(layer_input_rows: Layout) -> Tuple[Callable, Callable]:
@@ -536,12 +539,12 @@ class StageEntry(msgspec.Struct, frozen=True):
# Completes what the input owes, writes the previous stage's output into
# the residual and reads this stage's input:
# (hidden_states, residual, forward_batch, norm, context, **call).
# (hidden_states, residual, forward_batch, norm, **call).
prepare: Callable
# The rows prepare hands the stage.
input_rows: Layout
# Moves the input onto the stage's rows after prepare, when prepare does
# not: (hidden_states, forward_batch, context) -> hidden_states.
# not: (hidden_states, forward_batch) -> hidden_states.
input_move: Optional[Callable] = None
# Hands the stage its input once it is on its rows:
# (hidden_states, forward_batch, qkv_latent_func) -> hidden_states.
@@ -551,6 +554,7 @@ class StageEntry(msgspec.Struct, frozen=True):
# Return the updated residual to the producer's rows for aux capture.
capture_move: Optional[Callable] = None
capture_move_allocates: bool = False
preserves_residual: Optional[Callable] = None
capture_preserves_residual: Optional[Callable] = None
input_sum: Optional[SumGroup] = None
@@ -619,6 +623,7 @@ class Boundary(msgspec.Struct, frozen=True):
output_move: Optional[Callable] = None
# Whether output_move also completes the sum the producer leaves.
output_move_completes_sum: bool = False
preserves_residual: Optional[Callable] = None
@property
def input_rows(self) -> Layout:
@@ -682,6 +687,7 @@ def make_boundary(
raise NotImplementedError("update capabilities require different input moves")
return msgspec.structs.replace(
first,
preserves_residual=None,
prepare=partial(
_dispatch_consumer,
paths={capability: path.prepare for capability, path in paths.items()},
@@ -756,8 +762,24 @@ def _bind_consumer(
cp_moves=cp_moves,
enters_stack=False,
)
preserves_residual = None
if (
edge.produced.always_leaves
and edge.produced.group is SumGroup.ATTN_TP
and isinstance(step, partial)
and step.func is _mlp_input_without_dp
and not step.keywords["gathers_residual"]
):
selected = step.keywords["fusions"]
if selected:
# Only the first reachable candidate can certify ownership; an
# earlier custom candidate could otherwise mutate the residual.
preserves_residual = next(
f.preserves_residual for f in fusions if f.run == selected[0]
)
return Boundary(
edge,
preserves_residual=preserves_residual,
prepare=partial(
_consumer_step,
step=step,
@@ -0,0 +1,307 @@
# Copyright 2023-2024 SGLang Team
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
"""Bind one stage to predeclared boundaries and reusable batch paths."""
from dataclasses import dataclass
from enum import Enum, auto
from functools import cached_property
from typing import Callable, Optional, Protocol, Tuple
from sglang.srt.environ import envs
from sglang.srt.layers.communicator.adapters.attention import get_attn_tp_context
from sglang.srt.layers.communicator.boundary import (
BoundarySteps,
EdgeDecl,
FusedMlpInput,
StageDecl,
StageEntry,
StageKind,
make_boundary,
make_output_boundary,
)
from sglang.srt.layers.communicator.exit import OutputBoundary
from sglang.srt.layers.communicator.fusions.allreduce import (
attention_fusions,
ffn_fusions,
)
from sglang.srt.layers.communicator.layout import (
TokenAxis,
_batch_shards_over_cp,
_gathers_over_attention_cp,
_generic_prefill_cp_shards_tokens,
enable_moe_dense_fully_dp,
)
from sglang.srt.layers.communicator.stage import StageCommunicator
from sglang.srt.layers.moe import (
get_moe_a2a_backend,
)
from sglang.srt.model_executor.forward_batch_info import ForwardBatch
from sglang.srt.runtime_context import (
get_forward,
get_lora,
get_parallel,
get_spec,
)
from sglang.srt.speculative.spec_info import SpeculativeAlgorithm
_use_ag_after_qlora = envs.SGLANG_USE_AG_AFTER_QLORA.get()
class BoundaryFusions(Protocol):
"""The fused kernels a fusion backend gives a layer, bound to it and tried
before the layer's own: at its attention input (each takes what the previous
layer left, the residual, the batch and a post-residual addition), at its
FFN input, and at its FFN exit (each says what the exit does when its kernel
takes the batch)."""
def attention_input(self, layer: "StagePlan") -> Tuple[Callable, ...]: ...
def ffn_input(self, layer: "StagePlan") -> Tuple["FusedMlpInput", ...]: ...
requires_local_reduction: bool
def can_defer_finalize(
self, layer: "StagePlan", forward_batch: ForwardBatch
) -> bool: ...
def _refuse_uncovered_cp_moe(moe_on_local_rows: bool, cp_shards: bool) -> None:
"""A MoE layer under attention CP whose tokens the steps cannot bring
to it: under a prefill CP, one dispatched per DP shard under attention DP
and GQA CP, and one on the TP group whose data-parallel groups are the CP
ranks under DSA or MLA CP; and under attention DP, one on the TP group
whose data-parallel groups are the CP ranks."""
parallel = get_parallel()
gqa = not _gathers_over_attention_cp()
if moe_on_local_rows:
if cp_shards and gqa and parallel.attn_dp_size > 1:
raise NotImplementedError(
"a MoE dispatched per DP shard under attention DP and GQA prefill CP"
)
elif parallel.moe_dp_size == parallel.attn_cp_size:
if cp_shards and not gqa:
raise NotImplementedError(
"a MoE on the TP group with moe_dp_size == attn_cp_size under "
"DSA or MLA prefill CP"
)
if parallel.attn_dp_size > 1:
raise NotImplementedError(
"a MoE on the TP group with moe_dp_size == attn_cp_size under "
"attention DP and attention CP"
)
def _input_can_be_scattered() -> bool:
"""Whether a batch may run this layer with input-scattered attention:
configured, on TP without attention DP, a prefill CP, an a2a backend or
a dense MLP on every rank. The rest of what ``AttnTpContext.init_context`` requires is only
known once the model is built."""
parallel = get_parallel()
return (
parallel.enable_attn_tp_input_scattered
and parallel.tp_size > 1
and parallel.attn_dp_size == 1
and not _generic_prefill_cp_shards_tokens()
and get_moe_a2a_backend().is_none()
and not enable_moe_dense_fully_dp()
)
class BatchVariant(Enum):
ORDINARY = auto()
CONTEXT_PARALLEL = auto()
INPUT_SCATTERED = auto()
SEQUENCE_PARALLEL = auto()
@dataclass(frozen=True)
class StageEdges:
"""One stage's incoming and outgoing declarations for one batch variant.
No adjacent stage object is required. Edges can come from a local model
assembler, pipeline contract or branch adapter.
"""
incoming: EdgeDecl
outgoing: EdgeDecl
handoff: Optional[Callable] = None
cp_moves: object = None
def _requires_branch_input(*args, **kwargs):
raise RuntimeError("a prepared branch must enter through branch_input, not prepare")
class StagePlan:
"""Precomputed paths for one stage; never owns the neighbouring norm."""
def __init__(
self,
kind,
norm,
variants,
*,
enters_stack=False,
prepared_input=False,
terminal=False,
fixed_output=False,
is_sparse=False,
qkv_latent_func=None,
force_layernorm_before_dp_gather=False,
enable_fused_ar_quant=False,
fused_ar_quant_keep_bf16=False,
residual_in_hidden=False,
fusions=None,
):
self.kind = kind
self.norm = norm
self.variants = dict(variants)
self.is_first_layer = enters_stack
self.is_last_layer = terminal
self.fixed_output = fixed_output
self.is_sparse = is_sparse
self.qkv_latent_func = qkv_latent_func
self.enable_fused_ar_quant = enable_fused_ar_quant
self.fused_ar_quant_keep_bf16 = fused_ar_quant_keep_bf16
self.residual_in_hidden = residual_in_hidden
self.fusions = fusions
self._speculative_algo = SpeculativeAlgorithm.from_string(
get_spec().speculative_algorithm
)
self._publish_lora_layout = get_parallel().enable_dp_attention and bool(
get_lora().enable_lora
)
self._fusion_rows = None
carried = attention_fusions(self) if kind is StageKind.ATTENTION else ()
self._carried_fusions = carried
fused = ffn_fusions(self) if kind is StageKind.FFN else ()
self._paths = {}
for variant, edges in self.variants.items():
if prepared_input:
# The branch adapter moves an already-read input and forks its
# stream. Binding a normal prepare would add or norm it twice.
entry = StageEntry(
prepare=_requires_branch_input,
input_rows=edges.incoming.need.layout,
)
else:
into = make_boundary(
edges.incoming,
fusions=fused,
carried_fusions=carried,
force_layernorm_before_gather=force_layernorm_before_dp_gather,
cp_moves=edges.cp_moves,
enters_stack=enters_stack,
)
entry = StageEntry(
prepare=into.prepare,
input_rows=into.input_rows,
input_move=into.input_move,
handoff=edges.handoff,
fused=into.fused,
capture_move=into.capture_move,
capture_move_allocates=into.capture_move_allocates,
preserves_residual=into.preserves_residual,
input_sum=edges.incoming.produced.group
if edges.incoming.produced.always_leaves
else None,
)
out = (
None
if fixed_output
else make_output_boundary(edges.outgoing, cp_moves=edges.cp_moves)
)
self._paths[variant] = BoundarySteps(
attention=entry if kind is StageKind.ATTENTION else None,
ffn=entry if kind is StageKind.FFN else None,
ffn_output=edges.outgoing.produced,
ffn_output_move=None if out is None else out.output_move,
ffn_output_move_completes_sum=False
if out is None
else out.output_move_completes_sum,
ffn_sum_is_movable=edges.outgoing.produced.group is not None,
)
self._steps = self._paths[BatchVariant.ORDINARY]
self._cp_steps = self._paths.get(BatchVariant.CONTEXT_PARALLEL)
self._input_scattered_steps = self._paths.get(BatchVariant.INPUT_SCATTERED)
self._sp_steps = self._paths.get(BatchVariant.SEQUENCE_PARALLEL)
@property
def input_rows(self):
return self.variants[BatchVariant.ORDINARY].incoming.residual
@property
def input_on_attention_tp_slices(self):
return TokenAxis.ATTN_TP_SCATTER in self.input_rows.sharded
def _variant(self, forward_batch):
if self._sp_steps is not None and get_forward().sp_active:
return BatchVariant.SEQUENCE_PARALLEL
if (
self._input_scattered_steps is not None
and get_attn_tp_context().input_scattered
):
return BatchVariant.INPUT_SCATTERED
if self._cp_steps is not None and _batch_shards_over_cp(forward_batch):
return BatchVariant.CONTEXT_PARALLEL
return BatchVariant.ORDINARY
def _batch_steps(self, forward_batch):
return self._paths[self._variant(forward_batch)]
def fusion_rows(self, forward_batch):
if self._fusion_rows is not None:
return self._fusion_rows[self._variant(forward_batch)]
entry = self._batch_steps(forward_batch)
return (entry.attention or entry.ffn).input_rows
def produced(self, kind, forward_batch):
return self._batch_steps(forward_batch).ffn_output
@cached_property
def output(self):
return OutputBoundary(self)
def _branch_rows(self, forward_batch):
if self._variant(forward_batch) is not BatchVariant.ORDINARY:
raise NotImplementedError("branch transport requires ordinary batch rows")
edges = self.variants[BatchVariant.ORDINARY]
return (
edges.incoming.need.layout,
edges.incoming.residual_to,
edges.outgoing.need.layout,
)
def make_stage(
declaration: StageDecl,
*,
kind: StageKind,
norm,
incoming: EdgeDecl,
outgoing: EdgeDecl,
variants=None,
handoff=None,
**options,
) -> StageCommunicator:
"""Build one real stage from its own declaration and adjacent edges.
The optional variants supply the same stage's CP/SP/input-scattered
declarations. Neither a paired stage nor its layer facts/norm is needed.
"""
if incoming.need != declaration.input or outgoing.produced != declaration.output:
raise ValueError("stage declaration disagrees with its boundaries")
paths = dict(variants or {})
paths[BatchVariant.ORDINARY] = StageEdges(incoming, outgoing, handoff)
return StageCommunicator(StagePlan(kind, norm, paths, **options), kind, norm)
@@ -0,0 +1,521 @@
# Copyright 2023-2024 SGLang Team
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
"""Producer output choices, reduction scopes, and completion."""
from __future__ import annotations
from functools import partial
from typing import Callable, Optional, Tuple, Union
import msgspec
import torch
from sglang.srt.distributed import GroupCoordinator
from sglang.srt.environ import envs
from sglang.srt.layers.communicator.adapters.attention import get_attn_tp_context
from sglang.srt.layers.communicator.layout import (
SumGroup,
_ffn_has_tokens,
_sum_group,
)
from sglang.srt.layers.communicator.ops import (
_all_reduce_then_to_local_tokens,
_redistribute_output,
_reduce_and_redistribute_output_max_len,
_reduce_and_redistribute_output_varlen,
_to_local_tokens,
)
from sglang.srt.layers.communicator.output import (
UnreducedOutput,
)
from sglang.srt.layers.communicator.residual.add_norm import (
apply_aiter_all_reduce_fusion,
apply_flashinfer_allreduce_fusion,
)
from sglang.srt.layers.communicator.residual.stream import ResidualStream
from sglang.srt.layers.dp_attention import (
can_use_dp_reduce_scatter,
is_dp_attention_enabled,
is_enable_moe_cp_allgather,
)
from sglang.srt.layers.moe import (
can_merge_post_experts_all_reduce,
get_moe_a2a_backend,
post_experts_reduction_group,
post_experts_sum_is_one_all_reduce,
should_use_dp_reduce_scatterv,
)
from sglang.srt.model_executor.forward_batch_info import ForwardBatch
from sglang.srt.runtime_context import (
get_exec,
get_forward,
get_lora,
get_parallel,
)
def _reduce_and_redistribute_output_step(
forward_batch: ForwardBatch,
*,
leaves_for_reduce_scatter: bool,
leaves_for_reduce_scatterv: bool,
) -> Optional[Callable[[torch.Tensor, torch.Tensor, ForwardBatch], None]]:
"""The reduce-scatter that brings an FFN output gathered over attention
DP back to this rank's tokens when the FFN leaves its sum to it (see
StageOutput); None when the FFN reduces the output and only a scatter
remains."""
if should_use_dp_reduce_scatterv() and leaves_for_reduce_scatterv:
return _reduce_and_redistribute_output_varlen
if (
leaves_for_reduce_scatter
and forward_batch.dp_padding_mode.is_max_len()
and can_use_dp_reduce_scatter()
):
return _reduce_and_redistribute_output_max_len
return None
class OutputBoundary:
"""Choose and finish one producer's output; the batch owns its residual."""
def __init__(self, plan):
self.plan = plan
def postprocess_layer(
self,
hidden_states: torch.Tensor,
residual: torch.Tensor,
forward_batch: ForwardBatch,
):
"""Move a complete output from the operation-scheduled producer path."""
stream = residual if isinstance(residual, ResidualStream) else None
hidden_states, residual = self._complete_ffn_output_now(
hidden_states,
stream.residual if stream is not None else residual,
forward_batch=forward_batch,
dp_step=None,
)
return self._leave_ffn_output(
hidden_states,
residual,
stream,
self.plan._batch_steps(forward_batch).ffn_output.update,
)
@staticmethod
def _leave_ffn_output(hidden_states, residual, stream, update, declared_sum=None):
if stream is None:
return hidden_states, residual
stream.residual = residual
if update.at_producer:
return stream.write(hidden_states), stream
return stream.leave(hidden_states, update, declared_sum=declared_sum), stream
@staticmethod
def _declared_ffn_sum(steps, skipped_reduction):
# The input-scattered path leaves the sum for the next input's TP
# reduce-scatter. Other skip paths complete it in the output move.
if (
skipped_reduction
and not steps.returns_over_dp
and not steps.ffn_output_move_completes_sum
):
return SumGroup.TP
return None
def _local_token_move_can_go_to_next_layer(
self, forward_batch: ForwardBatch
) -> bool:
"""Whether the next layer's input can run this layer's move of its FFN
output back to this rank's tokens: the base postprocess scatter, when
the next layer's input also writes the output into the residual."""
steps = self.plan._batch_steps(forward_batch)
return steps.returns_over_dp and not steps.ffn_output.update.at_producer
def _postprocess_dp_step(
self, forward_batch: ForwardBatch
) -> Optional[Callable[[torch.Tensor, torch.Tensor, ForwardBatch], None]]:
"""The reduce-scatter that brings this layer's FFN output back to this
rank's tokens under attention DP; None when the base postprocess would
only scatter, or does not move tokens."""
steps = self.plan._batch_steps(forward_batch)
if not steps.returns_over_dp:
return None
return _reduce_and_redistribute_output_step(
forward_batch,
leaves_for_reduce_scatter=steps.ffn_output.leaves_for_reduce_scatter,
leaves_for_reduce_scatterv=steps.ffn_output.leaves_for_reduce_scatterv,
)
def _ffn_leaves_sum_to_reduce_scatter(
self, forward_batch: ForwardBatch, dp_step: Optional[Callable]
) -> bool:
"""Whether the FFN leaves its sum out because a reduce-scatter completes
it: the attention-DP one ``dp_step`` names, or the CP / input-scattered
one."""
steps = self.plan._batch_steps(forward_batch)
if dp_step is not None:
return True
if not steps.ffn_output.leaves_for_reduce_scatter:
return False
if steps.ffn_output_move_completes_sum:
return True
return get_attn_tp_context().input_scattered and not self.plan.is_last_layer
def ffn_reduction_group(self, forward_batch: ForwardBatch) -> GroupCoordinator:
"""The group this layer's FFN output owes its sum over: the MoE output's
group on a sparse layer, the TP group a dense MLP reduces over."""
return _sum_group(self.plan._batch_steps(forward_batch).ffn_output.group)
def _select_ffn_completion(self, forward_batch: ForwardBatch) -> FfnCompletion:
"""Decide once, before the FFN runs, what it skips and what completes its
output: the next layer's input, or this layer's postprocess step."""
dp_step = self._postprocess_dp_step(forward_batch)
mlp_reduce_scatter = self._ffn_leaves_sum_to_reduce_scatter(
forward_batch, dp_step
)
complete_now = partial(
self._complete_ffn_output_now,
forward_batch=forward_batch,
dp_step=dp_step,
)
steps = self.plan._batch_steps(forward_batch)
if not steps.ffn_output.leaves_for_next_layer:
return FfnCompletion(
defer_moe_finalize=False,
fuse_mlp_allreduce=False,
mlp_reduce_scatter=mlp_reduce_scatter,
complete=complete_now,
)
defer_moe_finalize = (
self.plan.fusions is not None
and self.plan.fusions.can_defer_finalize(self.plan, forward_batch)
)
# Producers declare remaining work independently of the kernel chosen
# by the consumer. Every handoff also carries an unfused completion.
fuse_mlp_allreduce = defer_moe_finalize or self._ffn_sum_moves_to_next_layer(
forward_batch, mlp_reduce_scatter=mlp_reduce_scatter, dp_step=dp_step
)
if fuse_mlp_allreduce:
group = self.ffn_reduction_group(forward_batch)
if steps.returns_over_dp:
# Under attention DP the next layer also brings the sum back to
# this rank's tokens.
wrap = partial(
UnreducedOutput,
reduce_and_redistribute=partial(
_all_reduce_then_to_local_tokens, group, forward_batch
),
)
else:
wrap = partial(UnreducedOutput, group=group)
complete = partial(_leave_to_next_layer, wrap)
elif (
dp_step is not None
and not self.plan.is_last_layer
and self._local_token_move_can_go_to_next_layer(forward_batch)
):
complete = partial(
_leave_to_next_layer,
partial(
UnreducedOutput,
reduce_and_redistribute=partial(
_to_local_tokens, dp_step, forward_batch
),
),
)
else:
complete = complete_now
return FfnCompletion(
defer_moe_finalize=defer_moe_finalize,
fuse_mlp_allreduce=fuse_mlp_allreduce,
mlp_reduce_scatter=mlp_reduce_scatter,
complete=complete,
)
def _complete_ffn_output_now(
self,
hidden_states: torch.Tensor,
residual: torch.Tensor,
*,
forward_batch: ForwardBatch,
dp_step: Optional[Callable],
) -> Tuple[torch.Tensor, torch.Tensor]:
"""This layer's postprocess, run with the attention-DP step already
chosen: the move back to where the next layer reads the FFN output, then
the write-back into the residual for a layer that does it itself."""
steps = self.plan._batch_steps(forward_batch)
if steps.ffn_output.transform is not None:
hidden_states = steps.ffn_output.transform.apply(hidden_states)
if steps.returns_over_dp:
hidden_states = _to_local_tokens(
dp_step or _redistribute_output, forward_batch, hidden_states
)
else:
hidden_states, residual = steps.ffn_output_move(
hidden_states=hidden_states,
residual=residual,
forward_batch=forward_batch,
)
update = steps.ffn_output.update
if residual is not None and update.at_producer:
hidden_states = update.update(hidden_states, residual)
residual = None
return hidden_states, residual
def mixer_exit(self, forward_batch: ForwardBatch) -> MixerExit:
"""Decide once whether this stage's mixer (an attention-like stage)
skips its output all-reduce. Use the result as a context manager around
the mixer, then call ``finish``."""
return MixerExit(self, forward_batch)
def ffn_exit(self, forward_batch: ForwardBatch) -> FfnExit:
"""Decide once how this layer's FFN output reduction completes. Use the
result as a context manager around the FFN call, then call ``finish``."""
return FfnExit(self, forward_batch)
def _ffn_sum_can_move_to_next_layer(self, forward_batch: ForwardBatch) -> bool:
# Under the MoE-CP all-gather the fusion path would skip postprocess_layer
# and its MoE-CP scatter, leaving hidden_states longer than the residual.
if (
(
self.plan.fusions is not None
and self.plan.fusions.requires_local_reduction
)
or is_enable_moe_cp_allgather()
or not self.plan._batch_steps(forward_batch).ffn_sum_is_movable
):
return False
# The fused residual+LN reduces over a single group. Hybrid EP+TP spans
# two disjoint groups; post_experts_all_reduce() merges them into one
# _TP reduction when moe_dp_size == 1, which the fused kernel can absorb.
# When merging is blocked, no single group covers both, so fusion stays off.
parallel = get_parallel()
if (
parallel.moe_ep_size > 1
and parallel.moe_tp_size > 1
and not can_merge_post_experts_all_reduce()
):
return False
if (
is_dp_attention_enabled()
and self.plan._speculative_algo is not None
and self.plan._speculative_algo.is_eagle()
):
return False
return not get_attn_tp_context().input_scattered
def _ffn_sum_moves_to_next_layer(
self,
forward_batch: ForwardBatch,
*,
mlp_reduce_scatter: bool,
dp_step: Optional[Callable],
) -> bool:
"""Whether the FFN leaves its output's all-reduce to the next layer's
input when no fused kernel takes it: when the next layer would run the
same all-reduce the FFN itself would have."""
return (
get_parallel().tp_size > 1
and not self.plan.is_last_layer
and self._ffn_sum_can_move_to_next_layer(forward_batch)
and _can_defer_ffn_reduction(forward_batch, self.plan)
and not mlp_reduce_scatter
# Under attention DP the next layer must also run postprocess's
# scatter back to this rank's tokens, and nothing more.
and (
not is_dp_attention_enabled()
or (
self._local_token_move_can_go_to_next_layer(forward_batch)
and dp_step is None
)
)
)
def _can_defer_ffn_reduction(forward_batch: ForwardBatch, boundary=None) -> bool:
"""Admit ordinary single-sum outputs and the existing fused LoRA path.
A fused-kernel fallback completes the partial-output contract selected
before the producer ran, without recomputing the producer's LoRA path.
"""
if not _ffn_has_tokens(forward_batch):
return False
if post_experts_sum_is_one_all_reduce():
return True
# LoRA-B is replicated and linear. TP1 shared experts add on rank zero
# when the sum is deferred. Preserve these fused paths, but keep their
# producer-side reduction order when no fused consumer is enabled.
if not (
(get_lora().enable_lora or envs.SGLANG_SHARED_EXPERT_TP1.get())
and get_moe_a2a_backend().is_none()
and not get_exec().comm.enable_quant_communications
and post_experts_reduction_group() is get_parallel().tp_group
):
return False
if (
boundary is not None
and boundary.fusions is not None
and boundary.fusions.can_defer_all_reduce(boundary, forward_batch)
):
return True
if apply_flashinfer_allreduce_fusion(forward_batch.input_ids.shape[0]):
return True
# Aiter also checks width and bytes, so use the actual residual storage.
residual = forward_batch.residual_stream.residual
return residual is not None and apply_aiter_all_reduce_fusion(
residual, forward_batch
)
class FfnCompletion(msgspec.Struct, frozen=True):
"""One FFN's reduction decision. The flags are published while the FFN runs;
``complete(hidden_states, residual)`` then either wraps the output for the
next layer's input to complete, or runs this layer's postprocess step."""
defer_moe_finalize: bool
fuse_mlp_allreduce: bool
mlp_reduce_scatter: bool
complete: Callable[[torch.Tensor, torch.Tensor], Tuple]
def _leave_to_next_layer(
wrap: Callable[[torch.Tensor], UnreducedOutput],
hidden_states: torch.Tensor,
residual: torch.Tensor,
) -> Tuple[UnreducedOutput, torch.Tensor]:
return wrap(hidden_states), residual
class MixerExit:
"""The scope that publishes a mixer's decision while it runs: inside the
``with`` block ``fuse_mlp_allreduce`` on ``get_forward()`` tells its
row-parallel output projection to skip the all-reduce. It skips when the
stage's output always leaves its sum (to an FFN stage, which completes it in
its input), and when its output declaration permits deferring the sum. The
consumer decides whether to fuse its completion with the input norm."""
__slots__ = (
"skips_reduction",
"_hands_on",
"_scope",
"_update",
"_declared_sum",
"_stream",
)
def __init__(self, communicator: OutputBoundary, forward_batch: ForwardBatch):
produced = communicator.plan._batch_steps(forward_batch).ffn_output
self._stream = None
self._update = produced.update
self._declared_sum = produced.group if produced.always_leaves else None
self._hands_on = (
produced.leaves_for_next_layer
and get_parallel().tp_size > 1
and not is_dp_attention_enabled()
and communicator._ffn_sum_can_move_to_next_layer(forward_batch)
)
self.skips_reduction = produced.always_leaves or self._hands_on
self._scope = get_forward().scoped(fuse_mlp_allreduce=self.skips_reduction)
def __enter__(self) -> MixerExit:
self._scope.__enter__()
return self
def __exit__(self, *exc_info):
return self._scope.__exit__(*exc_info)
def finish(self, hidden_states: torch.Tensor, residual=None):
"""The mixer's output: its partial sum to the FFN stage after it, as a
value that owes the sum to a mixer after it, or complete."""
if self._stream is not None:
residual = self._stream
if self._hands_on:
hidden_states = UnreducedOutput(
hidden_states, group=get_parallel().tp_group
)
if isinstance(residual, ResidualStream):
output = residual.leave(
hidden_states, self._update, declared_sum=self._declared_sum
)
return output if self._stream is not None else (output, residual)
return hidden_states
class FfnExit:
"""The scope that publishes an FfnCompletion while the FFN runs: inside the
``with`` block it is ``fuse_mlp_allreduce`` / ``mlp_reduce_scatter`` /
``defer_moe_finalize`` on ``get_forward()``."""
__slots__ = (
"communicator",
"forward_batch",
"defer_moe_finalize",
"fuse_mlp_allreduce",
"mlp_reduce_scatter",
"_complete",
"_update",
"_declared_sum",
"_scope",
"_stream",
)
def __init__(self, communicator: OutputBoundary, forward_batch: ForwardBatch):
self._stream = None
self.communicator = communicator
self.forward_batch = forward_batch
completion = communicator._select_ffn_completion(forward_batch)
self.defer_moe_finalize = completion.defer_moe_finalize
self.fuse_mlp_allreduce = completion.fuse_mlp_allreduce
self.mlp_reduce_scatter = completion.mlp_reduce_scatter
self._complete = completion.complete
steps = communicator.plan._batch_steps(forward_batch)
self._update = steps.ffn_output.update
self._declared_sum = communicator._declared_ffn_sum(
steps, completion.mlp_reduce_scatter
)
self._scope = get_forward().scoped(
fuse_mlp_allreduce=self.fuse_mlp_allreduce,
mlp_reduce_scatter=self.mlp_reduce_scatter,
defer_moe_finalize=self.defer_moe_finalize,
)
def __enter__(self) -> FfnExit:
self._scope.__enter__()
return self
def __exit__(self, *exc_info):
return self._scope.__exit__(*exc_info)
def finish(
self, hidden_states: torch.Tensor, residual: torch.Tensor = None
) -> Tuple[Union[torch.Tensor, UnreducedOutput], torch.Tensor]:
"""Leave the reduction to the next layer's input, or postprocess."""
if self._stream is not None:
residual = self._stream
stream = residual if isinstance(residual, ResidualStream) else None
if stream is not None:
residual = stream.residual
if not isinstance(hidden_states, torch.Tensor):
# A deferred MoE finalize handoff, consumed by the next prepare_attn.
assert self.defer_moe_finalize, "unrequested deferred MoE handoff"
else:
hidden_states, residual = self._complete(hidden_states, residual)
result = self.communicator._leave_ffn_output(
hidden_states, residual, stream, self._update, self._declared_sum
)
return result[0] if self._stream is not None else result
@@ -0,0 +1,705 @@
"""Construct one computing stage and its transport from a predecessor contract."""
from __future__ import annotations
from dataclasses import dataclass, replace
from types import MappingProxyType
from typing import Mapping, Optional
import msgspec
from sglang.srt.layers import layernorm_sp
from sglang.srt.layers.communicator.boundary import (
EdgeDecl,
StageDecl,
StageInput,
StageKind,
StageOutput,
_cp_moves,
)
from sglang.srt.layers.communicator.construction import (
BatchVariant,
StageEdges,
StagePlan,
_input_can_be_scattered,
_refuse_uncovered_cp_moe,
_use_ag_after_qlora,
)
from sglang.srt.layers.communicator.layout import (
Layout,
SumGroup,
TokenAxis,
_gathers_over_attention_cp,
_generic_prefill_cp_shards_tokens,
enable_moe_dense_fully_dp,
sparse_moe_gathers_over_moe_cp,
token_axis_sizes,
)
from sglang.srt.layers.communicator.ops import (
_hand_qkv_hook_its_input,
_hand_scattered_input_to_attention,
)
from sglang.srt.layers.communicator.output import OutputTransform
from sglang.srt.layers.communicator.residual import StageRead, StageUpdate
from sglang.srt.layers.communicator.residual.add_norm import (
ADD,
NORM_QUANT_READ,
NORM_READ,
)
from sglang.srt.layers.communicator.stage import StageCommunicator
from sglang.srt.layers.moe import is_moe_input_scattered_across_dp_ranks
from sglang.srt.runtime_context import get_exec, get_parallel
def _variants(*, ordinary_only=False):
yield BatchVariant.ORDINARY
if ordinary_only:
return
if _generic_prefill_cp_shards_tokens():
yield BatchVariant.CONTEXT_PARALLEL
if _input_can_be_scattered():
yield BatchVariant.INPUT_SCATTERED
if layernorm_sp.layernorm_sp_enabled():
yield BatchVariant.SEQUENCE_PARALLEL
def _rows(variant):
axes = token_axis_sizes(cp_active=variant is BatchVariant.CONTEXT_PARALLEL)
attention = Layout.sharded_over(
TokenAxis.ATTN_DP, TokenAxis.ATTN_CP, axis_sizes=axes
)
local = Layout.sharded_over(
TokenAxis.ATTN_DP, TokenAxis.ATTN_CP, TokenAxis.ATTN_TP_SCATTER, axis_sizes=axes
)
return axes, attention, local, Layout.sharded_over(axis_sizes=axes)
def _ffn_decl(
variant,
*,
sparse,
terminal=False,
next_sparse=False,
output=None,
read=NORM_READ,
update=ADD,
local_dense=True,
):
parallel = get_parallel()
reduction = get_exec().comm.boundary_reduction
if reduction not in ("ar", "rs", "rsv", "rs+rsv"):
raise ValueError(
"boundary_reduction must be resolved before model construction"
)
can_move_output = output is None or output.before_reduce_scatter
use_reduce_scatter = reduction in ("rs", "rs+rsv") and can_move_output
use_reduce_scatterv = reduction in ("rsv", "rs+rsv") and can_move_output
cp_shards = _generic_prefill_cp_shards_tokens()
axes, attention, local, full = _rows(variant)
on_local = (
is_moe_input_scattered_across_dp_ranks()
if sparse
else (local_dense and enable_moe_dense_fully_dp())
)
if parallel.attn_cp_size > 1 and sparse:
_refuse_uncovered_cp_moe(on_local, cp_shards)
on_cp = (
sparse
and parallel.attn_cp_size > 1
and parallel.moe_dp_size == parallel.attn_cp_size
and not _gathers_over_attention_cp()
)
group = SumGroup.MOE_OUTPUT if sparse else SumGroup.TP
if variant is BatchVariant.SEQUENCE_PARALLEL:
return (
StageDecl(
StageInput(
full,
gathers_itself=frozenset({TokenAxis.ATTN_TP_SCATTER}),
read=read,
),
StageOutput(local, update=update, transform=output),
),
local,
local,
)
if variant is BatchVariant.INPUT_SCATTERED:
scattered_residual = update.at_producer
residual = local if scattered_residual else attention
returned = local if scattered_residual and not terminal else attention
return (
StageDecl(
StageInput(full, read=read),
StageOutput(
attention,
group=group,
leaves_for_reduce_scatter=use_reduce_scatter
and (scattered_residual or not terminal),
update=update,
transform=output,
),
),
residual,
returned,
)
if cp_shards and _cp_moves().reduce_scatter is not None:
may_leave = variant is not BatchVariant.CONTEXT_PARALLEL
may_scatter = True
elif cp_shards or parallel.attn_dp_size > 1:
may_leave = may_scatter = parallel.attn_cp_size == 1 or on_cp
else:
may_leave = may_scatter = True
rows = (
local
if on_local
else Layout.sharded_over(
*((TokenAxis.ATTN_CP,) if on_cp else ()), axis_sizes=axes
)
)
output = (
StageOutput(rows, update=update, transform=output)
if on_local
else StageOutput(
rows,
group=group,
leaves_for_next_layer=may_leave
and not terminal
and not update.at_producer
and update.can_defer_across_layers
and output is None,
leaves_for_reduce_scatter=use_reduce_scatter and may_scatter,
leaves_for_reduce_scatterv=use_reduce_scatterv and may_leave,
update=update,
transform=output,
)
)
gathers_for_tbo = (
enable_moe_dense_fully_dp()
and get_exec().overlap.enable_two_batch_overlap
and not sparse
and bool(next_sparse)
)
returned = local if on_local and not terminal and not gathers_for_tbo else attention
return (
StageDecl(StageInput(rows, read=read), output),
local if on_local else attention,
returned,
)
@dataclass(frozen=True)
class StageDeclaration:
"""A computation and its input source, without an execution plan.
``previous`` consumes a producer's output. ``prepared_from`` reuses an
already-read input for a branch; the two sources are mutually exclusive.
Sources may be reconstructed locally, including across pipeline ranks.
"""
kind: StageKind
read: StageRead
update: StageUpdate
sparse: bool = False
terminal: bool = False
next_sparse: bool = False
output: Optional[OutputTransform] = None
ordinary_only: bool = False
# A mixer with its own exit publishes reduction decisions before compute.
mixer_exit: bool = False
next_kind: Optional[StageKind] = None
return_to_attention: bool = False
# Only declarations participate in construction, never executable stages.
previous: Optional[StageDeclaration] = None
prepared_from: Optional[StageDeclaration] = None
def __post_init__(self):
if self.previous is not None and self.prepared_from is not None:
raise ValueError("choose a previous output or a prepared branch input")
for source in (self.previous, self.prepared_from):
if source is not None and not isinstance(source, StageDeclaration):
raise TypeError("stage sources must be declarations")
@dataclass(frozen=True)
class StageConnection:
"""The producer's exit and consumer's entry for each batch variant.
Immutable, norm-free declarations. Independently constructed equivalent
connections work across pipeline partitions; object identity is irrelevant.
"""
producer: Optional[StageDeclaration]
consumer: Optional[StageDeclaration]
exits: Mapping[BatchVariant, EdgeDecl]
entries: Mapping[BatchVariant, EdgeDecl]
def __post_init__(self):
object.__setattr__(self, "exits", MappingProxyType(dict(self.exits)))
object.__setattr__(self, "entries", MappingProxyType(dict(self.entries)))
def declare_attn(
*,
previous: Optional[StageDeclaration] = None,
prepared_from: Optional[StageDeclaration] = None,
read=NORM_QUANT_READ,
update=ADD,
terminal=False,
ordinary_only=False,
mixer_exit=False,
next_kind=None,
):
return StageDeclaration(
StageKind.ATTENTION,
read,
update,
previous=previous,
prepared_from=prepared_from,
terminal=terminal,
ordinary_only=ordinary_only,
mixer_exit=mixer_exit,
next_kind=next_kind,
)
def declare_ffn(
*,
previous: Optional[StageDeclaration] = None,
prepared_from: Optional[StageDeclaration] = None,
sparse=False,
read=NORM_READ,
update=ADD,
terminal=False,
next_sparse=False,
output=None,
ordinary_only=False,
return_to_attention=False,
):
return StageDeclaration(
StageKind.FFN,
read,
update,
sparse=sparse,
previous=previous,
prepared_from=prepared_from,
terminal=terminal,
next_sparse=next_sparse,
output=output,
ordinary_only=ordinary_only,
return_to_attention=return_to_attention,
)
def _resolve(stage, variant):
axes, attention, local, full = _rows(variant)
if stage.kind is StageKind.FFN:
if stage.update.at_producer:
if stage.sparse and sparse_moe_gathers_over_moe_cp():
raise NotImplementedError(
"MHC does not support a MoE gathered over the MoE-CP group"
)
if get_parallel().attn_cp_size > 1 and _input_can_be_scattered():
raise NotImplementedError(
"MHC with input-scattered attention under attention CP"
)
declaration, residual, returned = _ffn_decl(
variant,
sparse=stage.sparse,
terminal=stage.terminal,
next_sparse=stage.next_sparse,
output=stage.output,
read=stage.read,
update=stage.update,
local_dense=not stage.return_to_attention,
)
if stage.return_to_attention:
returned = attention
return declaration, residual, returned
sp = variant is BatchVariant.SEQUENCE_PARALLEL
scattered = variant is BatchVariant.INPUT_SCATTERED
gathers = (
frozenset({TokenAxis.ATTN_TP_SCATTER})
if not stage.mixer_exit and (sp or scattered or _use_ag_after_qlora)
else frozenset()
)
owes = not sp and axes[TokenAxis.ATTN_TP_SCATTER] > 1
declaration = StageDecl(
StageInput(attention, gathers_itself=gathers, read=stage.read),
StageOutput(
local if sp else attention,
group=SumGroup.ATTN_TP if owes else None,
always_leaves=owes
and (not stage.mixer_exit or stage.next_kind is StageKind.FFN),
leaves_for_next_layer=owes
and stage.mixer_exit
and stage.next_kind is StageKind.ATTENTION,
update=stage.update,
),
)
return declaration, None, attention
def _connect(producer, consumer, *, residual_from=None):
"""Resolve producer and consumer declarations; None denotes an external endpoint.
A missing producer is the stack input; a missing consumer is a handoff
whose next read is bound separately (or the terminal output).
``residual_from`` names the boundary that placed the producer's residual.
It is needed for attention, whose residual can stay on finer rows than its
compute input. No executable stage participates in this construction.
"""
if producer is None and consumer is None:
raise ValueError("a boundary needs at least one declared side")
before, after = producer, consumer
ordinary_only = any(s.ordinary_only for s in (before, after) if s is not None)
exits, entries = {}, {}
for variant in _variants(ordinary_only=ordinary_only):
_, attention, local, _ = _rows(variant)
if before is None:
rows = local if variant is BatchVariant.SEQUENCE_PARALLEL else attention
owes = variant is BatchVariant.INPUT_SCATTERED
arrived = StageOutput(
rows,
group=SumGroup.TP if owes else None,
always_leaves=owes,
update=None,
)
residual = rows
capabilities = (True,)
else:
decl, during, returned = _resolve(before, variant)
if during is None:
if residual_from is not None:
if residual_from.consumer != producer:
raise ValueError("residual source must enter the producer")
during = residual_from.entries[variant].residual_to
elif before.mixer_exit:
during = attention
else:
raise ValueError(
"attention output needs its incoming residual placement"
)
exits[variant] = EdgeDecl(
decl.output, StageInput(returned), during, returned
)
if before.kind is StageKind.ATTENTION and not before.mixer_exit:
arrived, residual, capabilities = decl.output, during, ()
else:
owes = (
variant is BatchVariant.INPUT_SCATTERED
and not before.update.at_producer
)
carries = (before.mixer_exit or before.return_to_attention) and (
decl.output.always_leaves or decl.output.leaves_for_next_layer
)
arrived = StageOutput(
returned,
group=decl.output.group
if carries
else (SumGroup.TP if owes else None),
always_leaves=decl.output.always_leaves if carries else owes,
leaves_for_next_layer=decl.output.leaves_for_next_layer
if carries
else False,
update=None,
)
residual, capabilities = returned, (before.update.adds_plainly,)
if after is None:
continue
decl, during, _ = _resolve(after, variant)
if after.kind is StageKind.ATTENTION:
during = (
Layout(residual.sharded | {TokenAxis.ATTN_TP_SCATTER})
if variant is BatchVariant.INPUT_SCATTERED
and arrived.always_leaves
and arrived.update is None
and not after.mixer_exit
else residual
)
elif after.return_to_attention:
during = (
decl.input.layout
if residual.sharded <= decl.input.layout.sharded
else residual
)
joins = (
variant is BatchVariant.INPUT_SCATTERED
and arrived.update is not None
and arrived.update.adds_plainly
and after.kind is StageKind.FFN
)
edge = EdgeDecl(
arrived,
decl.input,
residual,
during,
residual_joins_sum=joins,
update_capabilities=capabilities,
)
entries[variant] = edge
if (
before is not None
and before.kind is StageKind.ATTENTION
and not before.mixer_exit
):
exits[variant] = edge
return StageConnection(producer, consumer, exits, entries)
def _fork_input(prepared, consumer):
"""Place an already-read input onto a branch's computation rows.
The branch adapter performs the move and forks the stream. It does not
execute the branch's input norm again.
"""
entries = {}
for variant in _variants(ordinary_only=consumer.ordinary_only):
source = prepared.entries[variant]
declaration, during, _ = _resolve(consumer, variant)
entries[variant] = EdgeDecl(
StageOutput(source.need.layout, update=None),
declaration.input,
source.residual_to,
during,
update_capabilities=(True,),
)
return StageConnection(prepared.consumer, consumer, {}, entries)
def _incoming(stage):
if stage.prepared_from is not None:
return _fork_input(_incoming(stage.prepared_from), stage)
previous = stage.previous
# A normal attention may retain a finer residual than its output rows.
# FFN and mixer exits declare their returned residual placement directly.
source = (
_incoming(previous)
if previous is not None
and previous.kind is StageKind.ATTENTION
and not previous.mixer_exit
else None
)
return _connect(
previous,
stage,
residual_from=source,
)
def _connections(stage, following=None):
if following is not None and following.previous != stage:
raise ValueError("the following declaration must consume this stage's output")
incoming = _incoming(stage)
outgoing = _connect(
stage,
following,
residual_from=incoming,
)
return incoming, outgoing
def _bind_stage(declaration, norm, incoming, outgoing, **options):
if incoming.consumer != declaration or outgoing.producer != declaration:
raise ValueError("connections do not match the stage declaration")
if incoming.entries.keys() != outgoing.exits.keys():
raise ValueError("incoming and outgoing batch variants disagree")
if declaration.update.at_producer:
if declaration.sparse and sparse_moe_gathers_over_moe_cp():
raise NotImplementedError(
"MHC does not support a MoE gathered over the MoE-CP group"
)
if get_parallel().attn_cp_size > 1 and _input_can_be_scattered():
raise NotImplementedError(
"MHC with input-scattered attention under attention CP"
)
variants = {}
for variant, edge in incoming.entries.items():
if (
declaration.update.at_producer
and TokenAxis.ATTN_CP
in edge.produced.layout.sharded - edge.need.layout.sharded
):
raise NotImplementedError("MHC with a gather over attention CP")
handoff = None
if declaration.kind is StageKind.ATTENTION:
handoff = (
_hand_scattered_input_to_attention
if variant is BatchVariant.INPUT_SCATTERED
and not declaration.update.adds_plainly
else _hand_qkv_hook_its_input
)
moves = _cp_moves() if variant is BatchVariant.CONTEXT_PARALLEL else None
variants[variant] = StageEdges(edge, outgoing.exits[variant], handoff, moves)
plan = StagePlan(
declaration.kind,
norm,
variants,
enters_stack=incoming.producer is None,
prepared_input=declaration.prepared_from is not None,
terminal=declaration.terminal,
fixed_output=declaration.kind is StageKind.ATTENTION
and not declaration.mixer_exit,
is_sparse=declaration.sparse,
**options,
)
if declaration.kind is StageKind.ATTENTION and outgoing.consumer is not None:
# Layout eligibility comes from the connected consumer, not a mutable
# link to its execution plan. Kernel binding remains consumer-owned.
from sglang.srt.layers.communicator.boundary import make_boundary
plan._fusion_rows = {
v: make_boundary(
edge,
cp_moves=_cp_moves() if v is BatchVariant.CONTEXT_PARALLEL else None,
force_layernorm_before_gather=options.get(
"force_layernorm_before_dp_gather", False
),
).input_rows
for v, edge in outgoing.entries.items()
}
return StageCommunicator(plan, declaration.kind, norm, declaration=declaration)
def make_attn_stage(
*,
declaration,
norm,
following: Optional[StageDeclaration] = None,
qkv_latent_func=None,
force_layernorm_before_dp_gather=False,
enable_fused_ar_quant=False,
fused_ar_quant_keep_bf16=False,
residual_in_hidden=False,
fusions=None,
):
"""Resolve this attention's boundaries, then bind its norm and hooks.
``following`` supplies the consumer declaration for a stage inside a
layer. With no following stage, the output is handed across a layer or
stack boundary; the receiver binds its own read independently.
"""
if declaration.kind is not StageKind.ATTENTION:
raise TypeError("make_attn_stage requires an attention declaration")
incoming, outgoing = _connections(declaration, following)
return _bind_stage(
declaration,
norm,
incoming,
outgoing,
qkv_latent_func=qkv_latent_func,
force_layernorm_before_dp_gather=force_layernorm_before_dp_gather,
enable_fused_ar_quant=enable_fused_ar_quant,
fused_ar_quant_keep_bf16=fused_ar_quant_keep_bf16,
residual_in_hidden=residual_in_hidden,
fusions=fusions,
)
def make_ffn_stage(
*,
declaration,
norm,
following: Optional[StageDeclaration] = None,
force_layernorm_before_dp_gather=False,
fusions=None,
):
"""Resolve this FFN's boundaries and bind its norm; EP stays inside compute."""
if declaration.kind is not StageKind.FFN:
raise TypeError("make_ffn_stage requires an FFN declaration")
incoming, outgoing = _connections(declaration, following)
return _bind_stage(
declaration,
norm,
incoming,
outgoing,
force_layernorm_before_dp_gather=force_layernorm_before_dp_gather,
residual_in_hidden=declaration.update.at_producer,
fusions=fusions,
)
def make_stages(*stages, previous=None, prepared_from=None, terminal=False):
"""Bind a local linear sequence and return its independent boundaries.
Each item is ``(declaration, norm)`` or ``(declaration, norm, options)``.
Options are the stage-specific keyword arguments of ``make_attn_stage``
or ``make_ffn_stage``; they are consumed only during construction.
Declarations in the sequence have no input source: this function connects
them in order. Only the final boundary is terminal when requested.
``previous`` describes an external producer's output. ``prepared_from``
describes an already-read branch input, obtainable from the source
boundary's immutable ``declaration``. Neither takes an executable boundary.
No sequence object or runtime routing is retained.
"""
if not stages:
raise ValueError("make_stages needs at least one stage")
if previous is not None and prepared_from is not None:
raise ValueError("choose a previous output or a prepared branch input")
declarations = []
bindings = []
for index, item in enumerate(stages):
if len(item) not in (2, 3):
raise ValueError("a stage needs (declaration, norm[, options])")
declaration, norm = item[:2]
if not isinstance(declaration, StageDeclaration):
raise TypeError("make_stages requires stage declarations")
if declaration.previous is not None or declaration.prepared_from is not None:
raise ValueError("pass input sources to make_stages, not its declarations")
if declaration.terminal:
raise ValueError("pass terminal to make_stages, not its declarations")
declaration = replace(
declaration,
previous=previous if index == 0 else declarations[-1],
prepared_from=prepared_from if index == 0 else None,
terminal=terminal and index == len(stages) - 1,
next_kind=stages[index + 1][0].kind
if declaration.mixer_exit and index + 1 < len(stages)
else declaration.next_kind,
)
declarations.append(declaration)
bindings.append((norm, item[2] if len(item) == 3 else {}))
boundaries = []
for index, (declaration, (norm, options)) in enumerate(zip(declarations, bindings)):
build = (
make_attn_stage
if declaration.kind is StageKind.ATTENTION
else make_ffn_stage
)
boundaries.append(
build(
declaration=declaration,
norm=norm,
following=declarations[index + 1]
if index + 1 < len(declarations)
else None,
**options,
)
)
for producer, consumer in zip(boundaries, boundaries[1:]):
if (
producer.kind is not StageKind.ATTENTION
or producer.declaration.mixer_exit
or consumer.kind is not StageKind.FFN
):
continue
for variant, steps in producer.plan._paths.items():
predicate = consumer.plan._paths[variant].ffn.preserves_residual
if predicate is not None:
producer.plan._paths[variant] = msgspec.structs.replace(
steps,
attention=msgspec.structs.replace(
steps.attention, capture_preserves_residual=predicate
),
)
producer.plan._steps = producer.plan._paths[BatchVariant.ORDINARY]
producer.plan._cp_steps = producer.plan._paths.get(
BatchVariant.CONTEXT_PARALLEL
)
producer.plan._input_scattered_steps = producer.plan._paths.get(
BatchVariant.INPUT_SCATTERED
)
producer.plan._sp_steps = producer.plan._paths.get(
BatchVariant.SEQUENCE_PARALLEL
)
return tuple(boundaries)
@@ -0,0 +1,148 @@
# Copyright 2023-2024 SGLang Team
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
"""Select fused completion and read kernels for each consumer."""
from functools import partial
from typing import Callable, Optional, Tuple
import torch
from sglang.srt.layers.communicator.boundary import FusedMlpInput
from sglang.srt.layers.communicator.layout import SumGroup
from sglang.srt.layers.communicator.output import UnreducedOutput
from sglang.srt.layers.communicator.residual.add_norm import (
apply_aiter_all_reduce_fusion,
apply_flashinfer_allreduce_fusion,
)
from sglang.srt.layers.layernorm import GemmaRMSNorm, RMSNorm
from sglang.srt.layers.moe import post_experts_reduction_group
from sglang.srt.model_executor.forward_batch_info import ForwardBatch
from sglang.srt.runtime_context import get_parallel
from sglang.srt.utils import get_bool_env_var, is_hip
_use_aiter = get_bool_env_var("SGLANG_USE_AITER") and is_hip()
def attention_fusions(plan) -> Tuple[Callable, ...]:
"""The fused kernels that complete what the previous layer left together
with the residual update and the input norm, in the order they are tried.
Each takes (owed, residual, forward_batch, post_residual_addition) and
returns None when it does not take the batch. They add the residual
plainly; the boundary tries them only when the update it writes in is a
plain add. A backend's come first."""
given = plan.fusions.attention_input(plan) if plan.fusions else ()
if not hasattr(plan.norm, "forward_with_allreduce_fusion"):
return given
plan._attn_input_fuses_quant = (
plan.enable_fused_ar_quant
and _use_aiter
and hasattr(plan.norm, "forward_with_allreduce_fusion_quant_per_group")
)
return (*given, partial(complete_attention_input, plan))
def complete_attention_input(
plan,
owed: UnreducedOutput,
residual: torch.Tensor,
forward_batch: ForwardBatch,
post_residual_addition: Optional[torch.Tensor],
) -> Optional[Tuple[torch.Tensor, torch.Tensor]]:
"""Complete the sum the previous layer left, add it to the residual and
apply the input norm in one aiter or flashinfer kernel; None when the
kernel does not take this batch. The result is not quantized for
``quant_format``."""
if (
not isinstance(owed, UnreducedOutput)
# The kernel does not add it.
or post_residual_addition is not None
# The kernel reduces over the MoE output's group.
or owed.group is not post_experts_reduction_group()
):
return None
hidden_states = owed.partial
if not (
apply_aiter_all_reduce_fusion(hidden_states, forward_batch)
or apply_flashinfer_allreduce_fusion(hidden_states.shape[0])
):
return None
if plan._attn_input_fuses_quant:
# Falls back to AR+RMSNorm + separate quant internally when the
# fully-fused kernel cannot service the shape.
quant_result = plan.norm.forward_with_allreduce_fusion_quant_per_group(
hidden_states,
residual,
use_attn_tp_group=False,
keep_bf16=plan.fused_ar_quant_keep_bf16,
)
if quant_result is not None:
return quant_result
return plan.norm.forward_with_allreduce_fusion(
hidden_states, residual, use_attn_tp_group=False
)
def ffn_fusions(plan) -> Tuple["FusedMlpInput", ...]:
"""The fused kernels that can take the attention -> FFN steps, in the
order they are tried. They add the residual plainly; the boundary tries
them only when the update it writes in is a plain add. A backend's come
first."""
given = plan.fusions.ffn_input(plan) if plan.fusions else ()
if not hasattr(plan.norm, "forward_with_allreduce_fusion"):
return given
return (
*given,
FusedMlpInput(
completes=SumGroup.ATTN_TP,
run=partial(complete_ffn_input, plan),
preserves_residual=(
flashinfer_preserves_residual
if isinstance(plan.norm, (RMSNorm, GemmaRMSNorm))
and type(plan.norm).forward_with_allreduce_fusion
in (
RMSNorm.forward_with_allreduce_fusion,
GemmaRMSNorm.forward_with_allreduce_fusion,
)
else None
),
),
)
def flashinfer_preserves_residual(value, forward_batch):
"""The CUDA wrapper returns a fresh residual, including backend fallback."""
return (
not _use_aiter
and get_parallel().attn_tp_size > 1
and apply_flashinfer_allreduce_fusion(value.shape[0])
)
def complete_ffn_input(
plan,
hidden_states: torch.Tensor,
residual: torch.Tensor,
forward_batch: ForwardBatch,
) -> Optional[Tuple[torch.Tensor, torch.Tensor]]:
"""The attention-TP all-reduce, residual add and norm in one aiter or
flashinfer kernel; None when neither takes the batch."""
if not (
apply_aiter_all_reduce_fusion(hidden_states, forward_batch)
or apply_flashinfer_allreduce_fusion(hidden_states.shape[0])
):
return None
return plan.norm.forward_with_allreduce_fusion(
hidden_states, residual, use_attn_tp_group=True
)
+27 -93
View File
@@ -42,8 +42,8 @@ from sglang.srt.layers.communicator.boundary import (
sequence_parallel_layer_sides,
with_residual,
)
from sglang.srt.layers.communicator.exit import _reduce_and_redistribute_output_step
from sglang.srt.layers.communicator.layout import (
CommunicateContext,
Layout,
SumGroup,
TokenAxis,
@@ -56,6 +56,7 @@ from sglang.srt.layers.communicator.layout import (
sparse_moe_gathers_over_moe_cp,
token_axis_sizes,
)
from sglang.srt.layers.communicator.legacy_stage import StageCommunicator
from sglang.srt.layers.communicator.ops import (
CommunicateSimpleFn,
_all_reduce_then_to_local_tokens,
@@ -63,7 +64,6 @@ from sglang.srt.layers.communicator.ops import (
_hand_scattered_input_to_attention,
_mlp_input_without_dp,
_redistribute_output,
_reduce_and_redistribute_output_step,
_to_local_tokens,
move_rows,
)
@@ -189,79 +189,6 @@ def _can_defer_ffn_reduction(forward_batch: ForwardBatch, boundary=None) -> bool
)
class StageCommunicator:
"""One stage of a layer, its attention or its FFN: the boundary into it,
run with the stage's norm on the steps the layer chose for the batch."""
def __init__(self, layer: "LayerCommunicator", name: str, norm_name: str):
self._layer = layer
self._name = name
self._norm_name = norm_name
@property
def norm(self) -> torch.nn.Module:
"""The norm the stage reads its input with, one of the layer's."""
return getattr(self._layer, self._norm_name)
def entry(
self, forward_batch: ForwardBatch, steps: Optional[BoundarySteps] = None
) -> StageEntry:
if steps is None:
steps = self._layer._batch_steps(forward_batch)
entry = getattr(steps, self._name)
if entry is None:
raise NotImplementedError(
f"a layer that is one stage has no {self._name} stage"
)
return entry
def prepare(
self,
hidden_states,
residual,
forward_batch: ForwardBatch,
steps: Optional[BoundarySteps] = None,
**call,
):
"""The stage's input and the residual, from the previous stage's output
(``call``: what the stage's read takes, e.g. the attention's
``quant_format``)."""
entry = self.entry(forward_batch, steps)
context = self._layer._context
stream = residual_batch.current(forward_batch)
if residual is not stream:
raise RuntimeError("residual alias belongs to a different invocation")
if (
stream.pending is None
and stream.residual is None
and entry.input_sum is not None
):
hidden_states = stream.leave(
hidden_states, ADD, declared_sum=entry.input_sum
)
if stream.pending is not None:
call["update"] = stream.pending.update
call["pending"] = stream.pending
if stream.pending is None and stream.residual is not None:
call["written"] = True
hidden_states, residual = stream.input(hidden_states)
hidden_states, residual = entry.prepare(
hidden_states, residual, forward_batch, self.norm, context, **call
)
if entry.input_move is not None:
hidden_states = entry.input_move(
hidden_states=hidden_states,
forward_batch=forward_batch,
context=context,
)
if entry.handoff is not None:
hidden_states = entry.handoff(
hidden_states, forward_batch, self._layer.qkv_latent_func
)
stream.write(residual)
return hidden_states, stream
class LayerFusions(Protocol):
"""The fused kernels a fusion backend gives a layer, bound to it and tried
before the layer's own: at its attention input (each takes what the previous
@@ -336,10 +263,6 @@ class LayerCommunicator:
self._residual = residual
self.fusions = fusions
self._context = CommunicateContext.init_new()
self._context.force_layernorm_before_dp_gather = (
force_layernorm_before_dp_gather
)
# The fused kernels every batch's attention input tries first.
self._attn_input_fusions = self._select_attn_input_fusions()
self._speculative_algo = SpeculativeAlgorithm.from_string(
@@ -799,7 +722,6 @@ class LayerCommunicator:
else move(
hidden_states=residual_value,
forward_batch=forward_batch,
context=self._context,
)
)
keeps = self.attn.entry(forward_batch).capture_preserves_residual
@@ -850,7 +772,7 @@ class LayerCommunicator:
)
if get_forward().sp_active:
hidden_states = layernorm_sp.sp_entry_scatter(hidden_states)
return self.attn.prepare(
return self.attn._prepare(
hidden_states,
residual,
forward_batch,
@@ -979,8 +901,6 @@ class LayerCommunicator:
):
steps = self._batch_steps(forward_batch)
self.publish_mlp_lora_layout(steps)
if cache is not None:
self._context.cache = cache
if isinstance(residual, ResidualStream) and self.stage_edges is None:
# The paired boundary records the attention contribution once.
@@ -989,12 +909,13 @@ class LayerCommunicator:
self._residual.attention_update,
declared_sum=steps.ffn.input_sum,
)
return self.ffn.prepare(
return self.ffn._prepare(
hidden_states,
residual,
forward_batch,
steps,
update=self._residual.attention_update,
cache=cache,
)
def maybe_prefetch_next_full_attention_kv(
@@ -1180,7 +1101,6 @@ class LayerCommunicator:
hidden_states=hidden_states,
residual=residual,
forward_batch=forward_batch,
context=self._context,
allow_reduce_scatter=self.allow_reduce_scatter,
is_layer_sparse=self.layer_facts.is_layer_sparse,
)
@@ -1317,7 +1237,7 @@ class LayerCommunicator:
input when no fused kernel takes it: when the next layer would run the
same all-reduce the FFN itself would have."""
return (
self._context.tp_size > 1
get_parallel().tp_size > 1
and not self.is_last_layer
and self._ffn_sum_can_move_to_next_layer(forward_batch)
and _can_defer_ffn_reduction(forward_batch, self)
@@ -1361,15 +1281,23 @@ class MixerExit:
its input), and when its output declaration permits deferring the sum. The
consumer decides whether to fuse its completion with the input norm."""
__slots__ = ("skips_reduction", "_hands_on", "_scope", "_update", "_declared_sum")
__slots__ = (
"skips_reduction",
"_hands_on",
"_scope",
"_update",
"_declared_sum",
"_stream",
)
def __init__(self, communicator: LayerCommunicator, forward_batch: ForwardBatch):
produced = communicator._batch_steps(forward_batch).ffn_output
self._stream = None
self._update = produced.update
self._declared_sum = produced.group if produced.always_leaves else None
self._hands_on = (
produced.leaves_for_next_layer
and communicator._context.tp_size > 1
and get_parallel().tp_size > 1
and not is_dp_attention_enabled()
and communicator._ffn_sum_can_move_to_next_layer(forward_batch)
)
@@ -1386,14 +1314,17 @@ class MixerExit:
def finish(self, hidden_states: torch.Tensor, residual=None):
"""The mixer's output: its partial sum to the FFN stage after it, as a
value that owes the sum to a mixer after it, or complete."""
if self._stream is not None:
residual = self._stream
if self._hands_on:
hidden_states = UnreducedOutput(
hidden_states, group=get_parallel().tp_group
)
if isinstance(residual, ResidualStream):
return residual.leave(
output = residual.leave(
hidden_states, self._update, declared_sum=self._declared_sum
), residual
)
return output if self._stream is not None else (output, residual)
return hidden_states
@@ -1412,9 +1343,11 @@ class FfnExit:
"_update",
"_declared_sum",
"_scope",
"_stream",
)
def __init__(self, communicator: LayerCommunicator, forward_batch: ForwardBatch):
self._stream = None
self.communicator = communicator
self.forward_batch = forward_batch
completion = communicator._select_ffn_completion(forward_batch)
@@ -1441,7 +1374,7 @@ class FfnExit:
return self._scope.__exit__(*exc_info)
def finish(
self, hidden_states: torch.Tensor, residual: torch.Tensor
self, hidden_states: torch.Tensor, residual: torch.Tensor = None
) -> Tuple[Union[torch.Tensor, UnreducedOutput], torch.Tensor]:
"""Leave the reduction to the next layer's input, or postprocess."""
stream = residual_batch.current(self.forward_batch)
@@ -1452,9 +1385,10 @@ class FfnExit:
assert self.defer_moe_finalize, "unrequested deferred MoE handoff"
else:
hidden_states, residual = self._complete(hidden_states, residual)
return self.communicator._leave_ffn_output(
result = self.communicator._leave_ffn_output(
hidden_states, residual, stream, self._update, self._declared_sum
)
return result[0] if self._stream is not None else result
class MHCLayerCommunicator(LayerCommunicator):
@@ -1501,7 +1435,7 @@ class MHCLayerCommunicator(LayerCommunicator):
residual=self.mhc.layer_residual(),
)
# MHC has not been run with input-scattered attention under attention CP.
if self._context.attn_cp_size > 1 and self._input_can_be_scattered():
if get_parallel().attn_cp_size > 1 and self._input_can_be_scattered():
raise NotImplementedError(
"MHCLayerCommunicator with input-scattered attention under attention CP"
)
@@ -14,7 +14,6 @@
"""Token axes, layouts and process groups of what a layer hands across its
boundaries, and the configuration and batch facts they depend on."""
from dataclasses import dataclass
from enum import Enum, auto
from typing import Dict, FrozenSet, List, Mapping, Optional
@@ -94,38 +93,6 @@ def _ffn_has_tokens(forward_batch: ForwardBatch) -> bool:
return _batch_size(forward_batch) > 0
@dataclass
class CommunicateContext:
attn_tp_rank: int
attn_tp_size: int
attn_dp_size: int
attn_cp_rank: int
attn_cp_size: int
tp_size: int
cache = None
tp_rank: int
force_layernorm_before_dp_gather: bool = False
@classmethod
def init_new(cls):
attn_tp_rank = get_parallel().attn_tp_rank
attn_tp_size = get_parallel().attn_tp_size
attn_dp_size = get_parallel().attn_dp_size
attn_cp_size = get_parallel().attn_cp_size
attn_cp_rank = get_parallel().attn_cp_rank
tp_size = get_parallel().tp_size
tp_rank = get_parallel().tp_rank
return cls(
attn_tp_rank=attn_tp_rank,
attn_tp_size=attn_tp_size,
attn_dp_size=attn_dp_size,
attn_cp_rank=attn_cp_rank,
attn_cp_size=attn_cp_size,
tp_size=tp_size,
tp_rank=tp_rank,
)
def moe_cp_gathered_rows(forward_batch: ForwardBatch) -> Optional[List[int]]:
"""The real rows each MoE-CP rank contributes when this batch's FFN input is
gathered across the MoE-CP group, or None when it is not: only a context
@@ -0,0 +1,160 @@
# Copyright 2023-2024 SGLang Team
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
"""A stage's model-facing input and output boundary."""
from typing import TYPE_CHECKING, Optional
from sglang.srt.layers.communicator.boundary import BoundarySteps, StageEntry
from sglang.srt.layers.communicator.residual.add_norm import ADD
from sglang.srt.layers.communicator.residual.batch import current
from sglang.srt.model_executor.forward_batch_info import ForwardBatch
if TYPE_CHECKING:
from sglang.srt.layers.communicator.layer import LayerCommunicator
class StageCommunicator:
"""One stage of a layer, its attention or its FFN: the boundary into it,
run with the stage's norm on the steps the layer chose for the batch."""
def __init__(self, layer: "LayerCommunicator", name: str, norm_name: str):
self._layer = layer
self._name = name
self.norm = getattr(layer, norm_name)
def entry(
self, forward_batch: ForwardBatch, steps: Optional[BoundarySteps] = None
) -> StageEntry:
if steps is None:
steps = self._layer._batch_steps(forward_batch)
entry = getattr(steps, self._name)
if entry is None:
raise NotImplementedError(
f"a layer that is one stage has no {self._name} stage"
)
return entry
def _prepare(
self,
hidden_states,
residual,
forward_batch: ForwardBatch,
steps: Optional[BoundarySteps] = None,
**call,
):
"""The stage's input and the residual, from the previous stage's output
(``call``: what the stage's read takes, e.g. the attention's
``quant_format``)."""
entry = self.entry(forward_batch, steps)
stream = current(forward_batch)
if residual is not stream:
raise RuntimeError("residual alias belongs to a different invocation")
if (
stream.pending is None
and stream.residual is None
and entry.input_sum is not None
):
hidden_states = stream.leave(
hidden_states, ADD, declared_sum=entry.input_sum
)
if stream.pending is not None:
call["update"] = stream.pending.update
call["pending"] = stream.pending
if stream.pending is None and stream.residual is not None:
call["written"] = True
hidden_states, residual = stream.input(hidden_states)
hidden_states, residual = entry.prepare(
hidden_states, residual, forward_batch, self.norm, **call
)
if entry.input_move is not None:
hidden_states = entry.input_move(
hidden_states=hidden_states,
forward_batch=forward_batch,
)
if entry.handoff is not None:
hidden_states = entry.handoff(
hidden_states, forward_batch, self._layer.qkv_latent_func
)
stream.write(residual)
return hidden_states, stream
@property
def input_rows(self):
return self._layer.input_rows
@property
def input_on_attention_tp_slices(self):
return self._layer.input_on_attention_tp_slices
def prepare(self, hidden_states, forward_batch, *, cache=None, **call):
stream = current(forward_batch)
if self._name == "attention":
hidden_states, stream = (
self._layer.prepare_attn_and_capture_last_layer_outputs(
hidden_states, stream, forward_batch, **call
)
)
else:
steps = self._layer._batch_steps(forward_batch)
self._layer.publish_mlp_lora_layout(steps)
if cache is not None:
call["cache"] = cache
hidden_states, stream = self._prepare(
hidden_states, stream, forward_batch, steps, **call
)
forward_batch.residual_stream = stream
return hidden_states
def finish(self, hidden_states, forward_batch):
"""Hand this attention's actual contribution to the following stage."""
if self._name != "attention" or self._layer.stage_edges is not None:
raise RuntimeError(
"use the stage exit scope for an FFN or single-stage mixer"
)
return current(forward_batch).leave(
hidden_states,
self._layer._residual.attention_update,
declared_sum=self._layer._batch_steps(forward_batch).ffn.input_sum,
)
def exit(self, forward_batch):
result = (
self._layer.ffn_exit(forward_batch)
if self._name == "ffn"
else self._layer.mixer_exit(forward_batch)
)
result._stream = current(forward_batch)
return result
def from_pp(self, tensors, forward_batch, **kwargs):
hidden_states, stream = self._layer.from_pp(tensors, forward_batch, **kwargs)
forward_batch.residual_stream = stream
return hidden_states
def snapshot(self, hidden_states, forward_batch, *, at_input=False):
return self._layer.snapshot(
hidden_states, current(forward_batch), at_input=at_input
)
def capture_output(self, hidden_states, forward_batch, **kwargs):
return self._layer.capture_output(
hidden_states, current(forward_batch), **kwargs
)
def postprocess(self, hidden_states, forward_batch):
hidden_states, stream = self._layer.postprocess_layer(
hidden_states, current(forward_batch), forward_batch
)
forward_batch.residual_stream = stream
return hidden_states
+58 -112
View File
@@ -38,7 +38,6 @@ from sglang.srt.layers.communicator.adapters.context_parallel import (
dsa_cp_reduce_scatter_hidden_states,
)
from sglang.srt.layers.communicator.layout import (
CommunicateContext,
Layout,
SumGroup,
TokenAxis,
@@ -56,7 +55,6 @@ from sglang.srt.layers.communicator.residual.stream import DeclaredSum
from sglang.srt.layers.dp_attention import (
attn_tp_all_gather_into_tensor,
attn_tp_reduce_scatter_tensor,
can_use_dp_reduce_scatter,
dp_gather_partial,
dp_gather_replicate,
dp_reduce_scatter_tensor,
@@ -69,7 +67,6 @@ from sglang.srt.layers.dp_attention import (
is_allocation_symmetric,
moe_cp_all_gather_into_tensor,
)
from sglang.srt.layers.moe import should_use_dp_reduce_scatterv
from sglang.srt.model_executor.forward_batch_info import ForwardBatch
from sglang.srt.runtime_context import get_exec, get_parallel
from sglang.srt.utils import (
@@ -86,28 +83,30 @@ if _is_npu:
def tp_reduce_scatter(
hidden_states: torch.Tensor,
residual: Optional[torch.Tensor],
context: "CommunicateContext",
) -> Tuple[torch.Tensor, Optional[torch.Tensor]]:
"""Module-level so MHC communicators can reuse it without a
``LayerCommunicator`` instance."""
"""Complete a TP sum onto each rank's token slice."""
parallel = get_parallel()
if hidden_states.shape[0] == 0:
return hidden_states, hidden_states
assert hidden_states.shape[0] % context.tp_size == 0, (
f"Expected total tokens {hidden_states.shape[0]} % tp_size {context.tp_size} to be 0"
assert hidden_states.shape[0] % parallel.tp_size == 0, (
f"Expected total tokens {hidden_states.shape[0]} % tp_size {parallel.tp_size} to be 0"
)
local_tokens = hidden_states.shape[0] // context.tp_size
local_tokens = hidden_states.shape[0] // parallel.tp_size
output = hidden_states.new_empty(local_tokens, *hidden_states.shape[1:])
get_parallel().tp_group.reduce_scatter_tensor(output, hidden_states)
parallel.tp_group.reduce_scatter_tensor(output, hidden_states)
if residual is not None:
residual = residual.tensor_split(context.tp_size)[context.tp_rank]
residual = residual.tensor_split(parallel.tp_size)[parallel.tp_rank]
return output, residual
def tp_slice(hidden_states, residual, context):
def tp_slice(hidden_states, residual):
"""The rows reduce-scatter would return, when the sum is already complete."""
hidden_states = hidden_states.tensor_split(context.tp_size)[context.tp_rank].clone()
parallel = get_parallel()
hidden_states = hidden_states.tensor_split(parallel.tp_size)[
parallel.tp_rank
].clone()
if residual is not None:
residual = residual.tensor_split(context.tp_size)[context.tp_rank]
residual = residual.tensor_split(parallel.tp_size)[parallel.tp_rank]
return hidden_states, residual
@@ -116,7 +115,6 @@ class CommunicateSimpleFn:
def _trivial(
hidden_states: torch.Tensor,
forward_batch: ForwardBatch,
context: CommunicateContext,
) -> torch.Tensor:
return hidden_states
@@ -124,18 +122,18 @@ class CommunicateSimpleFn:
def _scattered_to_tp_attn_full(
hidden_states: Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]],
forward_batch: ForwardBatch,
context: CommunicateContext,
) -> Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]]:
parallel = get_parallel()
if isinstance(hidden_states, tuple):
gathered_hidden_states = []
for local_hidden_states in hidden_states:
with use_symmetric_memory(
get_parallel().tp_group,
parallel.tp_group,
disabled=not is_allocation_symmetric(),
):
output = torch.empty(
(
local_hidden_states.shape[0] * context.attn_tp_size,
local_hidden_states.shape[0] * parallel.attn_tp_size,
*local_hidden_states.shape[1:],
),
dtype=local_hidden_states.dtype,
@@ -152,10 +150,11 @@ class CommunicateSimpleFn:
def _reduce_and_redistribute_output_to_attn_tp_shards(
hidden_states: torch.Tensor, context: CommunicateContext
hidden_states: torch.Tensor,
) -> torch.Tensor:
local_hidden_states = hidden_states.tensor_split(context.attn_tp_size)[
context.attn_tp_rank
parallel = get_parallel()
local_hidden_states = hidden_states.tensor_split(parallel.attn_tp_size)[
parallel.attn_tp_rank
]
attn_tp_reduce_scatter_tensor(local_hidden_states, hidden_states)
return local_hidden_states
@@ -250,8 +249,8 @@ def _mlp_input_without_dp(
residual: torch.Tensor,
forward_batch: ForwardBatch,
layernorm: torch.nn.Module,
context: CommunicateContext,
*,
cache=None,
gathers_residual: bool,
fusions: Tuple[Callable, ...],
group: SumGroup = SumGroup.ATTN_TP,
@@ -274,8 +273,8 @@ def _mlp_input_without_dp(
)
elif group is SumGroup.TP:
hidden_states = tensor_model_parallel_all_reduce(hidden_states)
if _is_npu and context.cache is not None:
_ = prepare_weight_cache(hidden_states, context.cache)
if _is_npu and cache is not None:
_ = prepare_weight_cache(hidden_states, cache)
return read.update_and_read(update, hidden_states, residual, layernorm)
@@ -284,8 +283,8 @@ def _mlp_input_dp_replicate(
residual: torch.Tensor,
forward_batch: ForwardBatch,
layernorm: torch.nn.Module,
context: CommunicateContext,
*,
cache=None,
gathers_residual: bool,
reduces_attention_tp: bool,
places_cp_shards: bool = False,
@@ -324,8 +323,8 @@ def _mlp_input_dp_partial(
residual: torch.Tensor,
forward_batch: ForwardBatch,
layernorm: torch.nn.Module,
context: CommunicateContext,
*,
cache=None,
gathers_residual: bool,
places_cp_shards: bool = False,
read: StageRead = NORM_READ,
@@ -336,7 +335,7 @@ def _mlp_input_dp_partial(
shard of the DP group's tokens beside the others' in the group's slot."""
if gathers_residual:
residual = _redistribute_from_attn_tp_shards(residual)
if context.attn_tp_rank == 0:
if get_parallel().attn_tp_rank == 0:
hidden_states += residual
cp_shard_counts = _cp_shard_token_rows(forward_batch) if places_cp_shards else None
hidden_states = _reduce_and_redistribute_output_to_dp(
@@ -395,7 +394,7 @@ def _tp_all_gather_scattered_rows(
def _dispatch_consumer(
hidden_states, residual, forward_batch, norm, context, *, paths, update=ADD, **call
hidden_states, residual, forward_batch, norm, *, paths, update=ADD, **call
):
if residual is None:
# Enter/read-only paths do not execute an update.
@@ -405,9 +404,7 @@ def _dispatch_consumer(
prepare = paths[update.adds_plainly]
except KeyError:
raise RuntimeError("producer update has no bound input path") from None
return prepare(
hidden_states, residual, forward_batch, norm, context, update=update, **call
)
return prepare(hidden_states, residual, forward_batch, norm, update=update, **call)
def _consumer_step(
@@ -415,7 +412,6 @@ def _consumer_step(
residual: Optional[torch.Tensor],
forward_batch: ForwardBatch,
norm: torch.nn.Module,
context: CommunicateContext,
*,
step: Callable,
adds_plainly: bool,
@@ -473,9 +469,7 @@ def _consumer_step(
hidden_states = reduce_output(owed)
else:
hidden_states = owed.partial
return step(
hidden_states, residual, forward_batch, norm, context, update=update, **call
)
return step(hidden_states, residual, forward_batch, norm, update=update, **call)
def _read_input(
@@ -483,8 +477,8 @@ def _read_input(
residual: Optional[torch.Tensor],
forward_batch: ForwardBatch,
norm: torch.nn.Module,
context: CommunicateContext,
*,
cache=None,
layer_input: Optional[Callable],
enters_stack: bool,
read: StageRead,
@@ -498,7 +492,7 @@ def _read_input(
starts its residual from its input."""
enters = residual is None and enters_stack
if layer_input is not None:
hidden_states, residual = layer_input(hidden_states, residual, context)
hidden_states, residual = layer_input(hidden_states, residual)
if enters:
hidden_states, residual = read.enter(hidden_states), None
if residual is None:
@@ -519,17 +513,15 @@ def _mlp_input_scatter(
residual: torch.Tensor,
forward_batch: ForwardBatch,
layernorm: torch.nn.Module,
context: CommunicateContext,
*,
cache=None,
scatters_residual: bool,
read: StageRead = NORM_READ,
update: StageUpdate = ADD,
):
hidden_states = _reduce_and_redistribute_output_to_attn_tp_shards(
hidden_states, context
)
hidden_states = _reduce_and_redistribute_output_to_attn_tp_shards(hidden_states)
if scatters_residual:
residual = update.residual_to_attn_tp_shard(residual, context)
residual = update.residual_to_attn_tp_shard(residual)
return read.update_and_read(update, hidden_states, residual, layernorm)
@@ -538,15 +530,15 @@ def _mlp_input_slice(
residual: Optional[torch.Tensor],
forward_batch: ForwardBatch,
layernorm: torch.nn.Module,
context: CommunicateContext,
*,
cache=None,
scatters_residual: bool,
read: StageRead = NORM_READ,
update: StageUpdate = ADD,
):
hidden_states = _redistribute_to_attn_tp_shards(hidden_states, context).clone()
hidden_states = _redistribute_to_attn_tp_shards(hidden_states).clone()
if scatters_residual and residual is not None:
residual = update.residual_to_attn_tp_shard(residual, context)
residual = update.residual_to_attn_tp_shard(residual)
return read.update_and_read(update, hidden_states, residual, layernorm)
@@ -555,8 +547,8 @@ def _mlp_input_on_residual_shard(
residual: torch.Tensor,
forward_batch: ForwardBatch,
layernorm: torch.nn.Module,
context: CommunicateContext,
*,
cache=None,
reduces: bool = True,
read: StageRead,
update: StageUpdate,
@@ -565,11 +557,12 @@ def _mlp_input_on_residual_shard(
rows: reduce-scatter the attention output onto the slice, which completes
its sum, write it into the residual and read the FFN input there, then
gather the input back into the attention output's rows."""
parallel = get_parallel()
if hidden_states.shape[0] == 0:
return hidden_states, hidden_states
shard = hidden_states.tensor_split(context.tp_size)[context.tp_rank]
shard = hidden_states.tensor_split(parallel.tp_size)[parallel.tp_rank]
if reduces:
get_parallel().tp_group.reduce_scatter_tensor(shard, hidden_states)
parallel.tp_group.reduce_scatter_tensor(shard, hidden_states)
else:
shard = shard.clone()
shard, residual = read.update_and_read(update, shard, residual, layernorm)
@@ -582,13 +575,13 @@ def _mlp_input_residual_into_sum(
residual: torch.Tensor,
forward_batch: ForwardBatch,
layernorm: torch.nn.Module,
context: CommunicateContext,
*,
cache=None,
read: StageRead = NORM_READ,
update: StageUpdate = ADD,
):
return _tp_all_reduce_with_scattered_residual(
hidden_states, residual, layernorm, context, read
hidden_states, residual, layernorm, read
)
@@ -596,13 +589,13 @@ def _tp_all_reduce_with_scattered_residual(
hidden_states: torch.Tensor,
residual: torch.Tensor,
layernorm: torch.nn.Module,
context: CommunicateContext,
read: StageRead = NORM_READ,
):
parallel = get_parallel()
if hidden_states.shape[0] == 0:
return hidden_states, hidden_states
scattered_states = hidden_states.tensor_split(context.tp_size)[context.tp_rank]
scattered_states = hidden_states.tensor_split(parallel.tp_size)[parallel.tp_rank]
scattered_states += residual
residual = tensor_model_parallel_all_reduce(hidden_states)
return read.read(residual, layernorm)
@@ -613,8 +606,8 @@ def _mlp_input_gather_attention_cp(
residual: torch.Tensor,
forward_batch: ForwardBatch,
layernorm: torch.nn.Module,
context: CommunicateContext,
*,
cache=None,
gather: Callable,
update: StageUpdate = ADD,
):
@@ -622,7 +615,7 @@ def _mlp_input_gather_attention_cp(
equal length, over the attention-CP group. The residual stays on the
shard."""
hidden_states, residual = gather(
hidden_states, residual, forward_batch, layernorm, context, update=update
hidden_states, residual, forward_batch, layernorm, update=update, cache=cache
)
return dsa_cp_gather_hidden_states(hidden_states), residual
@@ -632,8 +625,8 @@ def _mlp_input_gather_moe_cp(
residual: torch.Tensor,
forward_batch: ForwardBatch,
layernorm: torch.nn.Module,
context: CommunicateContext,
*,
cache=None,
gather: Callable,
update: StageUpdate = ADD,
):
@@ -649,7 +642,7 @@ def _mlp_input_gather_moe_cp(
return hidden_states, residual
hidden_states, residual = gather(
hidden_states, residual, forward_batch, layernorm, context, update=update
hidden_states, residual, forward_batch, layernorm, update=update, cache=cache
)
rows = moe_cp_gathered_rows(forward_batch)
@@ -709,27 +702,6 @@ def _redistribute_output_from_moe_cp(
).contiguous()
def _reduce_and_redistribute_output_step(
forward_batch: ForwardBatch,
*,
leaves_for_reduce_scatter: bool,
leaves_for_reduce_scatterv: bool,
) -> Optional[Callable[[torch.Tensor, torch.Tensor, ForwardBatch], None]]:
"""The reduce-scatter that brings an FFN output gathered over attention
DP back to this rank's tokens when the FFN leaves its sum to it (see
StageOutput); None when the FFN reduces the output and only a scatter
remains."""
if should_use_dp_reduce_scatterv() and leaves_for_reduce_scatterv:
return _reduce_and_redistribute_output_varlen
if (
leaves_for_reduce_scatter
and forward_batch.dp_padding_mode.is_max_len()
and can_use_dp_reduce_scatter()
):
return _reduce_and_redistribute_output_max_len
return None
def _to_local_tokens(
step: Callable[[torch.Tensor, torch.Tensor, ForwardBatch], None],
forward_batch: ForwardBatch,
@@ -756,38 +728,15 @@ class CommunicateSummableTensorPairFn:
hidden_states: torch.Tensor,
residual: torch.Tensor,
forward_batch: ForwardBatch,
context: CommunicateContext,
**kwargs,
):
return hidden_states, residual
@staticmethod
def _scatter_hidden_states(
hidden_states: torch.Tensor,
residual: torch.Tensor,
forward_batch: ForwardBatch,
context: CommunicateContext,
allow_reduce_scatter: bool = False,
is_layer_sparse: bool = False,
):
step = (
_reduce_and_redistribute_output_step(
forward_batch,
leaves_for_reduce_scatter=allow_reduce_scatter,
# A MoE block leaves its sum to reduce_scatterv whenever it
# applies (should_skip_post_experts_all_reduce).
leaves_for_reduce_scatterv=allow_reduce_scatter or is_layer_sparse,
)
or _redistribute_output
)
return _to_local_tokens(step, forward_batch, hidden_states), residual
@staticmethod
def _gather(
hidden_states: torch.Tensor,
residual: torch.Tensor,
forward_batch: ForwardBatch,
context: CommunicateContext,
update: StageUpdate = ADD,
**kwargs,
):
@@ -799,7 +748,6 @@ class CommunicateSummableTensorPairFn:
hidden_states: torch.Tensor,
residual: torch.Tensor,
forward_batch: ForwardBatch,
context: CommunicateContext,
*,
sums: bool,
gathers_back: bool,
@@ -810,17 +758,20 @@ class CommunicateSummableTensorPairFn:
a reduce-scatter that also completes its sum when ``sums``, else this
rank's slice of the complete output. With ``gathers_back`` it is
written into the residual there and the full rows are gathered back."""
parallel = get_parallel()
if sums:
hidden_states, _ = tp_reduce_scatter(hidden_states, None, context)
hidden_states, _ = tp_reduce_scatter(hidden_states, None)
else:
hidden_states = hidden_states.tensor_split(context.tp_size)[context.tp_rank]
hidden_states = hidden_states.tensor_split(parallel.tp_size)[
parallel.tp_rank
]
if not gathers_back:
return hidden_states, residual
local_states = update.update(hidden_states, residual)
hidden_states = local_states.new_empty(
local_states.shape[0] * context.tp_size, *local_states.shape[1:]
local_states.shape[0] * parallel.tp_size, *local_states.shape[1:]
)
get_parallel().tp_group.all_gather_into_tensor(hidden_states, local_states)
parallel.tp_group.all_gather_into_tensor(hidden_states, local_states)
return hidden_states, None
@staticmethod
@@ -828,17 +779,15 @@ class CommunicateSummableTensorPairFn:
hidden_states: torch.Tensor,
residual: torch.Tensor,
forward_batch: ForwardBatch,
context: CommunicateContext,
):
assert residual is None, "not yet handled residual!=None"
return _redistribute_to_attn_tp_shards(hidden_states, context), None
return _redistribute_to_attn_tp_shards(hidden_states), None
@staticmethod
def _take_back_attention_cp_shard(
hidden_states: torch.Tensor,
residual: torch.Tensor,
forward_batch: ForwardBatch,
context: CommunicateContext,
**kwargs,
):
"""DSA and MLA CP: this rank's shard of a complete output gathered in
@@ -852,7 +801,6 @@ class CommunicateSummableTensorPairFn:
hidden_states: torch.Tensor,
residual: torch.Tensor,
forward_batch: ForwardBatch,
context: CommunicateContext,
**kwargs,
):
"""DSA and MLA CP: sum the FFN output over the attention-CP group and
@@ -864,7 +812,6 @@ class CommunicateSummableTensorPairFn:
hidden_states: torch.Tensor,
residual: torch.Tensor,
forward_batch: ForwardBatch,
context: CommunicateContext,
**kwargs,
):
"""This rank's CP shard of the rows the DP gather put in its DP group's
@@ -886,7 +833,6 @@ class CommunicateSummableTensorPairFn:
hidden_states: torch.Tensor,
residual: torch.Tensor,
forward_batch: ForwardBatch,
context: CommunicateContext,
**kwargs,
):
"""Scatter MoE output back to TP_ATTN_FULL after MOE_FULL computation.
@@ -905,7 +851,7 @@ class CommunicateSummableTensorPairFn:
if rows is not None:
hidden_states = _redistribute_output_from_moe_cp(hidden_states, rows)
if context.attn_dp_size > 1:
if get_parallel().attn_dp_size > 1:
hidden_states = _to_local_tokens(
_redistribute_output, forward_batch, hidden_states
)
@@ -21,6 +21,18 @@ import torch
from sglang.srt.distributed import GroupCoordinator
class OutputTransform(msgspec.Struct, frozen=True):
"""An operation on the producer contribution, before residual addition.
Some existing implementations apply it to a partial before reduce-scatter
but to a complete value after all-reduce. These are explicit alternatives,
not an assumption that an arbitrary operation commutes with reduction.
"""
apply: Callable[[torch.Tensor], torch.Tensor]
before_reduce_scatter: bool = False
class UnreducedOutput(msgspec.Struct, frozen=True):
"""A layer output that still owes its sum, left for the next layer's input.
Hand it to the next layer, or pass it through reduce_output() before reading
@@ -38,7 +38,7 @@ class StageUpdate(Protocol):
def update(self, hidden_states, residual) -> torch.Tensor:
"""The residual with the output written into it."""
def residual_to_attn_tp_shard(self, residual, context) -> torch.Tensor:
def residual_to_attn_tp_shard(self, residual) -> torch.Tensor:
"""This attention-TP rank's slice of the residual, with whatever the
update reads along with it."""
@@ -105,7 +105,7 @@ def finish_layer_stack(
return reduce_output(hidden_states), residual
def norm_output(hidden_states, residual, norm, capture_output=None):
def norm_output(hidden_states, residual, norm, capture_output=None, **read_kwargs):
"""The final norm and an optional capture of the same updated residual.
Keep add and norm together: normalizing a separately rounded residual is
@@ -116,7 +116,7 @@ def norm_output(hidden_states, residual, norm, capture_output=None):
if capture_output is not None:
capture_output(hidden_states)
return norm(hidden_states)
hidden_states, residual = norm(hidden_states, residual)
hidden_states, residual = norm(hidden_states, residual, **read_kwargs)
if capture_output is not None:
capture_output(residual)
return hidden_states
@@ -273,8 +273,8 @@ class Add:
hidden_states += residual
return hidden_states
def residual_to_attn_tp_shard(self, residual, context):
return _redistribute_to_attn_tp_shards(residual, context)
def residual_to_attn_tp_shard(self, residual):
return _redistribute_to_attn_tp_shards(residual)
def residual_from_attn_tp_shards(self, residual):
return _redistribute_from_attn_tp_shards(residual)
@@ -1,18 +1,82 @@
"""Own one invocation's residual stream, including independent TBO children."""
# Copyright 2023-2024 SGLang Team
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
"""Residual access for one layer-stack invocation or TBO microbatch."""
from sglang.srt.layers.communicator.residual import access
from sglang.srt.layers.communicator.residual.stream import ResidualStream
from sglang.srt.model_executor.forward_batch_info import PPProxyTensors
def start(forward_batch):
"""Begin a fresh embedding-side invocation and return its stream."""
stream = ResidualStream()
forward_batch.residual_stream = stream
return stream
def current(forward_batch):
"""Return the invocation's authoritative stream while its stack is active."""
stream = forward_batch.residual_stream
if stream is None:
raise RuntimeError("start the layer stack before entering a stage")
return stream
def finish(hidden_states, forward_batch):
return current(forward_batch).complete(hidden_states)
def take_output(hidden_states, forward_batch):
"""Release a terminal output whose residual update has already been written."""
stream = current(forward_batch)
stream.check(hidden_states)
if stream.pending is not None:
raise RuntimeError("write the residual update before taking the final output")
forward_batch.residual_stream = None
return hidden_states
def norm(hidden_states, forward_batch, layernorm, capture_output=None, **read_kwargs):
hidden_states, residual = current(forward_batch).finish(hidden_states)
# The terminal consumer now owns the pair. Do not keep layer buffers alive
# through logits processing or the next forward on this batch.
forward_batch.residual_stream = None
return access.norm_output(
hidden_states, residual, layernorm, capture_output, **read_kwargs
)
def to_pp(hidden_states, forward_batch, *, preserve_declared=True):
hidden_states, residual = current(forward_batch).finish(
hidden_states, preserve_declared=preserve_declared
)
forward_batch.residual_stream = None
return PPProxyTensors({"hidden_states": hidden_states, "residual": residual})
def snapshot(hidden_states, forward_batch):
return current(forward_batch).snapshot(hidden_states)
def add_to_output(hidden_states, forward_batch, extra):
output, _ = access.add_to_output(hidden_states, current(forward_batch), extra)
return output
def fold(hidden_states, forward_batch):
output, _ = access.fold(hidden_states, current(forward_batch))
return output
def written(hidden_states, forward_batch):
forward_batch.residual_stream = ResidualStream(hidden_states)
return hidden_states
@@ -20,6 +20,7 @@ import torch
from sglang.kernels.ops.layernorm.mhc import hc_contract, hc_expand
from sglang.srt.layers.communicator.residual import LayerResidual
from sglang.srt.runtime_context import get_parallel
@dataclass
@@ -96,8 +97,9 @@ class MHCState:
self.h_res = None
self.h_post = None
def residual_to_attn_tp_shard(self, residual, context):
rank, size = context.attn_tp_rank, context.attn_tp_size
def residual_to_attn_tp_shard(self, residual):
parallel = get_parallel()
rank, size = parallel.attn_tp_rank, parallel.attn_tp_size
self.h_res = self.h_res.tensor_split(size)[rank]
self.h_post = self.h_post.tensor_split(size)[rank]
return residual.tensor_split(size)[rank]
@@ -151,8 +153,8 @@ class _AttentionUpdate:
def update(self, hidden_states, residual):
return self.state.mlp_combine(hidden_states, residual)
def residual_to_attn_tp_shard(self, residual, context):
return self.state.residual_to_attn_tp_shard(residual, context)
def residual_to_attn_tp_shard(self, residual):
return self.state.residual_to_attn_tp_shard(residual)
def residual_from_attn_tp_shards(self, residual):
return self.state.residual_from_attn_tp_shards(residual)
@@ -198,8 +200,8 @@ class _FfnUpdate:
hidden_states = hc_contract(hidden_states, self.state.hc_mult)
return hidden_states
def residual_to_attn_tp_shard(self, residual, context):
return self.state.residual_to_attn_tp_shard(residual, context)
def residual_to_attn_tp_shard(self, residual):
return self.state.residual_to_attn_tp_shard(residual)
def residual_from_attn_tp_shards(self, residual):
return self.state.residual_from_attn_tp_shards(residual)
@@ -0,0 +1,329 @@
# Copyright 2023-2024 SGLang Team
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
"""A stage's model-facing input and output boundary."""
from typing import TYPE_CHECKING, Callable, Optional
import torch
from sglang.srt.layers import layernorm_sp
from sglang.srt.layers.aux_hidden_states import AuxHiddenStateAccumulator
from sglang.srt.layers.communicator.adapters import branch
from sglang.srt.layers.communicator.adapters.lora import (
publish_attention,
publish_ffn,
)
from sglang.srt.layers.communicator.boundary import (
BoundarySteps,
StageEntry,
StageKind,
)
from sglang.srt.layers.communicator.ops import CommunicateSimpleFn
from sglang.srt.layers.communicator.residual.access import buffer, from_pp
from sglang.srt.layers.communicator.residual.add_norm import (
ADD,
)
from sglang.srt.layers.communicator.residual.batch import current
from sglang.srt.layers.communicator.residual.stream import DeclaredSum, ResidualStream
from sglang.srt.model_executor.forward_batch_info import ForwardBatch
from sglang.srt.runtime_context import get_forward
if TYPE_CHECKING:
from sglang.srt.layers.communicator.construction import StagePlan
from sglang.srt.layers.communicator.factories import StageDeclaration
class StageCommunicator:
"""One stage of a layer, its attention or its FFN: the boundary into it,
run with the stage's norm on the steps the layer chose for the batch."""
def __init__(
self,
plan: "StagePlan",
kind: StageKind,
norm,
*,
declaration: Optional["StageDeclaration"] = None,
):
self.plan = plan
self.kind = kind
self.norm = norm
self._declaration = declaration
@property
def declaration(self) -> "StageDeclaration":
"""Immutable source contract for constructing another local sequence."""
if self._declaration is None:
raise RuntimeError("this boundary was constructed from raw layout edges")
return self._declaration
def entry(
self, forward_batch: ForwardBatch, steps: Optional[BoundarySteps] = None
) -> StageEntry:
if steps is None:
steps = self.plan._batch_steps(forward_batch)
entry = steps.attention if self.kind is StageKind.ATTENTION else steps.ffn
if entry is None:
raise NotImplementedError(
f"a layer that is one stage has no {self.kind.name.lower()} stage"
)
return entry
def _prepare(
self,
hidden_states,
residual,
forward_batch: ForwardBatch,
steps: Optional[BoundarySteps] = None,
**call,
):
"""The stage's input and the residual, from the previous stage's output
(``call``: what the stage's read takes, e.g. the attention's
``quant_format``)."""
entry = self.entry(forward_batch, steps)
stream = (
residual
if isinstance(residual, ResidualStream)
else ResidualStream(residual)
)
if not isinstance(residual, ResidualStream):
if residual is not None or entry.input_sum is not None:
hidden_states = stream.leave(
hidden_states, call.get("update", ADD), declared_sum=entry.input_sum
)
elif (
stream.pending is None
and stream.residual is None
and entry.input_sum is not None
):
hidden_states = stream.leave(
hidden_states, ADD, declared_sum=entry.input_sum
)
if stream.pending is not None:
call["update"] = stream.pending.update
call["pending"] = stream.pending
if stream.pending is None and stream.residual is not None:
call["written"] = True
hidden_states, residual = stream.input(hidden_states)
hidden_states, residual = entry.prepare(
hidden_states, residual, forward_batch, self.norm, **call
)
if entry.input_move is not None:
hidden_states = entry.input_move(
hidden_states=hidden_states,
forward_batch=forward_batch,
)
if entry.handoff is not None:
hidden_states = entry.handoff(
hidden_states, forward_batch, self.plan.qkv_latent_func
)
stream.write(residual)
return hidden_states, stream
@property
def fusions(self):
return self.plan.fusions
@property
def input_rows(self):
return self.plan.input_rows
@property
def input_on_attention_tp_slices(self):
return self.plan.input_on_attention_tp_slices
def prepare(self, hidden_states, forward_batch, *, cache=None, **call):
if cache is not None:
call["cache"] = cache
prepare = (
self._prepare_attention
if self.kind is StageKind.ATTENTION
else self._prepare_input
)
hidden_states, forward_batch.residual_stream = prepare(
hidden_states, current(forward_batch), forward_batch, **call
)
return hidden_states
def finish(self, hidden_states, forward_batch):
"""Hand this attention's actual contribution to the following stage."""
if self.kind is not StageKind.ATTENTION or not self.plan.fixed_output:
raise RuntimeError(
"use the stage exit scope for an FFN or single-stage mixer"
)
produced = self.plan.produced(self.kind, forward_batch)
return current(forward_batch).leave(
hidden_states,
produced.update,
declared_sum=produced.group if produced.always_leaves else None,
)
def exit(self, forward_batch):
result = (
self.plan.output.ffn_exit(forward_batch)
if self.kind is StageKind.FFN
else self.plan.output.mixer_exit(forward_batch)
)
result._stream = current(forward_batch)
return result
def from_pp(self, tensors, forward_batch, *, allow_missing_residual: bool = False):
hidden_states, residual = from_pp(
tensors,
residual_in_hidden=self.plan.residual_in_hidden,
allow_missing_residual=allow_missing_residual,
)
declared_sum = None
if forward_batch is not None:
steps = self.plan._batch_steps(forward_batch)
entry = steps.attention or steps.ffn
declared_sum = entry.input_sum
hidden_states, forward_batch.residual_stream = ResidualStream.arrive(
hidden_states, residual, ADD, declared_sum=declared_sum
)
return hidden_states
def snapshot(self, hidden_states, forward_batch):
return current(forward_batch).snapshot(hidden_states)
def capture_output(self, hidden_states, forward_batch, *, skip_empty=False):
stream = current(forward_batch)
if skip_empty:
storage = buffer(hidden_states)
if storage is not None and storage.shape[0] == 0:
return hidden_states, None
if stream.pending is None or not isinstance(stream.pending.owed, DeclaredSum):
hidden_states = stream.complete(hidden_states)
return hidden_states, stream.snapshot(hidden_states)
def postprocess(self, hidden_states, forward_batch):
hidden_states, stream = self.plan.output.postprocess_layer(
hidden_states, current(forward_batch), forward_batch
)
forward_batch.residual_stream = stream
return hidden_states
def branch_input(self, source, hidden_states, forward_batch):
hidden_states, stream = branch.branch_input(
self.plan,
source.plan,
hidden_states,
current(forward_batch),
forward_batch,
)
forward_batch.residual_stream = stream
return hidden_states
def branch_output(self, hidden_states, forward_batch):
return branch.branch_output(self.plan, hidden_states, forward_batch)
def merge_branch(self, contribution, hidden_states, source, forward_batch):
hidden_states, stream = branch.merge_branch(
self.plan,
contribution,
hidden_states,
current(forward_batch),
source.plan,
forward_batch,
)
forward_batch.residual_stream = stream
return hidden_states
def _prepare_input(self, hidden_states, residual, forward_batch, **call):
plan = self.plan
if self.kind is StageKind.ATTENTION:
publish_attention(plan._publish_lora_layout)
if plan._sp_steps is not None and plan.is_first_layer:
get_forward().set(
"sp_active", layernorm_sp.runs_sp(forward_batch.forward_mode)
)
if get_forward().sp_active:
hidden_states = layernorm_sp.sp_entry_scatter(hidden_states)
else:
publish_ffn(plan._publish_lora_layout, self.entry(forward_batch).input_rows)
return self._prepare(hidden_states, residual, forward_batch, **call)
def _prepare_attention(
self,
hidden_states: torch.Tensor,
stream: ResidualStream,
forward_batch: ForwardBatch,
captured_last_layer_outputs: Optional[AuxHiddenStateAccumulator] = None,
post_residual_addition: Optional[torch.Tensor] = None,
quant_format: str = "",
capture_output: Optional[Callable] = None,
):
# Aux consumers need a materialized output before the input norm.
# Complete communication here, preserving the existing add+norm kernel
# and its FP32 accumulation. Its residual result also supplies capture.
capture_before_read = capture_output is not None and (
(stream.residual is None and stream.pending is None)
or post_residual_addition is not None
)
if capture_output is not None:
hidden_states = stream.complete(hidden_states)
if capture_before_read:
# Embeddings precede enter; HF deepstack capture precedes the
# extra addition. Neither is the residual returned by the read.
value, previous = stream.finish(hidden_states)
if previous is None:
capture_output(value)
else:
capture_output(value + previous, owned=True)
hidden_states, stream = self._prepare_input(
hidden_states,
stream,
forward_batch,
quant_format=quant_format,
post_residual_addition=post_residual_addition,
)
entry = self.entry(forward_batch)
if capture_output is not None and not capture_before_read:
value = stream.residual
move = entry.capture_move
if move is not None:
value = move(value, forward_batch=forward_batch)
keeps_input = entry.capture_preserves_residual
capture_output(
value,
owned=entry.capture_move_allocates
or (
move is None
and keeps_input is not None
and keeps_input(value, forward_batch)
),
)
if captured_last_layer_outputs is not None:
residual_value = stream.residual
move = entry.input_move
gathered_last_layer_output = (
residual_value
if move is None
else move(
hidden_states=residual_value,
forward_batch=forward_batch,
)
)
# Input gathers allocate fresh DP buffers.
# Without a gather this is the mutable residual, so retention copies.
captured_last_layer_outputs.capture(
gathered_last_layer_output,
owned=move is CommunicateSimpleFn._scattered_to_tp_attn_full
or (
move is None
and entry.capture_preserves_residual is not None
and entry.capture_preserves_residual(residual_value, forward_batch)
),
)
return hidden_states, stream
+8 -5
View File
@@ -559,7 +559,12 @@ def draft_model_build_scope():
original_scope = moe.in_speculative_scope
try:
moe.in_speculative_scope = True
yield
# Boundaries capture this resolved preference while the draft builds.
# Restoring it cannot change an already constructed target plan.
with get_exec().comm.override(
boundary_reduction=get_spec().speculative_boundary_reduction
):
yield
finally:
moe.in_speculative_scope = original_scope
moe.disable_shared_experts_fusion = original_fusion
@@ -759,10 +764,8 @@ def should_skip_post_experts_all_reduce(*, is_tp_path: bool) -> bool:
Pass ``is_tp_path=True`` for the TP all-reduce, ``False`` for the EP one.
"""
return (
should_skip_mlp_all_reduce()
or should_use_dp_reduce_scatterv()
or post_experts_output_is_complete(is_tp_path=is_tp_path)
return should_skip_mlp_all_reduce() or post_experts_output_is_complete(
is_tp_path=is_tp_path
)
@@ -565,7 +565,7 @@ class ForwardBatch(ForwardBatchDeepSeekMHAMixin):
dp_prefill_cuda_graph_max_prefix_len: int = 0
global_forward_mode: Optional[ForwardMode] = None
# Owned only while this invocation executes its local decoder stack.
# Current layer-stack invocation; each TBO child owns a separate stream.
residual_stream: Optional[ResidualStream] = None
# For two-batch overlap
+41 -58
View File
@@ -36,9 +36,10 @@ from sglang.srt.layers.aux_hidden_states import (
AuxHiddenStateList,
)
from sglang.srt.layers.communicator import (
LayerCommunicator,
LayerFacts,
declare_attn,
declare_ffn,
enable_moe_dense_fully_dp,
make_stages,
)
from sglang.srt.layers.communicator.residual import batch as residual_batch
from sglang.srt.layers.dp_attention import (
@@ -673,15 +674,6 @@ class BailingMoEBlock(nn.Module):
config, layer_id=layer_id + 1, is_nextn=False
)
self.layer_facts = LayerFacts.init_new(
layer_id=layer_id,
# A NextN draft is a one-layer model.
num_layers=1 if is_nextn else config.num_hidden_layers,
is_layer_sparse=self.is_layer_sparse,
is_previous_layer_sparse=is_previous_layer_sparse,
is_next_layer_sparse=is_next_layer_sparse,
)
if self.is_layer_sparse:
self.mlp = BailingMoESparseMoeBlock(
layer_id=layer_id,
@@ -706,11 +698,21 @@ class BailingMoEBlock(nn.Module):
self.post_attention_layernorm = RMSNorm(hidden_size, eps=config.rms_norm_eps)
self.layer_communicator = LayerCommunicator(
layer_facts=self.layer_facts,
input_layernorm=self.input_layernorm,
post_attention_layernorm=self.post_attention_layernorm,
allow_reduce_scatter=True,
self.attn_stage, self.ffn_stage = make_stages(
(declare_attn(), self.input_layernorm),
(
declare_ffn(
sparse=self.is_layer_sparse,
next_sparse=is_next_layer_sparse,
),
self.post_attention_layernorm,
),
previous=declare_ffn(
sparse=is_previous_layer_sparse, next_sparse=self.is_layer_sparse
)
if layer_id != 0
else None,
terminal=layer_id == (1 if is_nextn else config.num_hidden_layers) - 1,
)
def _is_layer_sparse(
@@ -725,18 +727,14 @@ class BailingMoEBlock(nn.Module):
positions: torch.Tensor,
hidden_states: torch.Tensor,
forward_batch: ForwardBatch,
residual: Optional[torch.Tensor],
captured_last_layer_outputs: Optional[AuxHiddenStateAccumulator] = None,
capture_output=None,
) -> torch.Tensor:
hidden_states, residual = (
self.layer_communicator.prepare_attn_and_capture_last_layer_outputs(
hidden_states,
residual,
forward_batch,
captured_last_layer_outputs=captured_last_layer_outputs,
capture_output=capture_output,
)
hidden_states = self.attn_stage.prepare(
hidden_states,
forward_batch,
captured_last_layer_outputs=captured_last_layer_outputs,
capture_output=capture_output,
)
if hidden_states.shape[0] != 0:
@@ -746,17 +744,17 @@ class BailingMoEBlock(nn.Module):
forward_batch=forward_batch,
)
hidden_states, residual = self.layer_communicator.prepare_mlp(
hidden_states = self.attn_stage.finish(hidden_states, forward_batch)
hidden_states = self.ffn_stage.prepare(
hidden_states=hidden_states,
residual=residual,
forward_batch=forward_batch,
)
with self.layer_communicator.ffn_exit(forward_batch) as ffn_exit:
with self.ffn_stage.exit(forward_batch) as ffn_exit:
hidden_states = self.mlp(hidden_states, forward_batch)
hidden_states, residual = ffn_exit.finish(hidden_states, residual)
hidden_states = ffn_exit.finish(hidden_states)
return hidden_states, residual
return hidden_states
class BailingMoEModel(nn.Module):
@@ -821,51 +819,36 @@ class BailingMoEModel(nn.Module):
hidden_states = self.word_embeddings(input_ids)
else:
hidden_states = input_embeds
residual = residual_batch.start(forward_batch)
residual_batch.start(forward_batch)
else:
assert pp_proxy_tensors is not None
hidden_states, residual = self.layers[
self.start_layer
].layer_communicator.from_pp(pp_proxy_tensors, forward_batch)
hidden_states = self.layers[self.start_layer].attn_stage.from_pp(
pp_proxy_tensors, forward_batch
)
aux_hidden_states = AuxHiddenStateList()
for i in range(self.start_layer, self.end_layer):
with get_global_expert_distribution_recorder().with_current_layer(i):
layer = self.layers[i]
hidden_states, residual = layer(
hidden_states = layer(
positions,
hidden_states,
forward_batch,
residual,
captured_last_layer_outputs=(
aux_hidden_states
if getattr(layer, "_is_layer_to_capture", False)
else None
),
captured_last_layer_outputs=aux_hidden_states
if getattr(layer, "_is_layer_to_capture", False)
else None,
capture_output=aux_hidden_states.capture
if i in self.layers_to_capture
else None,
)
last_layer = self.layers[self.end_layer - 1]
hidden_states, residual = last_layer.layer_communicator.finish_layer_stack(
hidden_states,
residual,
forward_batch,
preserve_declared=not self.pp_group.is_last_rank,
)
if not self.pp_group.is_last_rank:
return PPProxyTensors(
{
"hidden_states": hidden_states,
"residual": residual,
}
)
return residual_batch.to_pp(hidden_states, forward_batch)
else:
hidden_states = residual_batch.finish(hidden_states, forward_batch)
if not forward_batch.forward_mode.is_idle():
if residual is None:
hidden_states = self.norm(hidden_states)
else:
hidden_states, _ = self.norm(hidden_states, residual)
hidden_states = residual_batch.norm(
hidden_states, forward_batch, self.norm
)
if len(aux_hidden_states) == 0:
return hidden_states
+37 -48
View File
@@ -16,8 +16,9 @@ from sglang.srt.eplb.expert_distribution import get_global_expert_distribution_r
from sglang.srt.layers import deep_gemm_wrapper
from sglang.srt.layers.activation import SiluAndMul
from sglang.srt.layers.communicator import (
LayerCommunicator,
LayerFacts,
declare_attn,
declare_ffn,
make_stages,
)
from sglang.srt.layers.communicator.residual import batch as residual_batch
from sglang.srt.layers.dp_attention import (
@@ -814,26 +815,28 @@ class BailingMoELinearDecoderLayer(nn.Module):
self.input_layernorm = RMSNorm(self.hidden_size, eps=rms_norm_eps)
self.post_attention_layernorm = RMSNorm(self.hidden_size, eps=rms_norm_eps)
self.layer_facts = LayerFacts.init_new(
layer_id=layer_id,
# A NextN draft is a one-layer model.
num_layers=1 if is_nextn else config.num_hidden_layers,
is_layer_sparse=is_moe_layer,
is_previous_layer_sparse=is_previous_moe_layer,
is_next_layer_sparse=is_next_layer_moe_layer,
)
qkv_latent_func = (
self.attention.prepare_qkv_latent
if config.attention_type == 1 and self.use_mla
else None
)
self.layer_communicator = LayerCommunicator(
layer_facts=self.layer_facts,
input_layernorm=self.input_layernorm,
post_attention_layernorm=self.post_attention_layernorm,
allow_reduce_scatter=False,
qkv_latent_func=qkv_latent_func,
self.attn_stage, self.ffn_stage = make_stages(
(
declare_attn(),
self.input_layernorm,
{"qkv_latent_func": qkv_latent_func},
),
(
declare_ffn(
sparse=is_moe_layer,
next_sparse=is_next_layer_moe_layer,
),
self.post_attention_layernorm,
),
previous=declare_ffn(sparse=is_previous_moe_layer, next_sparse=is_moe_layer)
if layer_id != 0
else None,
terminal=layer_id == (1 if is_nextn else config.num_hidden_layers) - 1,
)
def _is_layer_sparse(
@@ -848,13 +851,10 @@ class BailingMoELinearDecoderLayer(nn.Module):
hidden_states: torch.Tensor,
positions: torch.Tensor,
forward_batch: ForwardBatch,
residual: Optional[torch.Tensor],
zero_allocator: BumpAllocator,
**kwargs,
) -> tuple[torch.Tensor, torch.Tensor]:
hidden_states, residual = self.layer_communicator.prepare_attn(
hidden_states, residual, forward_batch
)
) -> torch.Tensor:
hidden_states = self.attn_stage.prepare(hidden_states, forward_batch)
# logger.warning(
# f"===={self.layer_id=}, 1 shape= {hidden_states.shape}, {residual.shape}"
# )
@@ -875,16 +875,15 @@ class BailingMoELinearDecoderLayer(nn.Module):
# logger.warning(
# f"===={self.layer_id=}, 2 shape= {hidden_states.shape}, {residual.shape}"
# )
hidden_states, residual = self.layer_communicator.prepare_mlp(
hidden_states, residual, forward_batch
)
hidden_states = self.attn_stage.finish(hidden_states, forward_batch)
hidden_states = self.ffn_stage.prepare(hidden_states, forward_batch)
# logger.warning(
# f"===={self.layer_id=}, 3 shape= {hidden_states.shape}, {residual.shape}"
# )
with self.layer_communicator.ffn_exit(forward_batch) as ffn_exit:
with self.ffn_stage.exit(forward_batch) as ffn_exit:
hidden_states = self.mlp(hidden_states)
hidden_states, residual = ffn_exit.finish(hidden_states, residual)
return hidden_states, residual
hidden_states = ffn_exit.finish(hidden_states)
return hidden_states
@staticmethod
def shared_moe_coefficient_loader(
@@ -981,12 +980,12 @@ class BailingMoELinearModel(nn.Module):
hidden_states = self.word_embeddings(input_ids)
else:
hidden_states = inputs_embeds
residual = residual_batch.start(forward_batch)
residual_batch.start(forward_batch)
else:
assert pp_proxy_tensors is not None
hidden_states, residual = self.layers[
self.start_layer
].layer_communicator.from_pp(pp_proxy_tensors, forward_batch)
hidden_states = self.layers[self.start_layer].attn_stage.from_pp(
pp_proxy_tensors, forward_batch
)
total_num_layers = self.end_layer - self.start_layer
device = inputs_embeds.device if inputs_embeds is not None else input_ids.device
@@ -999,30 +998,20 @@ class BailingMoELinearModel(nn.Module):
for i in range(self.start_layer, self.end_layer):
with get_global_expert_distribution_recorder().with_current_layer(i):
layer = self.layers[i]
hidden_states, residual = layer(
hidden_states = layer(
hidden_states=hidden_states,
positions=positions,
forward_batch=forward_batch,
residual=residual,
zero_allocator=zero_allocator,
)
last_layer = self.layers[self.end_layer - 1]
hidden_states, residual = last_layer.layer_communicator.finish_layer_stack(
hidden_states,
residual,
forward_batch,
preserve_declared=not self.pp_group.is_last_rank,
)
if not self.pp_group.is_last_rank:
return PPProxyTensors(
{"hidden_states": hidden_states, "residual": residual}
)
return residual_batch.to_pp(hidden_states, forward_batch)
else:
hidden_states = residual_batch.finish(hidden_states, forward_batch)
if not forward_batch.forward_mode.is_idle():
if residual is None:
hidden_states = self.norm(hidden_states)
else:
hidden_states, _ = self.norm(hidden_states, residual)
hidden_states = residual_batch.norm(
hidden_states, forward_batch, self.norm
)
return hidden_states
+7 -13
View File
@@ -162,7 +162,7 @@ class BailingMoEModelNextN(nn.Module):
)
)
residual = residual_batch.start(forward_batch)
residual_batch.start(forward_batch)
if self.is_hybrid:
device = input_ids.device
zero_allocator = BumpAllocator(
@@ -172,26 +172,20 @@ class BailingMoEModelNextN(nn.Module):
dtype=torch.float32,
device=device,
)
hidden_states, residual = self.decoder(
hidden_states = self.decoder(
hidden_states=hidden_states,
positions=positions,
forward_batch=forward_batch,
residual=residual,
zero_allocator=zero_allocator,
)
else:
hidden_states, residual = self.decoder(
positions, hidden_states, forward_batch, residual
)
hidden_states = self.decoder(positions, hidden_states, forward_batch)
hidden_states, residual = self.decoder.layer_communicator.finish_layer_stack(
hidden_states, residual, forward_batch
)
hidden_states = residual_batch.finish(hidden_states, forward_batch)
if not forward_batch.forward_mode.is_idle():
if residual is not None:
hidden_states, _ = self.final_layernorm(hidden_states, residual)
else:
hidden_states = self.final_layernorm(hidden_states)
hidden_states = residual_batch.norm(
hidden_states, forward_batch, self.final_layernorm
)
return hidden_states
+44 -61
View File
@@ -30,9 +30,10 @@ from sglang.srt.layers import deep_gemm_wrapper
from sglang.srt.layers.activation import SiluAndMul
from sglang.srt.layers.aux_hidden_states import AuxHiddenStateList
from sglang.srt.layers.communicator import (
LayerCommunicator,
LayerFacts,
declare_attn,
declare_ffn,
enable_moe_dense_fully_dp,
make_stages,
)
from sglang.srt.layers.communicator.residual import batch as residual_batch
from sglang.srt.layers.dp_attention import is_dp_attention_enabled
@@ -1148,27 +1149,27 @@ class BailingMoELinearDecoderLayer(nn.Module):
self.input_layernorm = RMSNorm(self.hidden_size, eps=rms_norm_eps)
self.post_attention_layernorm = RMSNorm(self.hidden_size, eps=rms_norm_eps)
self.layer_facts = LayerFacts.init_new(
layer_id=layer_id,
# NextN wraps a single decoder layer whose checkpoint prefix is the
# post-model layer id. Treat it as a one-layer model for scatter-mode
# planning so A2A/DeepEP outputs are gathered before logits.
num_layers=1 if is_nextn else config.num_hidden_layers,
is_layer_sparse=is_moe_layer,
is_previous_layer_sparse=is_previous_moe_layer,
is_next_layer_sparse=is_next_layer_sparse,
)
self.layer_communicator = LayerCommunicator(
layer_facts=self.layer_facts,
input_layernorm=self.input_layernorm,
post_attention_layernorm=self.post_attention_layernorm,
allow_reduce_scatter=True,
qkv_latent_func=(
self.attention.prepare_qkv_latent
if self.attention_type == 1 and self.use_mla
else None
self.attn_stage, self.ffn_stage = make_stages(
(
declare_attn(),
self.input_layernorm,
{
"qkv_latent_func": self.attention.prepare_qkv_latent
if self.attention_type == 1 and self.use_mla
else None
},
),
(
declare_ffn(
sparse=is_moe_layer,
next_sparse=is_next_layer_sparse,
),
self.post_attention_layernorm,
),
previous=declare_ffn(sparse=is_previous_moe_layer, next_sparse=is_moe_layer)
if layer_id != 0
else None,
terminal=layer_id == (1 if is_nextn else config.num_hidden_layers) - 1,
)
@torch.inference_mode()
@@ -1177,18 +1178,12 @@ class BailingMoELinearDecoderLayer(nn.Module):
hidden_states: torch.Tensor,
positions: torch.Tensor,
forward_batch: ForwardBatch,
residual: Optional[torch.Tensor],
zero_allocator: BumpAllocator,
capture_output=None,
**kwargs,
) -> tuple[torch.Tensor, torch.Tensor]:
hidden_states, residual = (
self.layer_communicator.prepare_attn_and_capture_last_layer_outputs(
hidden_states,
residual,
forward_batch,
capture_output=capture_output,
)
) -> torch.Tensor:
hidden_states = self.attn_stage.prepare(
hidden_states, forward_batch, capture_output=capture_output
)
if not forward_batch.forward_mode.is_idle():
@@ -1213,11 +1208,10 @@ class BailingMoELinearDecoderLayer(nn.Module):
forward_batch=forward_batch,
)
hidden_states, residual = self.layer_communicator.prepare_mlp(
hidden_states, residual, forward_batch
)
hidden_states = self.attn_stage.finish(hidden_states, forward_batch)
hidden_states = self.ffn_stage.prepare(hidden_states, forward_batch)
with self.layer_communicator.ffn_exit(forward_batch) as ffn_exit:
with self.ffn_stage.exit(forward_batch) as ffn_exit:
if not (
enable_moe_dense_fully_dp()
and (not self.is_layer_sparse)
@@ -1227,9 +1221,9 @@ class BailingMoELinearDecoderLayer(nn.Module):
hidden_states,
forward_batch=forward_batch,
)
hidden_states, residual = ffn_exit.finish(hidden_states, residual)
hidden_states = ffn_exit.finish(hidden_states)
return hidden_states, residual
return hidden_states
@staticmethod
def shared_moe_coefficient_loader(
@@ -1347,12 +1341,12 @@ class BailingMoELinearModel(nn.Module):
hidden_states = self.word_embeddings(input_ids)
else:
hidden_states = input_embeds
residual = residual_batch.start(forward_batch)
residual_batch.start(forward_batch)
else:
assert pp_proxy_tensors is not None
hidden_states, residual = self.layers[
self.start_layer
].layer_communicator.from_pp(pp_proxy_tensors, forward_batch)
hidden_states = self.layers[self.start_layer].attn_stage.from_pp(
pp_proxy_tensors, forward_batch
)
total_num_layers = self.end_layer - self.start_layer
device = hidden_states.device
@@ -1386,41 +1380,30 @@ class BailingMoELinearModel(nn.Module):
for i in range(self.start_layer, self.end_layer):
with get_global_expert_distribution_recorder().with_current_layer(i):
layer = self.layers[i]
hidden_states, residual = layer(
hidden_states = layer(
hidden_states=hidden_states,
positions=positions,
forward_batch=forward_batch,
residual=residual,
zero_allocator=zero_allocator,
capture_output=capture_output
if capture_aux
and i > self.start_layer
and i - 1 in self.layers_to_capture
and (i - 1 in self.layers_to_capture)
else None,
)
last_layer = self.layers[self.end_layer - 1]
hidden_states, residual = last_layer.layer_communicator.finish_layer_stack(
hidden_states,
residual,
forward_batch,
preserve_declared=not self.pp_group.is_last_rank,
)
if not self.pp_group.is_last_rank:
return PPProxyTensors(
{"hidden_states": hidden_states, "residual": residual}
)
return residual_batch.to_pp(hidden_states, forward_batch)
else:
hidden_states = residual_batch.finish(hidden_states, forward_batch)
if not forward_batch.forward_mode.is_idle():
hidden_states = last_layer.layer_communicator.norm_output(
hidden_states = residual_batch.norm(
hidden_states,
residual,
forward_batch,
self.norm,
capture_output=(
capture_output
if capture_aux and self.end_layer - 1 in self.layers_to_capture
else None
),
capture_output=capture_output
if capture_aux and self.end_layer - 1 in self.layers_to_capture
else None,
)
if capture_aux and len(dspark_aux_hidden_states) > 0:
return hidden_states, dspark_aux_hidden_states
+6 -1
View File
@@ -3794,7 +3794,12 @@ class DeepseekV4DecoderLayer(nn.Module):
and forward_batch.dp_padding_mode.is_max_len()
and get_parallel().tp_size == get_parallel().attn_dp_size
)
mlp_reduce_scatter = _use_cp or _use_reduce_scatterv or _use_reduce_scatter
mlp_reduce_scatter = (
_use_cp
or _use_reduce_scatterv
or _use_reduce_scatter
or (_use_tp_moe_gather and should_use_dp_reduce_scatterv())
)
# PoC (SGLANG_DP_SHARED_EXPERT_LOCAL): compute the replicated shared expert
# on LOCAL hidden before the gather and add it back after the combine
# (reduce_scatterv OR dp_scatter), instead of on the gathered global buffer.
+38 -35
View File
@@ -11,7 +11,12 @@ from sglang.srt.layers.attention.hybrid_linear_attn_backend import (
Mamba2AttnBackend,
)
from sglang.srt.layers.attention.mamba.mamba import MambaMixer2
from sglang.srt.layers.communicator import LayerCommunicator, LayerFacts
from sglang.srt.layers.communicator import (
declare_attn,
declare_ffn,
make_stages,
)
from sglang.srt.layers.communicator.output import OutputTransform
from sglang.srt.layers.communicator.residual import batch as residual_batch
from sglang.srt.layers.dp_attention import (
is_dp_attention_enabled,
@@ -90,9 +95,11 @@ class FalconH1MLP(nn.Module):
x = self.act_fn(gate_up)
x, _ = self.down_proj(x)
x = x * self.down_multiplier
return x
def scale_output(self, x):
return x * self.down_multiplier
class FalconH1HybridAttentionDecoderLayer(nn.Module):
def __init__(
@@ -199,14 +206,6 @@ class FalconH1HybridAttentionDecoderLayer(nn.Module):
is_previous_layer_sparse = False
is_next_layer_sparse = False
self.layer_facts = LayerFacts.init_new(
layer_id=layer_id,
num_layers=config.num_hidden_layers,
is_layer_sparse=self.is_layer_sparse,
is_previous_layer_sparse=is_previous_layer_sparse,
is_next_layer_sparse=is_next_layer_sparse,
)
self.feed_forward = FalconH1MLP(
hidden_size=self.hidden_size,
intermediate_size=config.intermediate_size,
@@ -223,12 +222,24 @@ class FalconH1HybridAttentionDecoderLayer(nn.Module):
self.q_norm = RMSNorm(self.head_dim, eps=config.rms_norm_eps)
self.k_norm = RMSNorm(self.head_dim, eps=config.rms_norm_eps)
self.layer_communicator = LayerCommunicator(
layer_facts=self.layer_facts,
input_layernorm=self.input_layernorm,
post_attention_layernorm=self.pre_ff_layernorm,
allow_reduce_scatter=True,
allow_deferred_ffn_reduction=False,
self.attn_stage, self.ffn_stage = make_stages(
(declare_attn(), self.input_layernorm),
(
declare_ffn(
output=OutputTransform(
self.feed_forward.scale_output, before_reduce_scatter=True
),
sparse=self.is_layer_sparse,
next_sparse=is_next_layer_sparse,
),
self.pre_ff_layernorm,
),
previous=declare_ffn(
sparse=is_previous_layer_sparse, next_sparse=self.is_layer_sparse
)
if layer_id != 0
else None,
terminal=layer_id == config.num_hidden_layers - 1,
)
self.alt_stream = alt_stream
@@ -320,13 +331,10 @@ class FalconH1HybridAttentionDecoderLayer(nn.Module):
self,
positions: torch.Tensor,
hidden_states: torch.Tensor,
residual: Optional[torch.Tensor],
forward_batch: ForwardBatch,
**kwargs: Any,
):
hidden_states, residual = self.layer_communicator.prepare_attn(
hidden_states, residual, forward_batch
)
hidden_states = self.attn_stage.prepare(hidden_states, forward_batch)
if not forward_batch.forward_mode.is_idle():
# Attention block
@@ -355,12 +363,11 @@ class FalconH1HybridAttentionDecoderLayer(nn.Module):
hidden_states = attention_hidden_states + mamba_hidden_states
# Fully Connected
hidden_states, residual = self.layer_communicator.prepare_mlp(
hidden_states, residual, forward_batch
)
with self.layer_communicator.ffn_exit(forward_batch) as ffn_exit:
hidden_states = self.attn_stage.finish(hidden_states, forward_batch)
hidden_states = self.ffn_stage.prepare(hidden_states, forward_batch)
with self.ffn_stage.exit(forward_batch) as ffn_exit:
hidden_states = self.feed_forward(hidden_states, forward_batch)
return ffn_exit.finish(hidden_states, residual)
return ffn_exit.finish(hidden_states)
ALL_DECODER_LAYER_TYPES = {
@@ -422,26 +429,22 @@ class FalconH1Model(nn.Module):
else:
hidden_states = self.embed_tokens(input_ids) * self.embedding_multiplier
residual = residual_batch.start(forward_batch)
residual_batch.start(forward_batch)
for i in range(len(self.layers)):
layer = self.layers[i]
hidden_states, residual = layer(
hidden_states = layer(
layer_id=i,
positions=positions,
hidden_states=hidden_states,
residual=residual,
forward_batch=forward_batch,
)
hidden_states, residual = self.layers[-1].layer_communicator.finish_layer_stack(
hidden_states, residual, forward_batch
)
hidden_states = residual_batch.finish(hidden_states, forward_batch)
if not forward_batch.forward_mode.is_idle():
if residual is None:
hidden_states = self.final_layernorm(hidden_states)
else:
hidden_states, _ = self.final_layernorm(hidden_states, residual)
hidden_states = residual_batch.norm(
hidden_states, forward_batch, self.final_layernorm
)
return hidden_states
+39 -57
View File
@@ -31,8 +31,9 @@ from sglang.srt.eplb.expert_distribution import get_global_expert_distribution_r
from sglang.srt.eplb.expert_location import ModelConfigForExpertLocation
from sglang.srt.layers.aux_hidden_states import AuxHiddenStateList
from sglang.srt.layers.communicator import (
LayerCommunicator,
LayerFacts,
declare_attn,
declare_ffn,
make_stages,
)
from sglang.srt.layers.communicator.residual import batch as residual_batch
from sglang.srt.layers.dp_attention import (
@@ -570,14 +571,6 @@ class GptOssDecoderLayer(nn.Module):
is_previous_layer_sparse = True
is_next_layer_sparse = True
self.layer_facts = LayerFacts.init_new(
layer_id=layer_id,
num_layers=config.num_hidden_layers,
is_layer_sparse=self.is_layer_sparse,
is_previous_layer_sparse=is_previous_layer_sparse,
is_next_layer_sparse=is_next_layer_sparse,
)
if self.is_layer_sparse:
self.mlp = GptOssSparseMoeBlock(
layer_id=self.layer_id,
@@ -606,11 +599,21 @@ class GptOssDecoderLayer(nn.Module):
x_pad_to_multiple=post_attn_pad_multiple,
)
self.layer_communicator = LayerCommunicator(
layer_facts=self.layer_facts,
input_layernorm=self.input_layernorm,
post_attention_layernorm=self.post_attention_layernorm,
allow_reduce_scatter=True,
self.attn_stage, self.ffn_stage = make_stages(
(declare_attn(), self.input_layernorm),
(
declare_ffn(
sparse=self.is_layer_sparse,
next_sparse=is_next_layer_sparse,
),
self.post_attention_layernorm,
),
previous=declare_ffn(
sparse=is_previous_layer_sparse, next_sparse=self.is_layer_sparse
)
if layer_id != 0
else None,
terminal=layer_id == config.num_hidden_layers - 1,
)
def forward(
@@ -618,16 +621,10 @@ class GptOssDecoderLayer(nn.Module):
positions: torch.Tensor,
hidden_states: torch.Tensor,
forward_batch: ForwardBatch,
residual: Optional[torch.Tensor],
capture_output=None,
) -> Tuple[torch.Tensor, torch.Tensor]:
hidden_states, residual = (
self.layer_communicator.prepare_attn_and_capture_last_layer_outputs(
hidden_states,
residual,
forward_batch,
capture_output=capture_output,
)
) -> torch.Tensor:
hidden_states = self.attn_stage.prepare(
hidden_states, forward_batch, capture_output=capture_output
)
if hidden_states.shape[0] != 0:
@@ -637,15 +634,14 @@ class GptOssDecoderLayer(nn.Module):
forward_batch=forward_batch,
)
hidden_states, residual = self.layer_communicator.prepare_mlp(
hidden_states, residual, forward_batch
)
hidden_states = self.attn_stage.finish(hidden_states, forward_batch)
hidden_states = self.ffn_stage.prepare(hidden_states, forward_batch)
with self.layer_communicator.ffn_exit(forward_batch) as ffn_exit:
with self.ffn_stage.exit(forward_batch) as ffn_exit:
hidden_states = self.mlp(hidden_states, forward_batch)
hidden_states, residual = ffn_exit.finish(hidden_states, residual)
hidden_states = ffn_exit.finish(hidden_states)
return hidden_states, residual
return hidden_states
class GptOssModel(nn.Module):
@@ -708,12 +704,12 @@ class GptOssModel(nn.Module):
hidden_states = self.embed_tokens(input_ids)
else:
hidden_states = input_embeds
residual = residual_batch.start(forward_batch)
residual_batch.start(forward_batch)
else:
assert pp_proxy_tensors is not None
hidden_states, residual = self.layers[
self.start_layer
].layer_communicator.from_pp(pp_proxy_tensors, forward_batch)
hidden_states = self.layers[self.start_layer].attn_stage.from_pp(
pp_proxy_tensors, forward_batch
)
# Capture hidden-state boundaries: boundary 0 is the embedding output,
# and boundary i + 1 is the output after transformer block i.
@@ -721,44 +717,30 @@ class GptOssModel(nn.Module):
for i in range(self.start_layer, self.end_layer):
with get_global_expert_distribution_recorder().with_current_layer(i):
layer = self.layers[i]
hidden_states, residual = layer(
hidden_states = layer(
positions,
hidden_states,
forward_batch,
residual,
capture_output=aux_hidden_states.capture
if i in self.layers_to_capture
else None,
)
last_layer = self.layers[self.end_layer - 1]
hidden_states, residual = last_layer.layer_communicator.finish_layer_stack(
hidden_states,
residual,
forward_batch,
preserve_declared=not self.pp_group.is_last_rank,
)
if not self.pp_group.is_last_rank:
return PPProxyTensors(
{
"hidden_states": hidden_states,
"residual": residual,
}
)
return residual_batch.to_pp(hidden_states, forward_batch)
else:
hidden_states = residual_batch.finish(hidden_states, forward_batch)
if hidden_states.shape[0] != 0:
hidden_states = last_layer.layer_communicator.norm_output(
hidden_states = residual_batch.norm(
hidden_states,
residual,
forward_batch,
self.norm,
capture_output=(
aux_hidden_states.append
if self.end_layer in self.layers_to_capture
else None
),
capture_output=aux_hidden_states.capture
if self.end_layer in self.layers_to_capture
else None,
)
elif self.end_layer in self.layers_to_capture:
aux_hidden_states.append(
last_layer.layer_communicator.snapshot(hidden_states, residual)
residual_batch.snapshot(hidden_states, forward_batch)
)
if len(aux_hidden_states) == 0:
return hidden_states
+35 -54
View File
@@ -32,9 +32,10 @@ from sglang.srt.eplb.expert_location import ModelConfigForExpertLocation
from sglang.srt.eplb.expert_location_dispatch import ExpertLocationDispatchInfo
from sglang.srt.layers.activation import SiluAndMul
from sglang.srt.layers.communicator import (
LayerCommunicator,
LayerFacts,
declare_attn,
declare_ffn,
enable_moe_dense_fully_dp,
make_stages,
)
from sglang.srt.layers.communicator.residual import batch as residual_batch
from sglang.srt.layers.dp_attention import (
@@ -613,14 +614,6 @@ class LLaDA2MoeBlock(nn.Module):
is_previous_layer_sparse = self._is_layer_sparse(config, layer_id=layer_id - 1)
is_next_layer_sparse = self._is_layer_sparse(config, layer_id=layer_id + 1)
self.layer_facts = LayerFacts.init_new(
layer_id=layer_id,
num_layers=config.num_hidden_layers,
is_layer_sparse=self.is_layer_sparse,
is_previous_layer_sparse=is_previous_layer_sparse,
is_next_layer_sparse=is_next_layer_sparse,
)
if self.is_layer_sparse:
self.mlp = LLaDA2MoeSparseMoeBlock(
layer_id=layer_id,
@@ -645,12 +638,21 @@ class LLaDA2MoeBlock(nn.Module):
self.post_attention_layernorm = RMSNorm(hidden_size, eps=config.rms_norm_eps)
self.layer_communicator = LayerCommunicator(
layer_facts=self.layer_facts,
input_layernorm=self.input_layernorm,
post_attention_layernorm=self.post_attention_layernorm,
allow_reduce_scatter=True,
allow_deferred_ffn_reduction=False,
self.attn_stage, self.ffn_stage = make_stages(
(declare_attn(), self.input_layernorm),
(
declare_ffn(
sparse=self.is_layer_sparse,
next_sparse=is_next_layer_sparse,
),
self.post_attention_layernorm,
),
previous=declare_ffn(
sparse=is_previous_layer_sparse, next_sparse=self.is_layer_sparse
)
if layer_id != 0
else None,
terminal=layer_id == config.num_hidden_layers - 1,
)
def _is_layer_sparse(self, config: PretrainedConfig, layer_id: int) -> bool:
@@ -663,12 +665,9 @@ class LLaDA2MoeBlock(nn.Module):
positions: torch.Tensor,
hidden_states: torch.Tensor,
forward_batch: ForwardBatch,
residual: Optional[torch.Tensor],
) -> torch.Tensor:
hidden_states, residual = self.layer_communicator.prepare_attn(
hidden_states=hidden_states,
residual=residual,
forward_batch=forward_batch,
hidden_states = self.attn_stage.prepare(
hidden_states=hidden_states, forward_batch=forward_batch
)
hidden_states = self.attention(
@@ -677,17 +676,17 @@ class LLaDA2MoeBlock(nn.Module):
forward_batch=forward_batch,
)
hidden_states, residual = self.layer_communicator.prepare_mlp(
hidden_states = self.attn_stage.finish(hidden_states, forward_batch)
hidden_states = self.ffn_stage.prepare(
hidden_states=hidden_states,
residual=residual,
forward_batch=forward_batch,
)
with self.layer_communicator.ffn_exit(forward_batch) as ffn_exit:
with self.ffn_stage.exit(forward_batch) as ffn_exit:
hidden_states = self.mlp(hidden_states, forward_batch)
hidden_states, residual = ffn_exit.finish(hidden_states, residual)
hidden_states = ffn_exit.finish(hidden_states)
return hidden_states, residual
return hidden_states
class LLaDA2MoeModel(nn.Module):
@@ -747,44 +746,26 @@ class LLaDA2MoeModel(nn.Module):
hidden_states = self.word_embeddings(input_ids)
else:
hidden_states = input_embeds
residual = residual_batch.start(forward_batch)
residual_batch.start(forward_batch)
else:
assert pp_proxy_tensors is not None
hidden_states, residual = self.layers[
self.start_layer
].layer_communicator.from_pp(pp_proxy_tensors, forward_batch)
hidden_states = self.layers[self.start_layer].attn_stage.from_pp(
pp_proxy_tensors, forward_batch
)
for i in range(self.start_layer, self.end_layer):
with get_global_expert_distribution_recorder().with_current_layer(i):
layer = self.layers[i]
hidden_states, residual = layer(
positions,
hidden_states,
forward_batch,
residual,
)
hidden_states = layer(positions, hidden_states, forward_batch)
hidden_states, residual = self.layers[
self.end_layer - 1
].layer_communicator.finish_layer_stack(
hidden_states,
residual,
forward_batch,
preserve_declared=not self.pp_group.is_last_rank,
)
if not self.pp_group.is_last_rank:
return PPProxyTensors(
{
"hidden_states": hidden_states,
"residual": residual,
}
)
return residual_batch.to_pp(hidden_states, forward_batch)
else:
hidden_states = residual_batch.finish(hidden_states, forward_batch)
if not forward_batch.forward_mode.is_idle():
if residual is None:
hidden_states = self.norm(hidden_states)
else:
hidden_states, _ = self.norm(hidden_states, residual)
hidden_states = residual_batch.norm(
hidden_states, forward_batch, self.norm
)
return hidden_states
+41 -52
View File
@@ -25,7 +25,12 @@ import torch
from torch import nn
from transformers import Llama4TextConfig
from sglang.srt.layers.communicator import LayerCommunicator, LayerFacts
from sglang.srt.layers.aux_hidden_states import AuxHiddenStateList
from sglang.srt.layers.communicator import (
declare_attn,
declare_ffn,
make_stages,
)
from sglang.srt.layers.communicator.residual import batch as residual_batch
from sglang.srt.layers.dp_attention import (
is_dp_attention_enabled,
@@ -421,20 +426,19 @@ class Llama4DecoderLayer(nn.Module):
config.hidden_size, eps=config.rms_norm_eps
)
self.layer_facts = LayerFacts.init_new(
layer_id=layer_id,
num_layers=config.num_hidden_layers,
is_layer_sparse=is_moe_layer,
is_previous_layer_sparse=is_previous_moe_layer,
is_next_layer_sparse=is_next_moe_layer,
)
self.layer_communicator = LayerCommunicator(
layer_facts=self.layer_facts,
input_layernorm=self.input_layernorm,
post_attention_layernorm=self.post_attention_layernorm,
allow_reduce_scatter=True,
allow_deferred_ffn_reduction=False,
self.attn_stage, self.ffn_stage = make_stages(
(declare_attn(), self.input_layernorm),
(
declare_ffn(
sparse=is_moe_layer,
next_sparse=is_next_moe_layer,
),
self.post_attention_layernorm,
),
previous=declare_ffn(sparse=is_previous_moe_layer, next_sparse=is_moe_layer)
if layer_id != 0
else None,
terminal=layer_id == config.num_hidden_layers - 1,
)
def _is_moe_layer(self, layer_id: int) -> bool:
@@ -453,10 +457,10 @@ class Llama4DecoderLayer(nn.Module):
positions: torch.Tensor,
hidden_states: torch.Tensor,
forward_batch: ForwardBatch,
residual: Optional[torch.Tensor],
) -> Tuple[torch.Tensor, torch.Tensor]:
hidden_states, residual = self.layer_communicator.prepare_attn(
hidden_states, residual, forward_batch
capture_output=None,
) -> torch.Tensor:
hidden_states = self.attn_stage.prepare(
hidden_states, forward_batch, capture_output=capture_output
)
if hidden_states.shape[0] != 0:
@@ -466,16 +470,15 @@ class Llama4DecoderLayer(nn.Module):
forward_batch=forward_batch,
)
hidden_states, residual = self.layer_communicator.prepare_mlp(
hidden_states, residual, forward_batch
)
hidden_states = self.attn_stage.finish(hidden_states, forward_batch)
hidden_states = self.ffn_stage.prepare(hidden_states, forward_batch)
# Fully Connected
with self.layer_communicator.ffn_exit(forward_batch) as ffn_exit:
with self.ffn_stage.exit(forward_batch) as ffn_exit:
hidden_states = self.feed_forward(hidden_states, forward_batch)
hidden_states, residual = ffn_exit.finish(hidden_states, residual)
hidden_states = ffn_exit.finish(hidden_states)
return hidden_states, residual
return hidden_states
class Llama4Model(nn.Module):
@@ -519,28 +522,22 @@ class Llama4Model(nn.Module):
hidden_states = self.embed_tokens(input_ids)
else:
hidden_states = input_embeds
residual = residual_batch.start(forward_batch)
aux_hidden_states = []
residual_batch.start(forward_batch)
aux_hidden_states = AuxHiddenStateList()
for i in range(len(self.layers)):
if i in self.layers_to_capture:
aux_hidden_states.append(
self.layers[i].layer_communicator.snapshot(
hidden_states, residual, at_input=True
)
)
layer = self.layers[i]
hidden_states, residual = layer(
hidden_states = layer(
positions,
hidden_states,
forward_batch,
residual,
capture_output=aux_hidden_states.capture
if i in self.layers_to_capture
else None,
)
hidden_states, residual = self.layers[-1].layer_communicator.finish_layer_stack(
hidden_states, residual, forward_batch
)
hidden_states = residual_batch.finish(hidden_states, forward_batch)
if not forward_batch.forward_mode.is_idle():
hidden_states, _ = self.norm(hidden_states, residual)
hidden_states = residual_batch.norm(hidden_states, forward_batch, self.norm)
if len(aux_hidden_states) == 0:
return hidden_states
@@ -568,29 +565,21 @@ class Llama4ForCausalLM(LlamaForCausalLM):
):
start, end = split_interval
if start == 0:
forward_batch.residual = residual_batch.start(forward_batch)
residual_batch.start(forward_batch)
forward_batch.hidden_states = (
self.model.embed_tokens(input_ids)
if input_embeds is None
else input_embeds
)
for layer in self.model.layers[start:end]:
forward_batch.hidden_states, forward_batch.residual = layer(
positions,
forward_batch.hidden_states,
forward_batch,
forward_batch.residual,
forward_batch.hidden_states = layer(
positions, forward_batch.hidden_states, forward_batch
)
if end != self.model.config.num_hidden_layers:
return None
hidden, residual = LayerCommunicator.finish_layer_stack(
forward_batch.hidden_states, forward_batch.residual, forward_batch
forward_batch.hidden_states = residual_batch.norm(
forward_batch.hidden_states, forward_batch, self.model.norm
)
forward_batch.residual = None
if residual is None:
forward_batch.hidden_states = self.model.norm(hidden)
else:
forward_batch.hidden_states, _ = self.model.norm(hidden, residual)
return self.logits_processor(
input_ids, forward_batch.hidden_states, self.lm_head, forward_batch
)
+28 -46
View File
@@ -77,7 +77,7 @@ from sglang.srt.model_loader.weight_utils import (
replace_prefix,
replace_substrings,
)
from sglang.srt.models.nemotron_h_utils import make_layer_communicator
from sglang.srt.models.nemotron_h_utils import make_stage_boundary
from sglang.srt.models.utils import WeightsMapper
from sglang.srt.runtime_context import get_exec, get_forward, get_parallel
from sglang.srt.utils import (
@@ -375,8 +375,8 @@ class NemotronHMLPLikeDecoderLayer(nn.Module):
mixer's attention partial sum, or a value that is complete or carries the
sum an FFN before it left."""
def _init_layer_communicator(self, config: NemotronHConfig, layer_idx: int):
self.layer_communicator = make_layer_communicator(
def _init_stage_boundary(self, config: NemotronHConfig, layer_idx: int):
self.stage = make_stage_boundary(
self.norm, pattern=config.hybrid_override_pattern, layer_idx=layer_idx
)
@@ -384,15 +384,12 @@ class NemotronHMLPLikeDecoderLayer(nn.Module):
self,
*,
hidden_states,
residual: torch.Tensor | None,
forward_batch: ForwardBatch,
):
hidden_states, residual = self.layer_communicator.prepare_mlp(
hidden_states, residual, forward_batch
)
with self.layer_communicator.ffn_exit(forward_batch) as ffn_exit:
hidden_states = self.stage.prepare(hidden_states, forward_batch)
with self.stage.exit(forward_batch) as ffn_exit:
hidden_states = self.mixer.forward(hidden_states)
return ffn_exit.finish(hidden_states, residual)
return ffn_exit.finish(hidden_states)
class NemotronHMLPDecoderLayer(NemotronHMLPLikeDecoderLayer):
@@ -426,7 +423,7 @@ class NemotronHMLPDecoderLayer(NemotronHMLPLikeDecoderLayer):
)
self.norm = RMSNorm(config.hidden_size, eps=config.layer_norm_epsilon)
self._init_layer_communicator(config, layer_idx)
self._init_stage_boundary(config, layer_idx)
class NemotronHMoEDecoderLayer(NemotronHMLPLikeDecoderLayer):
@@ -449,7 +446,7 @@ class NemotronHMoEDecoderLayer(NemotronHMLPLikeDecoderLayer):
)
self.norm = RMSNorm(config.hidden_size, eps=config.layer_norm_epsilon)
self._init_layer_communicator(config, layer_idx)
self._init_stage_boundary(config, layer_idx)
class NemotronHAttnLikeDecoderLayer(nn.Module):
@@ -457,8 +454,8 @@ class NemotronHAttnLikeDecoderLayer(nn.Module):
output all-reduce to that stage's input; otherwise it reduces the output
itself unless the fused kernel takes the sum into the next input norm."""
def _init_layer_communicator(self, config: NemotronHConfig, layer_idx: int):
self.layer_communicator = make_layer_communicator(
def _init_stage_boundary(self, config: NemotronHConfig, layer_idx: int):
self.stage = make_stage_boundary(
self.norm, pattern=config.hybrid_override_pattern, layer_idx=layer_idx
)
@@ -466,20 +463,17 @@ class NemotronHAttnLikeDecoderLayer(nn.Module):
self,
*,
hidden_states,
residual: torch.Tensor | None,
forward_batch: ForwardBatch,
):
hidden_states, residual = self.layer_communicator.prepare_attn(
hidden_states, residual, forward_batch
)
hidden_states = self.stage.prepare(hidden_states, forward_batch)
if forward_batch.forward_mode.is_idle():
return hidden_states, residual
return hidden_states
with self.layer_communicator.mixer_exit(forward_batch) as mixer_exit:
with self.stage.exit(forward_batch) as mixer_exit:
hidden_states = self._forward_mixer(
hidden_states, forward_batch, mixer_exit.skips_reduction
)
return mixer_exit.finish(hidden_states, residual)
return mixer_exit.finish(hidden_states)
class NemotronHMambaDecoderLayer(NemotronHAttnLikeDecoderLayer):
@@ -507,7 +501,7 @@ class NemotronHMambaDecoderLayer(NemotronHAttnLikeDecoderLayer):
)
self.norm = RMSNorm(config.hidden_size, eps=config.layer_norm_epsilon)
self._init_layer_communicator(config, layer_idx)
self._init_stage_boundary(config, layer_idx)
def _forward_mamba(
self,
@@ -645,7 +639,7 @@ class NemotronHAttentionDecoderLayer(NemotronHAttnLikeDecoderLayer):
)
self.norm = RMSNorm(config.hidden_size, eps=config.layer_norm_epsilon)
self._init_layer_communicator(config, layer_idx)
self._init_stage_boundary(config, layer_idx)
def _forward_mixer(
self,
@@ -733,50 +727,38 @@ class NemotronHModel(nn.Module):
hidden_states = inputs_embeds
else:
hidden_states = self.embed_tokens(input_ids)
residual = residual_batch.start(forward_batch)
residual_batch.start(forward_batch)
else:
assert pp_proxy_tensors is not None
hidden_states, residual = self.layers[
self.start_layer
].layer_communicator.from_pp(pp_proxy_tensors, forward_batch)
hidden_states = self.layers[self.start_layer].stage.from_pp(
pp_proxy_tensors, forward_batch
)
aux_hidden_states = []
for i in range(self.start_layer, self.end_layer):
if i in self.layers_to_capture:
hidden_states, snapshot = self.layers[
i
].layer_communicator.capture_output(
hidden_states, residual, at_input=True
hidden_states, snapshot = self.layers[i].stage.capture_output(
hidden_states, forward_batch, at_input=True
)
aux_hidden_states.append(snapshot)
layer = self.layers[i]
if not isinstance(layer, Layers):
raise ValueError(f"Unknown layer type: {type(layer)}")
hidden_states, residual = layer.forward(
hidden_states = layer.forward(
hidden_states=hidden_states,
residual=residual,
forward_batch=forward_batch,
)
last_layer = self.layers[self.end_layer - 1]
hidden_states, residual = last_layer.layer_communicator.finish_layer_stack(
hidden_states,
residual,
forward_batch,
preserve_declared=not self.pp_group.is_last_rank,
)
if not self.pp_group.is_last_rank:
return PPProxyTensors(
{"hidden_states": hidden_states, "residual": residual}
return residual_batch.to_pp(
hidden_states, forward_batch, preserve_declared=True
)
hidden_states = residual_batch.finish(hidden_states, forward_batch)
if self.end_layer in self.layers_to_capture:
aux_hidden_states.append(
last_layer.layer_communicator.snapshot(hidden_states, residual)
residual_batch.snapshot(hidden_states, forward_batch)
)
if residual is None:
hidden_states = self.norm_f(hidden_states)
else:
hidden_states, _ = self.norm_f(hidden_states, residual)
hidden_states = residual_batch.norm(hidden_states, forward_batch, self.norm_f)
if aux_hidden_states:
return hidden_states, aux_hidden_states
return hidden_states
+23 -30
View File
@@ -89,9 +89,8 @@ class NemotronHMTPAttentionDecoderLayer(NemotronHAttentionDecoderLayer):
*,
inputs_embeds: torch.Tensor,
hidden_states: torch.Tensor,
residual: torch.Tensor | None = None,
forward_batch: ForwardBatch,
) -> tuple[torch.Tensor, torch.Tensor]:
) -> torch.Tensor:
if self.has_start_projections:
inputs_embeds_normed = self.enorm(inputs_embeds)
previous_hidden_states_normed = self.hnorm(hidden_states)
@@ -105,23 +104,22 @@ class NemotronHMTPAttentionDecoderLayer(NemotronHAttentionDecoderLayer):
hidden_states, dim=-1
)
hidden_states, residual = super().forward(
if self.has_start_projections:
hidden_states = residual_batch.written(hidden_states, forward_batch)
hidden_states = super().forward(
hidden_states=hidden_states,
residual=residual,
forward_batch=forward_batch,
)
if self.has_end_norm:
hidden_states, residual = self.layer_communicator.finish_layer_stack(
hidden_states, residual, forward_batch
)
hidden_states, residual = self.layer_communicator.fold(
hidden_states, residual
hidden_states = residual_batch.fold(hidden_states, forward_batch)
hidden_states = residual_batch.written(
self.final_layernorm(hidden_states), forward_batch
)
hidden_states = self.final_layernorm(hidden_states)
return hidden_states, residual
return hidden_states
class NemotronHMTPMoEDecoderLayer(NemotronHMoEDecoderLayer):
@@ -173,9 +171,8 @@ class NemotronHMTPMoEDecoderLayer(NemotronHMoEDecoderLayer):
*,
inputs_embeds: torch.Tensor,
hidden_states: torch.Tensor,
residual: torch.Tensor | None = None,
forward_batch: ForwardBatch,
) -> tuple[torch.Tensor, torch.Tensor]:
) -> torch.Tensor:
if self.has_start_projections:
inputs_embeds_normed = self.enorm(inputs_embeds)
previous_hidden_states_normed = self.hnorm(hidden_states)
@@ -189,23 +186,22 @@ class NemotronHMTPMoEDecoderLayer(NemotronHMoEDecoderLayer):
hidden_states, dim=-1
)
hidden_states, residual = super().forward(
if self.has_start_projections:
hidden_states = residual_batch.written(hidden_states, forward_batch)
hidden_states = super().forward(
hidden_states=hidden_states,
residual=residual,
forward_batch=forward_batch,
)
if self.has_end_norm:
hidden_states, residual = self.layer_communicator.finish_layer_stack(
hidden_states, residual, forward_batch
)
hidden_states, residual = self.layer_communicator.fold(
hidden_states, residual
hidden_states = residual_batch.fold(hidden_states, forward_batch)
hidden_states = residual_batch.written(
self.final_layernorm(hidden_states), forward_batch
)
hidden_states = self.final_layernorm(hidden_states)
return hidden_states, residual
return hidden_states
class NemotronHMultiTokenPredictor(nn.Module):
@@ -302,18 +298,15 @@ class NemotronHMultiTokenPredictor(nn.Module):
inputs_embeds = self.get_input_embeddings(input_ids)
hidden_states = forward_batch.spec_info.hidden_states
residual = None
residual_batch.start(forward_batch)
for i in range(self.pattern_len):
if residual is None:
residual = residual_batch.start(forward_batch)
hidden_states, residual = self.layers[str(i)](
hidden_states = self.layers[str(i)](
inputs_embeds=inputs_embeds,
hidden_states=hidden_states,
residual=residual,
forward_batch=forward_batch,
)
return hidden_states
return residual_batch.take_output(hidden_states, forward_batch)
class NemotronHForCausalLMMTP(NemotronHForCausalLM):
+4 -4
View File
@@ -101,14 +101,12 @@ def layer_stage(pattern: str, layer_idx: int) -> LayerStage:
)
def make_layer_communicator(
layer_norm: RMSNorm, *, pattern: str, layer_idx: int
) -> LayerCommunicator:
def make_stage_boundary(layer_norm: RMSNorm, *, pattern: str, layer_idx: int):
"""The communicator of a layer that is one stage: only its own norm, and
boundaries built from the stages next to it in the pattern."""
stage = layer_stage(pattern, layer_idx)
for_attn = stage.kind is StageKind.ATTENTION
return LayerCommunicator(
boundaries = LayerCommunicator(
layer_facts=LayerFacts(
is_layer_sparse=pattern[layer_idx] == MOE,
is_last_layer=layer_idx == len(pattern) - 1,
@@ -121,3 +119,5 @@ def make_layer_communicator(
allow_reduce_scatter=not for_attn,
stage=stage,
)
return boundaries.attn if for_attn else boundaries.ffn
+53 -83
View File
@@ -1,12 +1,16 @@
# Adapted from qwen2.py
import logging
from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
from typing import Any, Dict, Iterable, List, Optional, Tuple
import torch
from torch import nn
from sglang.srt.layers.communicator import LayerCommunicator, LayerFacts
from sglang.srt.layers.communicator.residual import access as residual_access
from sglang.srt.layers.aux_hidden_states import AuxHiddenStateList
from sglang.srt.layers.communicator import (
declare_attn,
declare_ffn,
make_stages,
)
from sglang.srt.layers.communicator.residual import batch as residual_batch
from sglang.srt.layers.layernorm import RMSNorm
from sglang.srt.layers.linear import QKVParallelLinear, RowParallelLinear
@@ -372,18 +376,14 @@ class Qwen3DecoderLayer(nn.Module):
config.hidden_size, eps=config.rms_norm_eps, **norm_kwargs
)
self.layer_facts = LayerFacts.init_new(
layer_id=layer_id,
num_layers=config.num_hidden_layers,
is_layer_sparse=False,
is_previous_layer_sparse=False,
is_next_layer_sparse=False,
)
self.layer_communicator = LayerCommunicator(
layer_facts=self.layer_facts,
input_layernorm=self.input_layernorm,
post_attention_layernorm=self.post_attention_layernorm,
allow_deferred_ffn_reduction=False,
self.attn_stage, self.ffn_stage = make_stages(
(declare_attn(), self.input_layernorm),
(
declare_ffn(sparse=False, next_sparse=False),
self.post_attention_layernorm,
),
previous=declare_ffn() if layer_id != 0 else None,
terminal=layer_id == config.num_hidden_layers - 1,
)
def forward(
@@ -391,15 +391,15 @@ class Qwen3DecoderLayer(nn.Module):
positions: torch.Tensor,
hidden_states: torch.Tensor,
forward_batch: ForwardBatch,
residual: Optional[torch.Tensor],
post_residual_addition: Optional[torch.Tensor] = None,
) -> Tuple[torch.Tensor, torch.Tensor]:
capture_output=None,
) -> torch.Tensor:
# Self Attention
hidden_states, residual = self.layer_communicator.prepare_attn(
hidden_states = self.attn_stage.prepare(
hidden_states,
residual,
forward_batch,
post_residual_addition=post_residual_addition,
capture_output=capture_output,
)
if hidden_states.shape[0] != 0:
hidden_states = self.self_attn(
@@ -409,9 +409,9 @@ class Qwen3DecoderLayer(nn.Module):
)
# Fully Connected
hidden_states, residual = self.layer_communicator.prepare_mlp(
hidden_states = self.attn_stage.finish(hidden_states, forward_batch)
hidden_states = self.ffn_stage.prepare(
hidden_states,
residual,
forward_batch,
cache=(
[self.mlp.gate_up_proj.weight, self.mlp.down_proj.weight]
@@ -424,12 +424,12 @@ class Qwen3DecoderLayer(nn.Module):
else None
),
)
with self.layer_communicator.ffn_exit(forward_batch) as ffn_exit:
with self.ffn_stage.exit(forward_batch) as ffn_exit:
hidden_states = self.mlp(hidden_states, forward_batch=forward_batch)
if _is_npu and get_cmo_stream():
wait_cmo_stream()
hidden_states, residual = ffn_exit.finish(hidden_states, residual)
return hidden_states, residual
hidden_states = ffn_exit.finish(hidden_states)
return hidden_states
class Qwen3Model(Qwen2Model):
@@ -450,60 +450,39 @@ class Qwen3Model(Qwen2Model):
def forward(
self,
input_ids: torch.Tensor,
positions: torch.Tensor,
forward_batch: ForwardBatch,
input_embeds: torch.Tensor = None,
pp_proxy_tensors: Optional[PPProxyTensors] = None,
) -> Union[torch.Tensor, PPProxyTensors]:
input_ids,
positions,
forward_batch,
input_embeds=None,
pp_proxy_tensors=None,
):
if self.pp_group.is_first_rank:
if input_embeds is None:
hidden_states = self.embed_tokens(input_ids)
else:
hidden_states = input_embeds
residual = residual_batch.start(forward_batch)
hidden_states = (
self.embed_tokens(input_ids) if input_embeds is None else input_embeds
)
residual_batch.start(forward_batch)
else:
assert pp_proxy_tensors is not None
hidden_states, residual = self.layers[
self.start_layer
].layer_communicator.from_pp(pp_proxy_tensors, forward_batch)
aux_hidden_states = []
hidden_states = self.layers[self.start_layer].attn_stage.from_pp(
pp_proxy_tensors, forward_batch
)
aux_hidden_states = AuxHiddenStateList()
for i in range(self.start_layer, self.end_layer):
if i in self.layers_to_capture:
aux_hidden_states.append(
residual_access.snapshot(hidden_states, residual)
)
layer = self.layers[i]
hidden_states, residual = layer(
hidden_states = self.layers[i](
positions,
hidden_states,
forward_batch,
residual,
capture_output=aux_hidden_states.capture
if i in self.layers_to_capture
else None,
)
hidden_states, residual = residual_access.finish_layer_stack(
hidden_states, residual, forward_batch
)
if not self.pp_group.is_last_rank:
return PPProxyTensors(
{
"hidden_states": hidden_states,
"residual": residual,
}
)
else:
if hidden_states.shape[0] != 0:
if residual is None:
hidden_states = self.norm(hidden_states)
else:
hidden_states, _ = self.norm(hidden_states, residual)
if len(aux_hidden_states) == 0:
return hidden_states
return hidden_states, aux_hidden_states
return residual_batch.to_pp(hidden_states, forward_batch)
hidden_states = residual_batch.finish(hidden_states, forward_batch)
if hidden_states.shape[0] != 0:
hidden_states = residual_batch.norm(hidden_states, forward_batch, self.norm)
return (
(hidden_states, aux_hidden_states) if aux_hidden_states else hidden_states
)
class Qwen3ForCausalLM(nn.Module):
@@ -613,7 +592,7 @@ class Qwen3ForCausalLM(nn.Module):
start, end = split_interval
# embed
if start == 0:
forward_batch.residual = residual_batch.start(forward_batch)
residual_batch.start(forward_batch)
if input_embeds is None:
forward_batch.hidden_states = self.model.embed_tokens(input_ids)
else:
@@ -621,25 +600,16 @@ class Qwen3ForCausalLM(nn.Module):
# decoder layer
for i in range(start, end):
layer = self.model.layers[i]
forward_batch.hidden_states, forward_batch.residual = layer(
forward_batch.hidden_states = layer(
positions,
forward_batch.hidden_states,
forward_batch,
forward_batch.residual,
)
if end == self.model.config.num_hidden_layers:
forward_batch.hidden_states, forward_batch.residual = self.model.layers[
end - 1
].layer_communicator.finish_layer_stack(
forward_batch.hidden_states, forward_batch.residual, forward_batch
forward_batch.hidden_states = residual_batch.norm(
forward_batch.hidden_states, forward_batch, self.model.norm
)
# norm
hidden_states, _ = self.model.norm(
forward_batch.hidden_states, forward_batch.residual
)
forward_batch.residual = None
forward_batch.hidden_states = hidden_states
# logits process
result = self.logits_processor(
input_ids, forward_batch.hidden_states, self.lm_head, forward_batch
+27 -39
View File
@@ -37,6 +37,7 @@ from sglang.srt.layers.attention.vision import (
VisionAttentionMetadata,
prepare_vision_attention_metadata,
)
from sglang.srt.layers.aux_hidden_states import AuxHiddenStateList
from sglang.srt.layers.communicator.residual import batch as residual_batch
from sglang.srt.layers.conv import Conv3dLayer
from sglang.srt.layers.dp_attention import (
@@ -1205,24 +1206,23 @@ class Qwen3LLMModel(Qwen3Model):
hidden_states = self.embed_tokens(input_ids)
else:
hidden_states = input_embeds
residual = residual_batch.start(forward_batch)
residual_batch.start(forward_batch)
else:
assert pp_proxy_tensors is not None
hidden_states, residual = self.layers[
self.start_layer
].layer_communicator.from_pp(pp_proxy_tensors, forward_batch)
hidden_states = self.layers[self.start_layer].attn_stage.from_pp(
pp_proxy_tensors, forward_batch
)
aux_hidden_states = []
aux_hidden_states = AuxHiddenStateList()
for layer_idx, layer in enumerate(
self.layers[self.start_layer : self.end_layer]
):
layer_idx = layer_idx + self.start_layer
if layer_idx in self.layers_to_capture:
aux_hidden_states.append(
layer.layer_communicator.snapshot(
hidden_states, residual, at_input=True
)
)
capture_output = (
aux_hidden_states.capture
if layer_idx in self.layers_to_capture
else None
)
if self.use_hf_deepstack_order:
# HF-order path (RL on-policy / FSDP). SGLang applies residual at the START of the
@@ -1231,40 +1231,33 @@ class Qwen3LLMModel(Qwen3Model):
deepstack_embeds = self.get_deepstack_embeds(
layer_idx - 1, input_deepstack_embeds
)
hidden_states, residual = layer(
hidden_states = layer(
positions,
hidden_states,
forward_batch,
residual,
post_residual_addition=deepstack_embeds,
capture_output=capture_output,
)
else:
# Inference path: add deepstack directly to hidden_states at the end of the layer
# (original, grounding-correct order).
hidden_states, residual = layer(
hidden_states = layer(
positions,
hidden_states,
forward_batch,
residual,
capture_output=capture_output,
)
if (
input_deepstack_embeds is not None
and layer_idx in self.deepstack_embed_to_decoder_layer
):
sep = self.hidden_size * layer_idx
hidden_states.add_(
input_deepstack_embeds[:, sep : sep + self.hidden_size]
hidden_states = residual_batch.add_to_output(
hidden_states,
forward_batch,
input_deepstack_embeds[:, sep : sep + self.hidden_size],
)
hidden_states, residual = self.layers[
self.end_layer - 1
].layer_communicator.finish_layer_stack(
hidden_states,
residual,
forward_batch,
preserve_declared=not self.pp_group.is_last_rank,
)
# Handle deepstack for the last processed layer (HF-order path only).
last_deepstack = (
self.get_deepstack_embeds(self.end_layer - 1, input_deepstack_embeds)
@@ -1273,20 +1266,15 @@ class Qwen3LLMModel(Qwen3Model):
)
if not self.pp_group.is_last_rank:
return PPProxyTensors(
{
"hidden_states": hidden_states,
"residual": residual,
}
return residual_batch.to_pp(hidden_states, forward_batch)
hidden_states = residual_batch.finish(hidden_states, forward_batch)
if hidden_states.shape[0] != 0:
hidden_states = residual_batch.norm(
hidden_states,
forward_batch,
self.norm,
post_residual_addition=last_deepstack,
)
else:
if hidden_states.shape[0] != 0:
if residual is None:
hidden_states = self.norm(hidden_states)
else:
hidden_states, _ = self.norm(
hidden_states, residual, post_residual_addition=last_deepstack
)
if len(aux_hidden_states) == 0:
return hidden_states
+5 -3
View File
@@ -72,7 +72,7 @@ from sglang.srt.models.qwen4_exp_ple_table import (
make_ple_file_prefetcher,
make_ple_file_rss_trimmer,
)
from sglang.srt.runtime_context import get_parallel
from sglang.srt.runtime_context import get_forward, get_parallel
from sglang.srt.utils import get_bool_env_var, is_hip, logger
_use_aiter = get_bool_env_var("SGLANG_USE_AITER") and is_hip()
@@ -1439,14 +1439,16 @@ class Qwen4ExpLayerExtensionMixin:
attn_tp_chunks = list(hidden_states.tensor_split(attn_tp_size))
hidden_states = attn_tp_chunks[get_parallel().attn_tp_rank].contiguous()
hidden_states = self.mlp(hidden_states, forward_batch)
use_reduce_scatterv = use_dp_moe_gather and should_use_dp_reduce_scatterv()
with get_forward().scoped(mlp_reduce_scatter=use_reduce_scatterv):
hidden_states = self.mlp(hidden_states, forward_batch)
if use_dp_moe_gather:
hidden_states, global_hidden_states = (
get_local_dp_buffer(get_parallel().tp_group),
hidden_states,
)
if should_use_dp_reduce_scatterv():
if use_reduce_scatterv:
get_parallel().tp_group.reduce_scatterv(
global_hidden_states,
output=hidden_states,
+8 -18
View File
@@ -1456,36 +1456,26 @@ class SarvamMoEForCausalLM(BailingMoEForCausalLM):
start, end = split_interval
if start == 0:
residual_batch.start(forward_batch)
if input_embeds is None:
forward_batch.hidden_states = self.model.word_embeddings(input_ids)
else:
forward_batch.hidden_states = input_embeds
forward_batch.residual = residual_batch.start(forward_batch)
for i in range(start, end):
with get_global_expert_distribution_recorder().with_current_layer(i):
layer = self.model.layers[i]
forward_batch.hidden_states, forward_batch.residual = layer(
positions,
forward_batch.hidden_states,
forward_batch,
forward_batch.residual,
forward_batch.hidden_states = layer(
positions, forward_batch.hidden_states, forward_batch
)
if end == self.model.config.num_hidden_layers:
forward_batch.hidden_states, forward_batch.residual = self.model.layers[
end - 1
].layer_communicator.finish_layer_stack(
forward_batch.hidden_states, forward_batch.residual, forward_batch
forward_batch.hidden_states = residual_batch.finish(
forward_batch.hidden_states, forward_batch
)
forward_batch.hidden_states = residual_batch.norm(
forward_batch.hidden_states, forward_batch, self.model.norm
)
if forward_batch.residual is None:
hidden_states = self.model.norm(forward_batch.hidden_states)
else:
hidden_states, _ = self.model.norm(
forward_batch.hidden_states, forward_batch.residual
)
forward_batch.residual = None
forward_batch.hidden_states = hidden_states
return self.logits_processor(
input_ids, forward_batch.hidden_states, self.lm_head, forward_batch
+31 -49
View File
@@ -11,7 +11,11 @@ from torch import nn
from transformers import PretrainedConfig
from sglang.srt.layers.activation import SiluAndMul
from sglang.srt.layers.communicator import LayerCommunicator, LayerFacts
from sglang.srt.layers.communicator import (
declare_attn,
declare_ffn,
make_stages,
)
from sglang.srt.layers.communicator.residual import batch as residual_batch
from sglang.srt.layers.dp_attention import (
is_dp_attention_enabled,
@@ -291,19 +295,17 @@ class SDARBlock(nn.Module):
prefix=add_prefix("mlp", prefix),
)
self.layer_facts = LayerFacts.init_new(
layer_id=layer_id,
num_layers=config.num_hidden_layers,
is_layer_sparse=False,
is_previous_layer_sparse=False,
is_next_layer_sparse=False,
)
self.layer_communicator = LayerCommunicator(
layer_facts=self.layer_facts,
input_layernorm=self.input_layernorm,
post_attention_layernorm=self.post_attention_layernorm,
allow_reduce_scatter=True,
allow_deferred_ffn_reduction=False,
self.attn_stage, self.ffn_stage = make_stages(
(declare_attn(), self.input_layernorm),
(
declare_ffn(
sparse=False,
next_sparse=False,
),
self.post_attention_layernorm,
),
previous=declare_ffn() if layer_id != 0 else None,
terminal=layer_id == config.num_hidden_layers - 1,
)
def forward(
@@ -311,13 +313,8 @@ class SDARBlock(nn.Module):
positions: torch.Tensor,
hidden_states: torch.Tensor,
forward_batch: ForwardBatch,
residual: Optional[torch.Tensor],
) -> Tuple[torch.Tensor, torch.Tensor]:
hidden_states, residual = self.layer_communicator.prepare_attn(
hidden_states,
residual,
forward_batch,
)
) -> torch.Tensor:
hidden_states = self.attn_stage.prepare(hidden_states, forward_batch)
if hidden_states.shape[0] != 0:
hidden_states = self.self_attn(
@@ -326,17 +323,14 @@ class SDARBlock(nn.Module):
forward_batch=forward_batch,
)
hidden_states, residual = self.layer_communicator.prepare_mlp(
hidden_states,
residual,
forward_batch,
)
hidden_states = self.attn_stage.finish(hidden_states, forward_batch)
hidden_states = self.ffn_stage.prepare(hidden_states, forward_batch)
with self.layer_communicator.ffn_exit(forward_batch) as ffn_exit:
with self.ffn_stage.exit(forward_batch) as ffn_exit:
hidden_states = self.mlp(hidden_states)
hidden_states, residual = ffn_exit.finish(hidden_states, residual)
hidden_states = ffn_exit.finish(hidden_states)
return hidden_states, residual
return hidden_states
class SDARModel(nn.Module):
@@ -404,37 +398,25 @@ class SDARModel(nn.Module):
hidden_states = (
self.embed_tokens(input_ids) if input_embeds is None else input_embeds
)
residual = residual_batch.start(forward_batch)
residual_batch.start(forward_batch)
else:
assert pp_proxy_tensors is not None
hidden_states, residual = self.layers[
self.start_layer
].layer_communicator.from_pp(
hidden_states = self.layers[self.start_layer].attn_stage.from_pp(
pp_proxy_tensors, forward_batch, allow_missing_residual=True
)
for i in range(self.start_layer, self.end_layer):
layer = self.layers[i]
hidden_states, residual = layer(
positions, hidden_states, forward_batch, residual
)
hidden_states, residual = self.layers[
self.end_layer - 1
].layer_communicator.finish_layer_stack(
hidden_states,
residual,
forward_batch,
preserve_declared=not self.pp_group.is_last_rank,
)
hidden_states = layer(positions, hidden_states, forward_batch)
if not self.pp_group.is_last_rank:
return PPProxyTensors(
{"hidden_states": hidden_states, "residual": residual}
)
return residual_batch.to_pp(hidden_states, forward_batch)
else:
hidden_states = residual_batch.finish(hidden_states, forward_batch)
if not forward_batch.forward_mode.is_idle():
hidden_states, residual = self.norm(hidden_states, residual)
hidden_states = residual_batch.norm(
hidden_states, forward_batch, self.norm
)
return hidden_states
+28 -44
View File
@@ -14,8 +14,9 @@ from sglang.srt.eplb.expert_distribution import get_global_expert_distribution_r
from sglang.srt.eplb.expert_location import ModelConfigForExpertLocation
from sglang.srt.eplb.expert_location_dispatch import ExpertLocationDispatchInfo
from sglang.srt.layers.communicator import (
LayerCommunicator,
LayerFacts,
declare_attn,
declare_ffn,
make_stages,
)
from sglang.srt.layers.communicator.residual import batch as residual_batch
from sglang.srt.layers.dp_attention import (
@@ -356,19 +357,16 @@ class SDARMoeBlock(nn.Module):
prefix=add_prefix("mlp", prefix),
)
self.layer_facts = LayerFacts.init_new(
layer_id=layer_id,
num_layers=config.num_hidden_layers,
is_layer_sparse=True,
is_previous_layer_sparse=True,
is_next_layer_sparse=True,
)
self.layer_communicator = LayerCommunicator(
layer_facts=self.layer_facts,
input_layernorm=self.input_layernorm,
post_attention_layernorm=self.post_attention_layernorm,
allow_reduce_scatter=True,
self.attn_stage, self.ffn_stage = make_stages(
(declare_attn(), self.input_layernorm),
(
declare_ffn(sparse=True, next_sparse=True),
self.post_attention_layernorm,
),
previous=declare_ffn(sparse=True, next_sparse=True)
if layer_id != 0
else None,
terminal=layer_id == config.num_hidden_layers - 1,
)
def forward(
@@ -376,12 +374,9 @@ class SDARMoeBlock(nn.Module):
positions: torch.Tensor,
hidden_states: torch.Tensor,
forward_batch: ForwardBatch,
residual: Optional[torch.Tensor],
) -> Tuple[torch.Tensor, torch.Tensor]:
) -> torch.Tensor:
hidden_states, residual = self.layer_communicator.prepare_attn(
hidden_states, residual, forward_batch
)
hidden_states = self.attn_stage.prepare(hidden_states, forward_batch)
if hidden_states.shape[0] != 0:
hidden_states = self.self_attn(
@@ -390,18 +385,17 @@ class SDARMoeBlock(nn.Module):
forward_batch=forward_batch,
)
hidden_states, residual = self.layer_communicator.prepare_mlp(
hidden_states, residual, forward_batch
)
hidden_states = self.attn_stage.finish(hidden_states, forward_batch)
hidden_states = self.ffn_stage.prepare(hidden_states, forward_batch)
with self.layer_communicator.ffn_exit(forward_batch) as ffn_exit:
with self.ffn_stage.exit(forward_batch) as ffn_exit:
hidden_states = self.mlp(
hidden_states,
forward_batch=forward_batch,
)
hidden_states, residual = ffn_exit.finish(hidden_states, residual)
hidden_states = ffn_exit.finish(hidden_states)
return hidden_states, residual
return hidden_states
class SDARMoeModel(nn.Module):
@@ -470,34 +464,24 @@ class SDARMoeModel(nn.Module):
hidden_states = (
self.embed_tokens(input_ids) if input_embeds is None else input_embeds
)
residual = residual_batch.start(forward_batch)
residual_batch.start(forward_batch)
else:
assert pp_proxy_tensors is not None
hidden_states, residual = self.layers[
self.start_layer
].layer_communicator.from_pp(pp_proxy_tensors, forward_batch)
hidden_states = self.layers[self.start_layer].attn_stage.from_pp(
pp_proxy_tensors, forward_batch
)
for i in range(self.start_layer, self.end_layer):
layer = self.layers[i]
with get_global_expert_distribution_recorder().with_current_layer(i):
hidden_states, residual = layer(
positions, hidden_states, forward_batch, residual
)
hidden_states = layer(positions, hidden_states, forward_batch)
last_layer = self.layers[self.end_layer - 1]
hidden_states, residual = last_layer.layer_communicator.finish_layer_stack(
hidden_states,
residual,
forward_batch,
preserve_declared=not self.pp_group.is_last_rank,
)
if not self.pp_group.is_last_rank:
return PPProxyTensors(
{"hidden_states": hidden_states, "residual": residual}
)
return residual_batch.to_pp(hidden_states, forward_batch)
hidden_states = residual_batch.finish(hidden_states, forward_batch)
if not forward_batch.forward_mode.is_idle():
hidden_states, residual = self.norm(hidden_states, residual)
hidden_states = residual_batch.norm(hidden_states, forward_batch, self.norm)
return hidden_states
+36 -42
View File
@@ -16,13 +16,14 @@ from sglang.srt.configs.step3_vl import (
Step3VisionEncoderConfig,
Step3VLConfig,
)
from sglang.srt.distributed import (
tensor_model_parallel_all_reduce,
)
from sglang.srt.eplb.expert_location import ModelConfigForExpertLocation
from sglang.srt.layers.activation import SiluAndMul
from sglang.srt.layers.attention.vision import VisionAttention
from sglang.srt.layers.communicator import LayerCommunicator, LayerFacts
from sglang.srt.layers.communicator import (
declare_attn,
declare_ffn,
make_stages,
)
from sglang.srt.layers.communicator.residual import batch as residual_batch
from sglang.srt.layers.conv import Conv2dLayer
from sglang.srt.layers.dp_attention import (
@@ -40,6 +41,7 @@ from sglang.srt.layers.moe import get_moe_a2a_backend
from sglang.srt.layers.moe.ep_moe.layer import get_moe_impl_class
from sglang.srt.layers.moe.fused_moe_triton import FusedMoE
from sglang.srt.layers.moe.topk import TopK
from sglang.srt.layers.moe.utils import reduce_moe_output
from sglang.srt.layers.quantization.base_config import QuantizationConfig
from sglang.srt.layers.radix_attention import RadixAttention
from sglang.srt.layers.rotary_embedding import get_rope
@@ -78,6 +80,7 @@ class Step3TextMLP(nn.Module):
hidden_act: str,
quant_config: Optional[QuantizationConfig] = None,
prefix: str = "",
reduce_results: bool = True,
) -> None:
super().__init__()
self.gate_up_proj = MergedColumnParallelLinear(
@@ -93,6 +96,7 @@ class Step3TextMLP(nn.Module):
bias=False,
quant_config=quant_config,
prefix=add_prefix("down_proj", prefix),
reduce_results=reduce_results,
)
if hidden_act != "silu":
raise ValueError(
@@ -163,8 +167,6 @@ class Step3TextMoEMLP(nn.Module):
hidden_states=hidden_states, topk_output=topk_output
)
if self.tp_size > 1:
final_hidden_states = tensor_model_parallel_all_reduce(final_hidden_states)
return final_hidden_states.view(num_tokens, hidden_dim)
@@ -339,14 +341,6 @@ class Step3TextDecoderLayer(nn.Module):
)
self.is_next_layer_sparse = True if layer_id + 1 in moe_layers_idx else False
self.layer_facts = LayerFacts.init_new(
layer_id=layer_id,
num_layers=config.num_hidden_layers,
is_layer_sparse=self.is_layer_sparse,
is_previous_layer_sparse=self.is_previous_layer_sparse,
is_next_layer_sparse=self.is_next_layer_sparse,
)
if not self.is_layer_sparse:
self.mlp = Step3TextMLP(
hidden_size=config.hidden_size,
@@ -370,6 +364,7 @@ class Step3TextDecoderLayer(nn.Module):
hidden_act="silu",
quant_config=quant_config,
prefix=add_prefix("share_expert", prefix),
reduce_results=False,
)
else:
self.moe = Step3TextMoEMLP(
@@ -379,11 +374,21 @@ class Step3TextDecoderLayer(nn.Module):
prefix=add_prefix("mlp", prefix),
)
self.layer_communicator = LayerCommunicator(
layer_facts=self.layer_facts,
input_layernorm=self.input_layernorm,
post_attention_layernorm=self.post_attention_layernorm,
allow_deferred_ffn_reduction=False,
self.attn_stage, self.ffn_stage = make_stages(
(declare_attn(), self.input_layernorm),
(
declare_ffn(
sparse=self.is_layer_sparse,
next_sparse=self.is_next_layer_sparse,
),
self.post_attention_layernorm,
),
previous=declare_ffn(
sparse=self.is_previous_layer_sparse, next_sparse=self.is_layer_sparse
)
if layer_id != 0
else None,
terminal=layer_id == config.num_hidden_layers - 1,
)
def moe_mlp_forward(self, hidden_states):
@@ -393,19 +398,16 @@ class Step3TextDecoderLayer(nn.Module):
hidden_states += self.share_expert(h)
else:
hidden_states = self.moe(hidden_states)
return hidden_states
return reduce_moe_output(hidden_states)
def forward(
self,
positions: torch.Tensor,
hidden_states: torch.Tensor,
forward_batch: ForwardBatch,
residual: Optional[torch.Tensor],
) -> Tuple[torch.Tensor, torch.Tensor]:
) -> torch.Tensor:
hidden_states, residual = self.layer_communicator.prepare_attn(
hidden_states, residual, forward_batch
)
hidden_states = self.attn_stage.prepare(hidden_states, forward_batch)
if hidden_states.shape[0] != 0:
hidden_states = self.self_attn(
@@ -414,17 +416,16 @@ class Step3TextDecoderLayer(nn.Module):
forward_batch=forward_batch,
)
hidden_states, residual = self.layer_communicator.prepare_mlp(
hidden_states, residual, forward_batch
)
with self.layer_communicator.ffn_exit(forward_batch) as ffn_exit:
hidden_states = self.attn_stage.finish(hidden_states, forward_batch)
hidden_states = self.ffn_stage.prepare(hidden_states, forward_batch)
with self.ffn_stage.exit(forward_batch) as ffn_exit:
if self.use_moe:
hidden_states = self.moe_mlp_forward(hidden_states)
else:
hidden_states = self.mlp(hidden_states)
hidden_states, residual = ffn_exit.finish(hidden_states, residual)
hidden_states = ffn_exit.finish(hidden_states)
return hidden_states, residual
return hidden_states
class Step3TextModel(nn.Module):
@@ -472,22 +473,15 @@ class Step3TextModel(nn.Module):
else:
hidden_states = input_embeds
residual = residual_batch.start(forward_batch)
residual_batch.start(forward_batch)
for i in range(len(self.layers)):
layer = self.layers[i]
hidden_states, residual = layer(
positions, hidden_states, forward_batch, residual
)
hidden_states = layer(positions, hidden_states, forward_batch)
hidden_states, residual = self.layers[-1].layer_communicator.finish_layer_stack(
hidden_states, residual, forward_batch
)
hidden_states = residual_batch.finish(hidden_states, forward_batch)
if hidden_states.shape[0] != 0:
if residual is None:
hidden_states = self.norm(hidden_states)
else:
hidden_states, _ = self.norm(hidden_states, residual)
hidden_states = residual_batch.norm(hidden_states, forward_batch, self.norm)
return hidden_states
+125
View File
@@ -0,0 +1,125 @@
"""Small fixtures around production boundaries; no parallel layout recipes."""
from types import SimpleNamespace
from sglang.srt.layers.communicator.construction import BatchVariant, StagePlan
from sglang.srt.layers.communicator.factories import (
declare_attn,
declare_ffn,
make_stages,
)
from sglang.srt.layers.communicator.residual.add_norm import PLAIN_RESIDUAL
from sglang.srt.layers.communicator.stage import StageCommunicator
def make_test_stages(
*,
attention_norm,
ffn_norm,
first=False,
last=False,
sparse=False,
previous_sparse=False,
next_sparse=False,
residual=PLAIN_RESIDUAL,
output=None,
**options,
):
previous = (
None
if first
else declare_ffn(
sparse=previous_sparse, next_sparse=sparse, update=residual.ffn_update
)
)
attention = declare_attn(
read=residual.attention_read, update=residual.attention_update
)
ffn = declare_ffn(
sparse=sparse,
next_sparse=next_sparse,
read=residual.ffn_read,
update=residual.ffn_update,
output=output,
)
common = {
key: options.pop(key)
for key in ("force_layernorm_before_dp_gather", "fusions")
if key in options
}
attn, ffn = make_stages(
(
attention,
attention_norm,
dict(
residual_in_hidden=residual.ffn_update.at_producer, **common, **options
),
),
(ffn, ffn_norm, common),
previous=previous,
terminal=last,
)
return SimpleNamespace(attn=attn, ffn=ffn)
def stub_plan():
"""Supply only defaults for tests that inject preselected paths/kernels."""
plan = StagePlan.__new__(StagePlan)
plan.norm = None
plan.fusions = None
plan._publish_lora_layout = False
plan._fusion_rows = None
plan._cp_steps = plan._sp_steps = plan._input_scattered_steps = None
plan.is_first_layer = False
plan.residual_in_hidden = False
# Keep production variant selection while tests replace individual paths.
plan._batch_steps = lambda batch: {
BatchVariant.ORDINARY: plan._steps,
BatchVariant.CONTEXT_PARALLEL: plan._cp_steps,
BatchVariant.INPUT_SCATTERED: plan._input_scattered_steps,
BatchVariant.SEQUENCE_PARALLEL: plan._sp_steps,
}[plan._variant(batch)]
return plan
def stub_stage(plan, kind):
"""Keep the same concrete boundary when a test patches its methods."""
stages = plan.__dict__.setdefault("_stub_stages", {})
if kind not in stages:
stages[kind] = StageCommunicator(plan, kind, plan.norm)
return stages[kind]
def sp_region_steps():
"""Local SP rows with no owed sum, for tests of activation and exits."""
from sglang.srt.layers.communicator import (
NORM_QUANT_READ,
NORM_READ,
BoundarySteps,
CommunicateSummableTensorPairFn,
EdgeDecl,
Layout,
StageEntry,
StageInput,
StageOutput,
TokenAxis,
make_boundary,
)
from sglang.srt.layers.communicator.ops import _hand_qkv_hook_its_input
rows = Layout(frozenset({TokenAxis.ATTN_TP_SCATTER}))
output = StageOutput(rows)
def entry(read, handoff=None):
selected = make_boundary(
EdgeDecl(output, StageInput(rows, read=read), rows, rows)
)
return StageEntry(selected.prepare, rows, handoff=handoff)
return BoundarySteps(
entry(NORM_QUANT_READ, _hand_qkv_hook_its_input),
entry(NORM_READ),
output,
CommunicateSummableTensorPairFn._trivial,
False,
)
+35 -5
View File
@@ -8,14 +8,40 @@ the name, not only into the package.
import contextlib
from unittest import mock
from sglang.srt.layers import communicator
from sglang.srt.layers.communicator import boundary, layer, layout, ops, output
from sglang.srt.layers import communicator as communicator
from sglang.srt.layers.communicator import (
boundary,
construction,
exit,
factories,
layer,
layout,
legacy_stage,
ops,
output,
)
from sglang.srt.layers.communicator import residual as residual_contract
from sglang.srt.layers.communicator.adapters import attention
from sglang.srt.layers.communicator.residual import access, add_norm, mhc
from sglang.srt.layers.communicator import (
stage,
)
from sglang.srt.layers.communicator.adapters import attention, branch, lora
from sglang.srt.layers.communicator.fusions import allreduce
from sglang.srt.layers.communicator.residual import (
access,
add_norm,
batch,
mhc,
stream,
)
COMMUNICATOR_MODULES = (
layer,
legacy_stage,
communicator,
allreduce,
exit,
branch,
lora,
layout,
output,
attention,
@@ -25,7 +51,11 @@ COMMUNICATOR_MODULES = (
mhc,
ops,
boundary,
layer,
construction,
factories,
stage,
batch,
stream,
)
@@ -57,7 +57,6 @@ def _communicator():
comm.enable_fused_ar_quant = False
comm._attn_input_fusions = comm._select_attn_input_fusions()
# Only the ordinary batches' attention input half, with these entries.
comm._context = SimpleNamespace()
comm._sp_steps = comm._input_scattered_steps = comm._cp_steps = None
comm._steps = SimpleNamespace(
attention=SimpleNamespace(
@@ -548,7 +547,7 @@ class TestDeferredLoraAllReduce(unittest.TestCase):
from types import SimpleNamespace
from unittest.mock import patch
from sglang.srt.layers.communicator import layer as exits
from sglang.srt.layers.communicator import exit as exits
from sglang.srt.layers.communicator.fusions.cutedsl import CuteDSLFusion
group = object()
@@ -91,7 +91,7 @@ class TestReduceMoeOutput(CustomTestCase):
for flags, config in (
({"fuse_mlp_allreduce": True}, {}),
({"mlp_reduce_scatter": True}, {}),
({}, {"reduce_scatterv": True}),
({"mlp_reduce_scatter": True}, {"reduce_scatterv": True}),
):
with self.subTest(flags=flags, config=config), moe_config(**config):
self.assertEqual(self.all_reduces(**flags)[0], 0)
@@ -138,14 +138,19 @@ class TestReplicatedMoeOutput(CustomTestCase):
for flags, config in (
({"fuse_mlp_allreduce": True}, {}),
({"mlp_reduce_scatter": True}, {}),
({}, {"reduce_scatterv": True}),
({"mlp_reduce_scatter": True}, {"reduce_scatterv": True}),
):
with self.subTest(flags=flags, config=config):
outputs = self.summed_over_two_ranks(flags=flags, **config)
torch.testing.assert_close(sum(outputs), torch.full((2, 3), 13.0))
def test_every_rank_adds_it_to_a_reduced_or_complete_output(self):
for config in ({}, {"backend": "flashinfer"}, {"fp4_allgather": True}):
for config in (
{},
{"backend": "flashinfer"},
{"fp4_allgather": True},
{"reduce_scatterv": True},
):
with self.subTest(**config):
outputs = self.summed_over_two_ranks(**config)
torch.testing.assert_close(outputs[1], torch.full((2, 3), 12.0))
@@ -158,6 +163,102 @@ class TestReplicatedMoeOutput(CustomTestCase):
self.assertTrue(should_add_replicated_moe_output())
class TestModelsWithExplicitDpCompletion(CustomTestCase):
def test_direct_dp_exits_publish_their_selected_sum(self):
# Execute the real orchestration methods without constructing weights.
# These models own their DP exit instead of using StageCommunicator.
import __future__
for filename, method in (
("deepseek_v4.py", "_run_moe_ffn_dp_sync"),
("qwen4_exp.py", "_run_qwen4_exp_mlp"),
):
path = MODELS_DIR / filename
node = next(
n
for n in ast.walk(ast.parse(path.read_text()))
if isinstance(n, ast.FunctionDef) and n.name == method
)
for use_rsv in (False, True):
with self.subTest(model=filename, reduce_scatterv=use_rsv):
trace = []
def rsv(value, *, output, sizes):
trace.append("RSv")
output.copy_(value[:2] * 2)
def scatter(output, value, batch):
trace.append("slice")
output.copy_(value[:2])
group = types.SimpleNamespace(reduce_scatterv=rsv)
parallel = types.SimpleNamespace(
attn_dp_size=2, attn_tp_size=1, tp_size=2, tp_group=group
)
namespace = dict(
torch=torch,
get_forward=get_forward,
get_parallel=lambda: parallel,
get_moe_a2a_backend=lambda: a2a(),
should_use_dp_reduce_scatterv=lambda: use_rsv,
is_dp_gatherv_active=lambda: False,
envs=types.SimpleNamespace(
SGLANG_DP_USE_REDUCE_SCATTER=types.SimpleNamespace(
get=lambda: False
)
),
_SHARED_EXPERT_LOCAL=False,
nullcontext=contextlib.nullcontext,
get_global_dp_buffer=lambda g: torch.empty(4, 3),
get_local_dp_buffer=lambda g: torch.empty(2, 3),
get_dp_global_num_tokens=lambda: [2, 2],
dp_gather_replicate=lambda output, value, batch: output.fill_(
1
),
dp_scatter=scatter,
)
exec(
compile(
ast.Module(body=[node], type_ignores=[]),
str(path),
"exec",
flags=__future__.annotations.compiler_flag,
),
namespace,
)
model = types.SimpleNamespace(
dsa_enable_prefill_cp=False,
config=types.SimpleNamespace(num_experts=4),
_qwen4_exp_use_dp_moe_gather=lambda: True,
_qwen4_exp_use_attn_tp_a2a_scatter=lambda: False,
mlp=lambda value, batch, **kwargs: reduce_moe_output(value),
)
batch = types.SimpleNamespace(
dp_padding_mode=types.SimpleNamespace(is_max_len=lambda: True)
)
kwargs = (
dict(input_ids=None, input_ids_global=None)
if filename == "deepseek_v4.py"
else {}
)
with (
moe_config(),
get_forward().scoped(
fuse_mlp_allreduce=False, mlp_reduce_scatter=False
),
patch(
"sglang.srt.distributed.communication_op.tensor_model_parallel_all_reduce",
side_effect=lambda value: trace.append("AR") or value * 2,
),
):
output = namespace[method](
model, torch.ones(2, 3), batch, **kwargs
)
self.assertFalse(get_forward().mlp_reduce_scatter)
self.assertEqual(trace, ["RSv"] if use_rsv else ["AR", "slice"])
torch.testing.assert_close(output, torch.full((2, 3), 2.0))
def reads_the_tp1_shared_expert_switch(tree):
return any(
isinstance(node, ast.Attribute) and node.attr == "SGLANG_SHARED_EXPERT_TP1"
@@ -1,3 +1,6 @@
from sglang.srt.layers.communicator import StageKind
from sglang.test.boundary_fixtures import stub_plan, stub_stage
"""Aux owns retained storage, including captures from reusable gather buffers."""
import unittest
@@ -7,7 +10,6 @@ from unittest.mock import Mock, patch
import torch
from sglang.srt.layers.aux_hidden_states import AuxHiddenStateList, AuxHiddenStatePacker
from sglang.srt.layers.communicator import LayerCommunicator as BoundaryPlan
from sglang.srt.layers.communicator.residual.access import norm_output
from sglang.srt.layers.communicator.residual.stream import ResidualStream
from sglang.test.ci.ci_register import register_cpu_ci
@@ -17,28 +19,6 @@ register_cpu_ci(est_time=5, suite="base-a-test-cpu")
class TestAuxStorage(CustomTestCase):
def test_constructed_layer_binds_read_only_residual_capture(self):
import test_declared_decoder_boundary as fixture
from sglang.srt.layers import communicator as comm
from sglang.srt.layers import layernorm
from sglang.test.communicator_patch import patch_communicator
parallel = fixture.parallel_of(attn_dp=1, attn_tp=2)
with (
fixture.planning(parallel),
patch_communicator("_use_aiter", False),
patch_communicator("apply_flashinfer_allreduce_fusion", return_value=True),
):
layer = comm.LayerCommunicator(
layer_facts=fixture.layer_facts(1, 3),
input_layernorm=layernorm.RMSNorm(4),
post_attention_layernorm=layernorm.RMSNorm(4),
)
keeps = layer._steps.attention.capture_preserves_residual
self.assertIsNotNone(keeps)
self.assertTrue(keeps(torch.ones(2, 4), SimpleNamespace()))
def test_breakable_graph_copies_aux_collector_into_plain_list(self):
from sglang.srt.model_executor.runner_backend.breakable_cuda_graph_backend import (
BreakableCudaGraphBackend,
@@ -111,19 +91,18 @@ class TestAuxStorage(CustomTestCase):
self.assertIs(outputs[0], value)
def boundary(self, stream, *, move=None):
boundary = BoundaryPlan.__new__(BoundaryPlan)
boundary.input_layernorm = None
boundary._context = None
boundary.prepare_attn = Mock(return_value=(stream.residual, stream))
boundary.attn = SimpleNamespace(
entry=lambda _: SimpleNamespace(
input_move=move,
capture_move=None,
capture_move_allocates=False,
capture_preserves_residual=None,
)
boundary = stub_plan()
boundary.norm = None
stub_stage(boundary, StageKind.ATTENTION)._prepare_input = Mock(
return_value=(stream.residual, stream)
)
return boundary
stub_stage(boundary, StageKind.ATTENTION).entry = lambda _: SimpleNamespace(
input_move=move,
capture_move=None,
capture_move_allocates=False,
capture_preserves_residual=None,
)
return stub_stage(boundary, StageKind.ATTENTION)
def test_packer_receives_borrowed_boundary_value_without_intermediate_clone(self):
for callback in (False, True):
@@ -140,9 +119,7 @@ class TestAuxStorage(CustomTestCase):
with patch.object(
torch.Tensor, "clone", side_effect=AssertionError("extra clone")
):
stage.prepare_attn_and_capture_last_layer_outputs(
value, stream, None, **kwargs
)
stage._prepare_attention(value, stream, None, **kwargs)
value.zero_()
torch.testing.assert_close(outputs.finalize(), torch.full((2, 3), 4.0))
@@ -153,7 +130,7 @@ class TestAuxStorage(CustomTestCase):
move = Mock(return_value=gathered)
stage = self.boundary(stream, move=move)
outputs = AuxHiddenStateList()
stage.prepare_attn_and_capture_last_layer_outputs(source, stream, None, outputs)
stage._prepare_attention(source, stream, None, outputs)
gathered.zero_()
source.zero_()
torch.testing.assert_close(outputs[0], torch.full((4, 3), 7.0))
@@ -190,11 +167,11 @@ class TestAuxStorage(CustomTestCase):
stream.write(value)
return value, stream
stage.prepare_attn = prepare
stage._prepare_input = prepare
with patch.object(
torch.Tensor, "clone", side_effect=AssertionError("extra clone")
):
stage.prepare_attn_and_capture_last_layer_outputs(
stage._prepare_attention(
hidden, stream, None, capture_output=outputs.capture
)
torch.testing.assert_close(outputs.finalize(), torch.full((2, 3), 4.0))
@@ -21,7 +21,7 @@ from sglang.srt.layers.communicator import (
StageOutput,
make_boundary,
)
from sglang.srt.layers.communicator.layer import StageCommunicator
from sglang.srt.layers.communicator.legacy_stage import StageCommunicator
from sglang.srt.layers.communicator.output import UnreducedOutput
from sglang.srt.layers.communicator.residual import batch
from sglang.srt.model_executor.forward_batch_info import (
@@ -73,7 +73,7 @@ class TestBatchOwnedResidual(CustomTestCase):
def test_pp_receive_attaches_and_export_releases_the_same_stream(self):
communicator = LayerCommunicator.__new__(LayerCommunicator)
communicator._batch_steps = lambda _: SimpleNamespace(
communicator._batch_steps = lambda fb: SimpleNamespace(
attention=SimpleNamespace(input_sum=None), ffn=None
)
fb = SimpleNamespace(residual_stream=None)
@@ -174,15 +174,17 @@ class TestBatchOwnedResidual(CustomTestCase):
class Layer:
layer_communicator = LayerCommunicator.__new__(LayerCommunicator)
def __call__(self, positions, hidden, forward_batch, residual):
def __call__(self, positions, hidden, forward_batch, *legacy_residual):
stream = batch.current(forward_batch)
self_outer.assertIs(residual, stream)
if legacy_residual:
self_outer.assertIs(legacy_residual[0], stream)
owner_ids.append(id(stream))
hidden, old = stream.finish(hidden)
stream.write(hidden if old is None else hidden + old)
return stream.leave(
result = stream.leave(
UnreducedOutput(torch.ones_like(hidden), group=group), ADD
), stream
)
return (result, stream) if legacy_residual else result
self_outer = self
fb = SimpleNamespace(
@@ -214,6 +216,43 @@ class TestBatchOwnedResidual(CustomTestCase):
self.assertEqual(group.all_reduce.call_count, 4)
torch.testing.assert_close(result, torch.full((2, 4), 9.0))
def test_mtp_returns_written_normalized_output_and_releases_batch(self):
from sglang.srt.models.nemotron_h_mtp import NemotronHMultiTokenPredictor
def terminal_layer(*, inputs_embeds, hidden_states, forward_batch):
stream = batch.current(forward_batch)
stream.write(hidden_states)
hidden_states = stream.leave(torch.ones_like(hidden_states), ADD)
hidden_states = batch.fold(hidden_states, forward_batch)
normalized = hidden_states * 3
return batch.written(normalized, forward_batch)
predictor = SimpleNamespace(pattern_len=1, layers={"0": terminal_layer})
fb = SimpleNamespace(
residual_stream=None,
spec_info=SimpleNamespace(hidden_states=torch.ones(2, 4)),
)
result = NemotronHMultiTokenPredictor.forward(
predictor, None, None, fb, inputs_embeds=torch.zeros(2, 4)
)
torch.testing.assert_close(result, torch.full((2, 4), 6.0))
self.assertIsNone(fb.residual_stream)
def test_take_output_rejects_pending_or_mismatched_outputs(self):
fb = SimpleNamespace(residual_stream=None)
stream = batch.start(fb)
value = torch.ones(2, 4)
stream.write(value)
pending = stream.leave(value * 2, ADD)
with self.assertRaises(RuntimeError):
batch.take_output(pending, fb)
self.assertIs(batch.current(fb), stream)
stream.write(value)
with self.assertRaises(RuntimeError):
batch.take_output(value.clone(), fb)
self.assertIs(batch.take_output(value, fb), value)
self.assertIsNone(fb.residual_stream)
def test_interleaved_stages_share_only_their_own_batch_stream(self):
class Read:
norms_plainly = False
@@ -236,23 +275,23 @@ class TestBatchOwnedResidual(CustomTestCase):
StageEntry(boundary.prepare, rows), None, StageOutput(rows), None, False
)
stage = StageCommunicator(
SimpleNamespace(input_layernorm=None, _context=None),
SimpleNamespace(input_layernorm=None),
"attention",
"input_layernorm",
)
a, b = [SimpleNamespace(residual_stream=None) for _ in range(2)]
sa, sb = batch.start(a), batch.start(b)
first, first_alias = stage.prepare(torch.ones(2, 4), sa, a, steps)
second, second_alias = stage.prepare(torch.full((2, 4), 10.0), sb, b, steps)
first, first_alias = stage._prepare(torch.ones(2, 4), sa, a, steps)
second, second_alias = stage._prepare(torch.full((2, 4), 10.0), sb, b, steps)
self.assertIs(first_alias, batch.current(a))
self.assertIs(second_alias, batch.current(b))
first = sa.leave(first * 3, ADD)
second = sb.leave(second * 5, ADD)
torch.testing.assert_close(
stage.prepare(first, sa, a, steps)[0], torch.full((2, 4), 14.0)
stage._prepare(first, sa, a, steps)[0], torch.full((2, 4), 14.0)
)
torch.testing.assert_close(
stage.prepare(second, sb, b, steps)[0], torch.full((2, 4), 220.0)
stage._prepare(second, sb, b, steps)[0], torch.full((2, 4), 220.0)
)
self.assertIsNot(batch.current(a), batch.current(b))
batch.start(a)
@@ -16,12 +16,6 @@ from sglang.srt.layers.communicator import (
StageOutput,
SumGroup,
TokenAxis,
)
from sglang.srt.layers.communicator import boundary as comm_boundary
from sglang.srt.layers.communicator import (
decoder_layer_edges,
decoder_layer_sides,
input_scattered_layer_sides,
make_boundary,
make_output_boundary,
)
@@ -48,104 +42,6 @@ def rows(axis_sizes, *axes):
return Layout.sharded_over(*axes, axis_sizes=axis_sizes)
class TestDecoderLayerEdges(CustomTestCase):
"""A decoder layer's three boundaries hand the residual on from one to the
next: each edge starts on the rows the one before it ends on."""
def test_each_edge_starts_where_the_one_before_ends(self):
for dp, tp, ffn_local, previous_local, last in itertools.product(
(1, 2), (1, 2), (False, True), (False, True), (False, True)
):
with self.subTest(
dp=dp, tp=tp, ffn_local=ffn_local, previous_local=previous_local
):
sides = decoder_layer_sides(
axis_sizes=sizes(dp=dp, tp=tp),
ffn_on_local_rows=ffn_local,
previous_on_local_rows=previous_local,
is_last_layer=last,
attention_gathers_local_rows=False,
ffn_group=SumGroup.TP,
leaves_for_next_layer=True,
leaves_for_reduce_scatter=True,
leaves_for_reduce_scatterv=True,
)
edges = decoder_layer_edges(sides)
self.assertEqual(edges.into_attention.residual, sides.input_rows)
self.assertEqual(
edges.into_attention.residual_to, edges.into_ffn.residual
)
self.assertEqual(edges.into_ffn.residual_to, edges.out_of_ffn.residual)
self.assertEqual(edges.out_of_ffn.residual_to, sides.output_rows)
self.assertEqual(edges.out_of_ffn.need.layout, sides.output_rows)
# Nothing owed by construction: a sum left for a batch comes
# with the value.
self.assertFalse(edges.into_attention.produced.always_leaves)
def test_an_input_owed_by_construction_is_completed_onto_the_slice(self):
axis_sizes = sizes(tp=2)
sides = input_scattered_layer_sides(
axis_sizes=axis_sizes, ffn_group=SumGroup.TP, hands_on_partial=True
)
edge = decoder_layer_edges(sides).into_attention
self.assertIs(edge.produced.group, SumGroup.TP)
self.assertTrue(edge.produced.always_leaves)
self.assertEqual(edge.residual_to, rows(axis_sizes, TokenAxis.ATTN_TP_SCATTER))
step = make_boundary(edge).prepare.keywords["step"]
self.assertIs(step.keywords["layer_input"], comm.tp_reduce_scatter)
def test_each_entry_keeps_the_move_after_its_read(self):
sides = decoder_layer_sides(
axis_sizes=sizes(dp=2, tp=2),
ffn_on_local_rows=False,
previous_on_local_rows=False,
is_last_layer=False,
attention_gathers_local_rows=False,
ffn_group=SumGroup.TP,
leaves_for_next_layer=True,
leaves_for_reduce_scatter=True,
leaves_for_reduce_scatterv=True,
)
edges = decoder_layer_edges(sides)
moves = {edges.into_attention: MagicMock(), edges.into_ffn: MagicMock()}
real = comm_boundary.make_boundary
def with_move(edge, **kwargs):
return msgspec.structs.replace(real(edge, **kwargs), input_move=moves[edge])
with patch.object(comm_boundary, "make_boundary", with_move):
steps = comm_boundary._select_boundary_steps(sides)
self.assertIs(steps.attention.input_move, moves[edges.into_attention])
self.assertIs(steps.ffn.input_move, moves[edges.into_ffn])
class TestAMoeOnEachCpShard(CustomTestCase):
"""A MoE whose data-parallel groups are the CP ranks computes each CP shard
on its own ranks: its input stays sharded over CP, and the attention output
reaches it without a gather."""
def test_the_ffn_rows_stay_sharded_over_cp(self):
axis_sizes = sizes(cp=2, tp=2)
sides = decoder_layer_sides(
axis_sizes=axis_sizes,
ffn_on_local_rows=False,
previous_on_local_rows=False,
is_last_layer=False,
attention_gathers_local_rows=False,
ffn_group=SumGroup.MOE_OUTPUT,
leaves_for_next_layer=False,
leaves_for_reduce_scatter=False,
leaves_for_reduce_scatterv=False,
ffn_shards_over_cp=True,
)
cp_rows = rows(axis_sizes, TokenAxis.ATTN_CP)
self.assertEqual(sides.ffn.input.layout, cp_rows)
self.assertEqual(sides.ffn.output.layout, cp_rows)
self.assertIs(sides.ffn.output.group, SumGroup.MOE_OUTPUT)
edges = decoder_layer_edges(sides)
self.assertEqual(edges.into_ffn.need.layout, edges.into_ffn.produced.layout)
class TestTheConsumerHalfReadsOnlyItsOwnSide(CustomTestCase):
"""Two layers build the two halves of one boundary without sharing it; the
consumer's half never depends on what the producer may leave for a batch,
@@ -276,7 +172,6 @@ class TestNonAlternatingEdges(CustomTestCase):
torch.full((3, 4), 5.0),
None,
lambda hidden, residual: (hidden + residual, hidden + residual),
None,
)
all_reduce.assert_called_once()
torch.testing.assert_close(hidden, torch.full((3, 4), 7.0))
@@ -292,7 +187,7 @@ class _WrittenIn:
def update(self, hidden_states, residual):
return 2 * hidden_states + residual
def residual_to_attn_tp_shard(self, residual, context):
def residual_to_attn_tp_shard(self, residual):
return residual
def residual_from_attn_tp_shards(self, residual):
@@ -360,7 +255,6 @@ class TestTheProducersUpdateChoosesTheOrder(CustomTestCase):
torch.full((2, 4), 3.0),
None,
norm,
None,
update=update,
)
torch.testing.assert_close(hidden, torch.full((2, 4), expected))
@@ -421,12 +315,11 @@ class TestTheProducersUpdateChoosesTheOrder(CustomTestCase):
residual,
None,
norm,
None,
)
group.all_reduce.assert_called_once()
torch.testing.assert_close(hidden, torch.full((3, 4), 7.0))
group.all_reduce.reset_mock()
hidden, _ = step(torch.full((3, 4), 2.0), residual, None, norm, None)
hidden, _ = step(torch.full((3, 4), 2.0), residual, None, norm)
group.all_reduce.assert_not_called()
torch.testing.assert_close(hidden, torch.full((3, 4), 7.0))
@@ -526,6 +419,7 @@ class TestTheConsumerRunsItsDeclaredRead(CustomTestCase):
self.assertIs(step.func, comm_ops._mlp_input_dp_partial)
hidden, residual = torch.ones(2, 4), torch.full((2, 4), 3.0)
with (
get_parallel().override(attn_tp_rank=0),
patch.object(
comm_ops,
"_reduce_and_redistribute_output_to_dp",
@@ -533,9 +427,7 @@ class TestTheConsumerRunsItsDeclaredRead(CustomTestCase):
),
patch.object(comm_ops, "dp_scatter", lambda *args: None),
):
out, out_residual = step(
hidden, residual, None, MagicMock(), SimpleNamespace(attn_tp_rank=0)
)
out, out_residual = step(hidden, residual, None, MagicMock())
self.assertEqual(read.reads, 1)
torch.testing.assert_close(out, torch.full((2, 4), 108.0))
self.assertIs(out_residual, residual)
@@ -549,15 +441,15 @@ class TestTheConsumerRunsItsDeclaredRead(CustomTestCase):
step, _ = self._ffn_input(read, residual_joins_sum=True)
self.assertIs(step.func, comm_ops._mlp_input_residual_into_sum)
hidden, residual = torch.ones(4, 4), torch.full((2, 4), 3.0)
with patch.object(
comm_ops, "tensor_model_parallel_all_reduce", lambda h: h * 2
with (
get_parallel().override(tp_size=2, tp_rank=0),
patch.object(comm_ops, "tensor_model_parallel_all_reduce", lambda h: h * 2),
):
out, out_residual = step(
hidden,
residual,
None,
MagicMock(),
SimpleNamespace(tp_size=2, tp_rank=0),
)
self.assertEqual(read.reads, 1)
expected = torch.tensor([8.0, 8.0, 2.0, 2.0])[:, None].expand(4, 4)
@@ -0,0 +1,249 @@
"""Output operations and collective fallbacks share one boundary decision."""
import unittest
from types import SimpleNamespace
from unittest.mock import patch
import test_declared_decoder_boundary as fixture
import torch
from sglang.srt.arg_groups.fields.exec_ import ExecComm
from sglang.srt.layers.communicator.output import OutputTransform
from sglang.srt.layers.moe import utils as moe_utils
from sglang.srt.runtime_context import get_forward
from sglang.test.ci.ci_register import register_cpu_ci
from sglang.test.communicator_patch import patch_communicator
register_cpu_ci(est_time=5, suite="base-a-test-cpu")
class TestBoundaryOutputPolicy(unittest.TestCase):
def batch(self, max_len=True):
return SimpleNamespace(
dp_padding_mode=SimpleNamespace(is_max_len=lambda: max_len),
forward_mode=SimpleNamespace(
is_context_parallel_extend=lambda: False, is_decode_or_idle=lambda: True
),
)
def test_policy_defaults_to_model_selection(self):
self.assertEqual(ExecComm().boundary_reduction, "auto")
def test_npu_weight_cache_belongs_to_one_prepare_call(self):
from sglang.srt.layers.communicator import ops
from sglang.srt.layers.communicator.residual.stream import ResidualStream
parallel = fixture.parallel_of(attn_dp=1, attn_tp=2)
layer = fixture.build(fixture.layer_case(0, 2), parallel)
batch = self.batch()
cache = object()
events = []
def reduce(value):
events.append("reduce")
return value * 2
def preload(value, weights):
events.append("cache")
self.assertIs(weights, cache)
torch.testing.assert_close(value, torch.full((2, 4), 2.0))
with (
fixture.planning(parallel),
patch.object(ops, "_is_npu", True),
patch.object(ops, "prepare_weight_cache", side_effect=preload, create=True),
patch_communicator("attention_tensor_model_parallel_all_reduce", reduce),
):
for weights in (cache, None):
batch.residual_stream = ResidualStream(torch.zeros(2, 4))
hidden = layer.attn.finish(torch.ones(2, 4), batch)
result = layer.ffn.prepare(hidden, batch, cache=weights)
torch.testing.assert_close(result, torch.full((2, 4), 4.0))
self.assertEqual(events, ["reduce", "cache", "reduce"])
self.assertFalse(hasattr(layer.ffn.plan, "_context"))
def test_token_slice_reads_the_current_parallel_rank(self):
from sglang.srt.layers.communicator.ops import tp_slice
hidden = torch.arange(8).reshape(4, 2)
for rank in (0, 1):
with fixture.planning(
fixture.parallel_of(attn_dp=1, attn_tp=2, tp_rank=rank)
):
value, residual = tp_slice(hidden, hidden)
torch.testing.assert_close(value, hidden.chunk(2)[rank])
torch.testing.assert_close(residual, hidden.chunk(2)[rank])
def test_cli_policy_reaches_execution_config(self):
import argparse
from sglang.srt.runtime_context import get_exec, publish, reset_context
from sglang.srt.server_args import ServerArgs
parser = argparse.ArgumentParser()
ServerArgs.add_cli_args(parser)
for policy in ("auto", "ar", "rs", "rsv", "rs+rsv"):
with self.subTest(policy=policy):
args = ["--model", "dummy", "--boundary-reduction", policy]
record = ServerArgs.from_cli_args(parser.parse_args(args))
reset_context()
try:
record.resolve_once()
publish(record, role="test")
self.assertEqual(
get_exec().comm.boundary_reduction,
"rs+rsv" if policy == "auto" else policy,
)
finally:
reset_context()
def test_step3_shared_and_routed_outputs_have_one_completion(self):
from sglang.srt.models.step3_vl import Step3TextDecoderLayer
parallel = SimpleNamespace(tp_size=2)
for fused_shared in (False, True):
for deferred in (False, True):
calls = []
layer = SimpleNamespace(
num_fused_shared_experts=int(fused_shared),
moe=lambda x: x * (0.75 if fused_shared else 0.25),
share_expert=lambda x: x * 0.5,
)
with (
patch.object(moe_utils, "get_parallel", return_value=parallel),
patch.object(
moe_utils, "post_experts_output_is_complete", return_value=False
),
patch(
"sglang.srt.distributed.communication_op.tensor_model_parallel_all_reduce",
side_effect=lambda x: calls.append("AR") or x * 2,
),
get_forward().scoped(
fuse_mlp_allreduce=deferred, mlp_reduce_scatter=False
),
):
result = Step3TextDecoderLayer.moe_mlp_forward(
layer, torch.ones(2, 4)
)
self.assertEqual(calls, [] if deferred else ["AR"])
torch.testing.assert_close(
result, torch.full((2, 4), 0.75 if deferred else 1.5)
)
def test_dp_skip_and_collective_are_selected_together(self):
import itertools
for policy, varlen, max_len, can_rs in itertools.product(
("ar", "rs", "rsv", "rs+rsv"), (False, True), (False, True), (False, True)
):
with self.subTest(
policy=policy, varlen=varlen, max_len=max_len, can_rs=can_rs
):
parallel = fixture.parallel_of(attn_dp=2, attn_tp=1)
layer = fixture.build(
fixture.layer_case(2, 3), parallel, boundary_reduction=policy
)
calls = []
def move(step, batch, value):
calls.append(step.__name__)
return value[:1]
selected = (
"_reduce_and_redistribute_output_varlen"
if varlen and policy in ("rsv", "rs+rsv")
else "_reduce_and_redistribute_output_max_len"
if max_len and can_rs and policy in ("rs", "rs+rsv")
else "_redistribute_output"
)
with (
fixture.planning(parallel),
patch_communicator("should_use_dp_reduce_scatterv", lambda: varlen),
patch_communicator("can_use_dp_reduce_scatter", lambda: can_rs),
patch_communicator("_to_local_tokens", move),
):
with layer.ffn.plan.output.ffn_exit(self.batch(max_len)) as output:
self.assertEqual(
get_forward().mlp_reduce_scatter,
selected != "_redistribute_output",
)
result, _ = output.finish(torch.ones(2, 4), torch.zeros(1, 4))
self.assertEqual(calls, [selected])
self.assertEqual(result.shape, (1, 4))
def test_moe_cannot_skip_from_topology_without_boundary_request(self):
with (
patch.object(moe_utils, "should_use_dp_reduce_scatterv", return_value=True),
patch.object(
moe_utils, "post_experts_output_is_complete", return_value=False
),
):
for skip in (False, True):
with get_forward().scoped(
fuse_mlp_allreduce=False, mlp_reduce_scatter=skip
):
self.assertEqual(
moe_utils.should_skip_post_experts_all_reduce(is_tp_path=True),
skip,
)
def test_output_transform_stays_before_transport_and_after_producer_ar(self):
# The compute stub obeys the same skip flag as RowParallelLinear.
# Its trace distinguishes AR -> multiply from multiply -> RS.
for disabled in (False, True):
with self.subTest(disabled=disabled):
calls = []
def scale(value):
calls.append("multiply")
return value * 3
parallel = fixture.parallel_of(attn_dp=2, attn_tp=1)
layer = fixture.build(
fixture.layer_case(1, 3),
parallel,
output=OutputTransform(scale, before_reduce_scatter=True),
boundary_reduction="ar" if disabled else "rs+rsv",
)
def move(step, batch, value):
if "reduce" in step.__name__:
calls.append("RS")
value = value * 2
else:
calls.append("slice")
return value[:1]
with (
fixture.planning(parallel),
patch_communicator("should_use_dp_reduce_scatterv", lambda: False),
patch_communicator("can_use_dp_reduce_scatter", lambda: True),
patch_communicator("_to_local_tokens", move),
):
with layer.ffn.plan.output.ffn_exit(self.batch()) as output:
value = torch.ones(2, 4)
if not get_forward().mlp_reduce_scatter:
calls.append("AR")
value = value * 2
value, _ = output.finish(value, torch.zeros(1, 4))
self.assertEqual(
calls,
["AR", "multiply", "slice"] if disabled else ["multiply", "RS"],
)
torch.testing.assert_close(value, torch.full((1, 4), 6.0))
def test_transform_without_pre_rs_implementation_uses_ar(self):
parallel = fixture.parallel_of(attn_dp=2, attn_tp=1)
layer = fixture.build(
fixture.layer_case(1, 3),
parallel,
output=OutputTransform(lambda x: x.square()),
)
output = layer.ffn.plan._steps.ffn_output
self.assertFalse(output.leaves_for_next_layer)
self.assertFalse(output.leaves_for_reduce_scatter)
self.assertFalse(output.leaves_for_reduce_scatterv)
if __name__ == "__main__":
unittest.main()
@@ -10,21 +10,18 @@ import torch
import sglang
from sglang.srt.layers import communicator as comm
from sglang.srt.layers.communicator import (
LayerCommunicator,
Layout,
StageOutput,
SumGroup,
TokenAxis,
UnreducedOutput,
)
from sglang.srt.layers.communicator import boundary as comm_boundary
from sglang.srt.layers.communicator import ops as comm_ops
from sglang.srt.layers.communicator import (
reduce_output,
sequence_parallel_layer_sides,
)
from sglang.srt.layers.communicator.residual import batch as residual_batch
from sglang.srt.layers.communicator.residual.access import finish_layer_stack
from sglang.srt.runtime_context import get_forward
from sglang.test.boundary_fixtures import sp_region_steps, stub_plan
from sglang.test.ci.ci_register import register_cpu_ci
from sglang.test.communicator_patch import patch_communicator
from sglang.test.test_utils import CustomTestCase
@@ -32,19 +29,6 @@ from sglang.test.test_utils import CustomTestCase
register_cpu_ci(est_time=10, suite="base-a-test-cpu")
def sp_region_steps():
"""The steps a layer runs while a LayerNorm SP region is active."""
return comm_boundary._select_boundary_steps(
sequence_parallel_layer_sides(
axis_sizes={
TokenAxis.ATTN_DP: 1,
TokenAxis.ATTN_CP: 1,
TokenAxis.ATTN_TP_SCATTER: 2,
}
)
)
def make_group(scale=3):
"""The group a deferred sum is owed over, with a stubbed all-reduce."""
return types.SimpleNamespace(all_reduce=MagicMock(side_effect=lambda h: h * scale))
@@ -95,12 +79,13 @@ def make_communicator(
reduce_scatter_step=None,
scatters_to_local_tokens=False,
group=None,
cls=LayerCommunicator,
fusions=None,
allow_deferred=True,
):
"""A communicator whose decisions and postprocess are stubbed, built
without the process-wide parallel state."""
communicator = cls.__new__(cls)
communicator = stub_plan()
communicator.fusions = fusions
communicator._steps = ordinary_steps(
StageOutput(
Layout(frozenset()),
@@ -109,36 +94,29 @@ def make_communicator(
),
returns_over_dp=scatters_to_local_tokens,
)
if cls is LayerCommunicator:
communicator._ffn_sum_moves_to_next_layer = MagicMock(return_value=fuse)
if fusions is None:
communicator.output._ffn_sum_moves_to_next_layer = MagicMock(return_value=fuse)
else:
# A subclass decides through its own fused kernels; no TP left to defer
# over otherwise.
communicator._context = types.SimpleNamespace(tp_size=1)
communicator._ffn_leaves_sum_to_reduce_scatter = MagicMock(
communicator.output._ffn_sum_moves_to_next_layer = MagicMock(return_value=False)
communicator.output._ffn_leaves_sum_to_reduce_scatter = MagicMock(
return_value=reduce_scatter
)
communicator.is_last_layer = False
communicator._sp_steps = None
communicator._input_scattered_steps = None
communicator._cp_steps = None
communicator._postprocess_dp_step = MagicMock(return_value=reduce_scatter_step)
communicator.ffn_reduction_group = MagicMock(return_value=group or make_group())
communicator._complete_ffn_output_now = MagicMock(
communicator.output._postprocess_dp_step = MagicMock(
return_value=reduce_scatter_step
)
communicator.output.ffn_reduction_group = MagicMock(
return_value=group or make_group()
)
communicator.output._complete_ffn_output_now = MagicMock(
side_effect=lambda hidden_states, residual, **_: (hidden_states + 1, residual)
)
return communicator
def finish_exit(ffn_exit, hidden, residual):
"""Inspect the selected completion form inside a batch-owned stream."""
stream = residual_batch.start(ffn_exit.forward_batch)
stream.write(residual)
hidden, output_stream = ffn_exit.finish(hidden, stream)
assert output_stream is residual_batch.current(ffn_exit.forward_batch)
return output_stream.input(hidden)
def published_flags():
forward = get_forward()
return forward.fuse_mlp_allreduce, forward.mlp_reduce_scatter
@@ -148,13 +126,13 @@ class TestFfnExit(CustomTestCase):
def setUp(self):
self.hidden_states = torch.ones(3, 4)
self.residual = torch.zeros(3, 4)
self.forward_batch = types.SimpleNamespace(residual_stream=None)
self.forward_batch = object()
def run_exit(self, communicator):
with communicator.ffn_exit(self.forward_batch) as ffn_exit:
with communicator.output.ffn_exit(self.forward_batch) as ffn_exit:
seen = published_flags()
hidden_states = self.hidden_states * 2
return seen, finish_exit(ffn_exit, hidden_states, self.residual)
return seen, ffn_exit.finish(hidden_states, self.residual)
def test_reduction_left_to_next_layer_is_unreduced(self):
communicator = make_communicator(fuse=True, reduce_scatter=False)
@@ -162,7 +140,7 @@ class TestFfnExit(CustomTestCase):
self.assertEqual(seen, (True, False))
self.assertIsInstance(hidden_states, UnreducedOutput)
self.assertIs(residual, self.residual)
communicator._complete_ffn_output_now.assert_not_called()
communicator.output._complete_ffn_output_now.assert_not_called()
def test_reduction_left_to_next_layer_declares_its_group(self):
group = make_group()
@@ -188,7 +166,7 @@ class TestFfnExit(CustomTestCase):
self.assertIs(bound.func, comm_ops._to_local_tokens)
self.assertEqual(bound.args, (step, self.forward_batch))
self.assertIs(residual, self.residual)
communicator._complete_ffn_output_now.assert_not_called()
communicator.output._complete_ffn_output_now.assert_not_called()
step.assert_not_called()
def test_a_deferred_sum_carries_the_scatter_back_under_attention_dp(self):
@@ -201,7 +179,7 @@ class TestFfnExit(CustomTestCase):
bound = hidden_states.reduce_and_redistribute
self.assertIs(bound.func, comm_ops._all_reduce_then_to_local_tokens)
self.assertEqual(bound.args, (group, self.forward_batch))
communicator._complete_ffn_output_now.assert_not_called()
communicator.output._complete_ffn_output_now.assert_not_called()
group.all_reduce.assert_not_called()
def test_postprocess_completes_other_exits(self):
@@ -213,7 +191,7 @@ class TestFfnExit(CustomTestCase):
seen, (hidden_states, residual) = self.run_exit(communicator)
self.assertEqual(seen, (False, reduce_scatter))
self.assertNotIsInstance(hidden_states, UnreducedOutput)
communicator._complete_ffn_output_now.assert_called_once()
communicator.output._complete_ffn_output_now.assert_called_once()
torch.testing.assert_close(hidden_states, self.hidden_states * 2 + 1)
self.assertIs(residual, self.residual)
@@ -224,8 +202,8 @@ class TestFfnExit(CustomTestCase):
seen, (hidden_states, residual) = self.run_exit(communicator)
self.assertEqual(seen, (False, True))
communicator._ffn_sum_moves_to_next_layer.assert_not_called()
communicator._complete_ffn_output_now.assert_called_once()
communicator.output._ffn_sum_moves_to_next_layer.assert_not_called()
communicator.output._complete_ffn_output_now.assert_called_once()
torch.testing.assert_close(hidden_states, self.hidden_states * 2 + 1)
def test_finish_applies_the_decision_the_ffn_saw(self):
@@ -234,15 +212,17 @@ class TestFfnExit(CustomTestCase):
for fuse in (True, False):
with self.subTest(fuse=fuse):
communicator = make_communicator(fuse=fuse, reduce_scatter=False)
with communicator.ffn_exit(self.forward_batch) as ffn_exit:
with communicator.output.ffn_exit(self.forward_batch) as ffn_exit:
seen = published_flags()
decide = communicator._ffn_sum_moves_to_next_layer
decide = communicator.output._ffn_sum_moves_to_next_layer
decide.return_value = not fuse
hidden_states = self.hidden_states * 2
hidden_states, _ = finish_exit(ffn_exit, hidden_states, self.residual)
hidden_states, _ = ffn_exit.finish(hidden_states, self.residual)
self.assertEqual(seen, (fuse, False))
self.assertEqual(isinstance(hidden_states, UnreducedOutput), fuse)
self.assertEqual(communicator._complete_ffn_output_now.called, not fuse)
self.assertEqual(
communicator.output._complete_ffn_output_now.called, not fuse
)
def test_flags_are_restored_after_the_ffn(self):
before = published_flags()
@@ -255,28 +235,28 @@ class TestFfnExit(CustomTestCase):
communicator = make_communicator(fuse=allows, reduce_scatter=False)
seen, _ = self.run_exit(communicator)
self.assertEqual(seen, (allows, False))
communicator._ffn_sum_moves_to_next_layer.assert_called_once()
communicator.output._ffn_sum_moves_to_next_layer.assert_called_once()
def test_deferral_implies_fusion_and_passes_the_handoff_through(self):
"""A deferring communicator publishes both flags, and a non-tensor
handoff leaves finish() untouched for the next layer's input norm."""
class Defers(LayerCommunicator):
fusions = types.SimpleNamespace(can_defer_finalize=lambda layer, fb: True)
communicator = make_communicator(fuse=False, reduce_scatter=False, cls=Defers)
fusions = types.SimpleNamespace(can_defer_finalize=lambda layer, fb: True)
communicator = make_communicator(
fuse=False, reduce_scatter=False, fusions=fusions
)
handoff = object()
with communicator.ffn_exit(self.forward_batch) as ffn_exit:
with communicator.output.ffn_exit(self.forward_batch) as ffn_exit:
seen = published_flags() + (get_forward().defer_moe_finalize,)
self.assertEqual(seen, (True, False, True))
self.assertEqual(
finish_exit(ffn_exit, handoff, self.residual), (handoff, self.residual)
ffn_exit.finish(handoff, self.residual), (handoff, self.residual)
)
communicator._complete_ffn_output_now.assert_not_called()
communicator.output._complete_ffn_output_now.assert_not_called()
self.assertFalse(get_forward().defer_moe_finalize)
# The MoE may decline per forward; a tensor then takes the fused exit.
hidden_states, _ = finish_exit(ffn_exit, self.hidden_states * 2, self.residual)
hidden_states, _ = ffn_exit.finish(self.hidden_states * 2, self.residual)
self.assertIsInstance(hidden_states, UnreducedOutput)
@@ -290,11 +270,9 @@ class TestReduceOutput(CustomTestCase):
def test_reduction_left_by_the_last_layer_runs_once(self):
communicator = self.communicator(fuse=True)
with communicator.ffn_exit(
types.SimpleNamespace(residual_stream=None)
) as ffn_exit:
with communicator.output.ffn_exit(object()) as ffn_exit:
hidden_states = torch.ones(3, 4)
hidden_states, _ = finish_exit(ffn_exit, hidden_states, torch.zeros(3, 4))
hidden_states, _ = ffn_exit.finish(hidden_states, torch.zeros(3, 4))
hidden_states = reduce_output(hidden_states)
self.all_reduce.assert_called_once()
@@ -317,11 +295,9 @@ class TestReduceOutput(CustomTestCase):
def test_complete_hidden_states_pass_through(self):
communicator = self.communicator(fuse=False)
with communicator.ffn_exit(
types.SimpleNamespace(residual_stream=None)
) as ffn_exit:
with communicator.output.ffn_exit(object()) as ffn_exit:
hidden_states = torch.ones(3, 4)
hidden_states, _ = finish_exit(ffn_exit, hidden_states, torch.zeros(3, 4))
hidden_states, _ = ffn_exit.finish(hidden_states, torch.zeros(3, 4))
self.assertIs(reduce_output(hidden_states), hidden_states)
self.assertIsNone(reduce_output(None))
@@ -333,13 +309,11 @@ class TestReduceOutput(CustomTestCase):
self.all_reduce.reset_mock()
communicator = self.communicator(fuse=fuse)
residual = torch.zeros(3, 4)
with communicator.ffn_exit(
types.SimpleNamespace(residual_stream=None)
) as ffn_exit:
with communicator.output.ffn_exit(object()) as ffn_exit:
hidden_states = torch.ones(3, 4)
hidden_states, _ = finish_exit(ffn_exit, hidden_states, residual)
hidden_states, _ = ffn_exit.finish(hidden_states, residual)
hidden_states, residual_out = communicator.finish_layer_stack(
hidden_states, residual_out = finish_layer_stack(
hidden_states, residual, object()
)
self.assertEqual(self.all_reduce.call_count, int(fuse))
@@ -355,7 +329,7 @@ class TestSelectFfnCompletion(CustomTestCase):
def communicator(
self, *, fuse=False, is_last_layer=False, scatters=True, sp_region=False
):
communicator = LayerCommunicator.__new__(LayerCommunicator)
communicator = stub_plan()
communicator.is_last_layer = is_last_layer
communicator._sp_steps = sp_region_steps() if sp_region else None
communicator._input_scattered_steps = None
@@ -370,20 +344,22 @@ class TestSelectFfnCompletion(CustomTestCase):
),
returns_over_dp=scatters,
)
communicator._ffn_sum_moves_to_next_layer = lambda forward_batch, **_: fuse
communicator._ffn_leaves_sum_to_reduce_scatter = lambda forward_batch, dp_step: (
not fuse
communicator.output._ffn_sum_moves_to_next_layer = lambda forward_batch, **_: (
fuse
)
communicator._complete_ffn_output_now = lambda h, r, **_: ("now", r)
communicator.output._ffn_leaves_sum_to_reduce_scatter = (
lambda forward_batch, dp_step: not fuse
)
communicator.output._complete_ffn_output_now = lambda h, r, **_: ("now", r)
self.group = make_group()
communicator.ffn_reduction_group = lambda forward_batch: self.group
communicator.output.ffn_reduction_group = lambda forward_batch: self.group
return communicator
def left(self, communicator, step, forward_batch=None):
with patch_communicator(
"_reduce_and_redistribute_output_step", return_value=step
):
completion = communicator._select_ffn_completion(forward_batch)
completion = communicator.output._select_ffn_completion(forward_batch)
hidden_states, _ = completion.complete(torch.ones(3, 4), None)
return None if hidden_states == "now" else hidden_states
@@ -559,7 +535,13 @@ def ffn_exit_classes(tree, source):
users = {
cls.name
for cls in classes
if ".ffn_exit(" in ast.get_source_segment(source, cls)
if ".output.ffn_exit(" in ast.get_source_segment(source, cls)
or any(
isinstance(node, ast.withitem)
and isinstance(node.optional_vars, ast.Name)
and node.optional_vars.id == "ffn_exit"
for node in ast.walk(cls)
)
}
grew = True
while grew:
@@ -584,9 +566,8 @@ class TestFfnExitUsersOweATpSum(CustomTestCase):
layers, ffn_classes, offenders = [], set(), []
for module, path in sorted(index.paths.items()):
source = path.read_text()
if (
not module.startswith("sglang.srt.models.")
or ".ffn_exit(" not in source
if not module.startswith("sglang.srt.models.") or (
".output.ffn_exit(" not in source and " as ffn_exit:" not in source
):
continue
for cls in ffn_exit_classes(index.tree(module), source):
@@ -1,3 +1,5 @@
from sglang.test.boundary_fixtures import make_test_stages
"""A MoE layer on the TP group under DSA (and MLA) prefill CP, rank by rank.
Two CP ranks build the layer's communicator for real and run prepare_mlp and
@@ -62,13 +64,6 @@ def group(name, ranks):
return SimpleNamespace(name=name, ranks=list(ranks))
def prepare_mlp(communicator, hidden, residual, forward_batch):
forward_batch.residual_stream = ResidualStream(residual)
return communicator.prepare_mlp(
hidden, forward_batch.residual_stream, forward_batch
)
class TestAttentionCpBoundary(CustomTestCase):
def setUp(self):
generator = torch.Generator().manual_seed(0)
@@ -166,27 +161,34 @@ class TestAttentionCpBoundary(CustomTestCase):
)
yield SimpleNamespace(parallel=parallel, flags=flags)
def build(self, allow_reduce_scatter):
def build(self, use_reduce_scatter):
# A MoE layer after a dense one; dense layers run on every rank here.
return comm.LayerCommunicator(
layer_facts=SimpleNamespace(
is_first_layer=False,
is_last_layer=False,
is_layer_sparse=True,
is_previous_layer_sparse=False,
is_next_layer_sparse=False,
with patch_communicator(
"get_exec",
lambda: SimpleNamespace(
comm=SimpleNamespace(
boundary_reduction="rs+rsv" if use_reduce_scatter else "ar"
),
overlap=SimpleNamespace(enable_two_batch_overlap=False),
),
input_layernorm=layernorm,
post_attention_layernorm=layernorm,
allow_reduce_scatter=allow_reduce_scatter,
)
):
return make_test_stages(
first=False,
last=False,
sparse=True,
previous_sparse=False,
next_sparse=False,
attention_norm=layernorm,
ffn_norm=layernorm,
)
def cp_extend(self):
return SimpleNamespace(
forward_mode=SimpleNamespace(is_context_parallel_extend=lambda: True)
)
def run_ranks(self, allow_reduce_scatter):
def run_ranks(self, use_reduce_scatter):
"""Gather on each rank, run an FFN that leaves a partial sum or not,
finish the exit and return what each rank got back and published."""
handed = {}
@@ -208,20 +210,16 @@ class TestAttentionCpBoundary(CustomTestCase):
with self.as_rank(
cp, dict(gather=record_gather(cp), reduce_scatter=unused)
):
prepare_mlp(
self.build(allow_reduce_scatter),
self.values[cp],
self.residuals[cp],
self.cp_extend(),
self.build(use_reduce_scatter).ffn._prepare_input(
self.values[cp], self.residuals[cp], self.cp_extend()
)
gathered, residuals = {}, {}
for cp in range(CP_SIZE):
with self.as_rank(cp, dict(gather=fill_gather, reduce_scatter=unused)):
gathered[cp], residuals[cp] = prepare_mlp(
self.build(allow_reduce_scatter),
self.values[cp],
self.residuals[cp],
self.cp_extend(),
gathered[cp], residuals[cp] = self.build(
use_reduce_scatter
).ffn._prepare_input(
self.values[cp], self.residuals[cp], self.cp_extend()
)
expected_rows = torch.cat(
[self.values[cp] + self.residuals[cp] for cp in range(CP_SIZE)]
@@ -258,17 +256,19 @@ class TestAttentionCpBoundary(CustomTestCase):
with self.as_rank(
cp, dict(gather=fill_gather, reduce_scatter=reduce_scatter)
) as rank:
communicator = self.build(allow_reduce_scatter)
batch = self.cp_extend()
batch.residual_stream = ResidualStream(residuals[cp].residual)
with communicator.ffn_exit(batch) as exit_:
communicator = self.build(use_reduce_scatter)
with communicator.ffn.plan.output.ffn_exit(
self.cp_extend()
) as exit_:
published[cp] = rank.flags.mlp_reduce_scatter
output = ffn_output(cp, leaves=published[cp])
back[cp], _ = exit_.finish(output, batch.residual_stream)
back[cp], _ = exit_.finish(
output, ResidualStream(residuals[cp].residual)
)
return published, back, reduced
def test_the_reduce_scatter_completes_the_sum_the_moe_leaves(self):
published, back, reduced = self.run_ranks(allow_reduce_scatter=True)
published, back, reduced = self.run_ranks(use_reduce_scatter=True)
self.assertEqual(published, {0: True, 1: True}, "the MoE leaves its sum")
self.assertEqual(sorted(reduced), [0, 1], "every rank joins the reduce-scatter")
for cp in range(CP_SIZE):
@@ -277,7 +277,7 @@ class TestAttentionCpBoundary(CustomTestCase):
)
def test_a_complete_output_is_only_taken_back(self):
published, back, reduced = self.run_ranks(allow_reduce_scatter=False)
published, back, reduced = self.run_ranks(use_reduce_scatter=False)
self.assertEqual(published, {0: False, 1: False}, "the MoE sums itself")
self.assertEqual(reduced, {}, "nothing is summed again")
for cp in range(CP_SIZE):
@@ -294,7 +294,7 @@ class TestAttentionCpBoundary(CustomTestCase):
0, dict(gather=unused, reduce_scatter=unused), moe_group=group("moe", [0])
):
with self.assertRaises(NotImplementedError):
self.build(allow_reduce_scatter=True)
self.build(use_reduce_scatter=True)
if __name__ == "__main__":
File diff suppressed because it is too large Load Diff
@@ -1,16 +1,13 @@
import unittest
import test_declared_decoder_boundary as fixture
from sglang.srt.layers import communicator as comm
from sglang.srt.layers.communicator import (
SumGroup,
TokenAxis,
decoder_layer_edges,
decoder_layer_sides,
make_boundary,
make_output_boundary,
)
from sglang.srt.layers.communicator import ops as comm_ops
from sglang.srt.layers.communicator.boundary import CpMoves
from sglang.test.boundary_fixtures import make_test_stages
from sglang.test.ci.ci_register import register_cpu_ci
from sglang.test.test_utils import CustomTestCase
@@ -23,33 +20,24 @@ class TestDenseMlpUnderPrefillCP(CustomTestCase):
(issue #38019: Qwen3-32B emitted garbage that never reached EOS)."""
def test_dense_mlp_gathers_across_cp(self):
sides = decoder_layer_sides(
axis_sizes={
TokenAxis.ATTN_DP: 1,
TokenAxis.ATTN_CP: 2,
TokenAxis.ATTN_TP_SCATTER: 2,
},
ffn_on_local_rows=False,
previous_on_local_rows=False,
is_last_layer=False,
attention_gathers_local_rows=False,
ffn_group=SumGroup.TP,
leaves_for_next_layer=False,
leaves_for_reduce_scatter=False,
leaves_for_reduce_scatterv=False,
parallel = fixture.parallel_of(
attn_dp=1, attn_tp=2, attn_cp=2, enable_prefill_cp=True
)
# The TP group spans every CP rank, so the MLP takes every row.
self.assertIn(TokenAxis.ATTN_CP, sides.attention.input.layout.sharded)
self.assertNotIn(TokenAxis.ATTN_CP, sides.ffn.input.layout.sharded)
moves = CpMoves(
gather=comm_ops._mlp_input_gather_moe_cp,
take_back=comm.CommunicateSummableTensorPairFn._scatter_hidden_states_moe,
with fixture.planning(parallel):
stages = make_test_stages(
attention_norm=fixture.Norm(), ffn_norm=fixture.Norm()
)
attention = stages.attn.plan._cp_steps.attention
steps = stages.ffn.plan._cp_steps
self.assertIn(TokenAxis.ATTN_CP, attention.input_rows.sharded)
self.assertNotIn(TokenAxis.ATTN_CP, steps.ffn.input_rows.sharded)
self.assertIs(
steps.ffn.prepare.keywords["step"].func, comm_ops._mlp_input_gather_moe_cp
)
self.assertIs(
steps.ffn_output_move,
comm.CommunicateSummableTensorPairFn._scatter_hidden_states_moe,
)
edges = decoder_layer_edges(sides)
step = make_boundary(edges.into_ffn, cp_moves=moves).prepare.keywords["step"]
self.assertIs(step.func, comm_ops._mlp_input_gather_moe_cp)
out = make_output_boundary(edges.out_of_ffn, cp_moves=moves)
self.assertIs(out.output_move, moves.take_back)
if __name__ == "__main__":
@@ -1,3 +1,5 @@
from sglang.test.boundary_fixtures import make_test_stages
"""The dense-FFN gather and take-back under attention DP x CP x TP, rank by rank.
Every rank builds its layer's communicator and runs its prepare_mlp and
@@ -15,7 +17,6 @@ import torch
from sglang.srt.layers import communicator as comm
from sglang.srt.layers import dp_attention, layernorm_sp
from sglang.srt.layers.communicator import ops as comm_ops
from sglang.srt.layers.communicator.residual.stream import ResidualStream
from sglang.srt.layers.cp import base as cp_base
from sglang.srt.layers.cp import padding as cp_padding
from sglang.srt.layers.cp.zigzag import ZigzagCPStrategy
@@ -126,13 +127,6 @@ def cp_replicated(forward_mode, rows, seed):
return group
def prepare_mlp(communicator, hidden, residual, forward_batch):
forward_batch.residual_stream = ResidualStream(residual)
return communicator.prepare_mlp(
hidden, forward_batch.residual_stream, forward_batch
)
class TestDpCpGather(CustomTestCase):
def run_ranks(
self,
@@ -252,16 +246,23 @@ class TestDpCpGather(CustomTestCase):
if target[0] is comm
else patch.object(*target, value)
)
communicator = comm.LayerCommunicator(
layer_facts=SimpleNamespace(
is_first_layer=False,
is_last_layer=False,
is_layer_sparse=sparse,
is_previous_layer_sparse=sparse,
is_next_layer_sparse=False,
),
input_layernorm=norm,
post_attention_layernorm=norm,
stack.enter_context(
patch_communicator(
"get_exec",
lambda: SimpleNamespace(
comm=SimpleNamespace(boundary_reduction="rs+rsv"),
overlap=SimpleNamespace(enable_two_batch_overlap=False),
),
)
)
communicator = make_test_stages(
first=False,
last=False,
sparse=sparse,
previous_sparse=sparse,
next_sparse=False,
attention_norm=norm,
ffn_norm=norm,
force_layernorm_before_dp_gather=force_layernorm_before_dp_gather,
)
yield SimpleNamespace(
@@ -281,13 +282,15 @@ class TestDpCpGather(CustomTestCase):
def gather(rank, all_reduce):
with as_rank(*rank, all_reduce) as r:
self.assertIsNotNone(r.communicator._cp_steps, "declared under CP")
self.assertIsNotNone(
r.communicator.ffn.plan._cp_steps, "declared under CP"
)
if ffn_input is not None:
steps = r.communicator._batch_steps(r.forward_batch)
steps = r.communicator.ffn.plan._batch_steps(r.forward_batch)
self.assertIs(steps.ffn.prepare.keywords["step"].func, ffn_input)
hidden_states, residual = rank_inputs(*rank)
return prepare_mlp(
r.communicator, hidden_states, residual, r.forward_batch
return r.communicator.ffn._prepare_input(
hidden_states, residual, r.forward_batch
)
# The all-reduce is a sum over every rank: record what each rank hands
@@ -327,8 +330,7 @@ class TestDpCpGather(CustomTestCase):
torch.testing.assert_close(hidden_states, expected, rtol=0, atol=0)
torch.testing.assert_close(residual.residual[:tokens], own, rtol=0, atol=0)
with as_rank(*rank, None) as r:
r.forward_batch.residual_stream = residual
back, _ = r.communicator.postprocess_layer(
back, _ = r.communicator.ffn.plan.output.postprocess_layer(
3 * hidden_states, residual, r.forward_batch
)
expected_back = torch.zeros(group.held_rows[cp], HIDDEN).double()
@@ -0,0 +1,351 @@
"""Independent stages use adjacent declarations without a decoder-layer plan."""
import itertools
import unittest
from types import SimpleNamespace
import test_declared_decoder_boundary as fixture
import torch
from sglang.srt.layers.communicator import (
EdgeDecl,
Layout,
StageDecl,
StageInput,
StageKind,
StageOutput,
declare_attn,
declare_ffn,
make_attn_stage,
make_ffn_stage,
make_stage,
make_stages,
)
from sglang.srt.layers.communicator.residual import batch
from sglang.srt.layers.communicator.residual.add_norm import ADD
from sglang.test.ci.ci_register import register_cpu_ci
from sglang.test.test_utils import CustomTestCase
register_cpu_ci(est_time=5, suite="base-a-test-cpu")
class TestIndependentStageConstruction(CustomTestCase):
def make(self, kind, *, first=False, update=ADD, read=None):
rows = Layout(frozenset())
need = StageInput(rows) if read is None else StageInput(rows, read=read)
produced = StageOutput(rows, update=update)
return make_stage(
StageDecl(need, produced),
kind=kind,
norm=fixture.Norm(),
incoming=EdgeDecl(StageOutput(rows), need, rows, rows),
outgoing=EdgeDecl(produced, StageInput(rows), rows, rows),
enters_stack=first,
terminal=True,
fixed_output=kind is StageKind.ATTENTION,
)
def test_any_two_stage_kinds_share_the_same_construction(self):
for first_kind in StageKind:
for second_kind in StageKind:
with (
self.subTest(first=first_kind, second=second_kind),
fixture.planning(fixture.parallel_of(attn_dp=1, attn_tp=1)),
):
first = self.make(first_kind, first=True)
second = self.make(second_kind)
self.assertIsNot(first.plan, second.plan)
self.assertFalse(hasattr(first.plan, "layer_facts"))
fb = SimpleNamespace()
batch.start(fb)
hidden = torch.ones(2, 4)
for stage in (first, second):
hidden = stage.prepare(hidden, fb)
if stage.kind is StageKind.ATTENTION:
hidden = stage.finish(hidden, fb)
else:
with stage.exit(fb) as output:
pass
hidden = output.finish(hidden)
torch.testing.assert_close(
batch.snapshot(hidden, fb), torch.full((2, 4), 9.0)
)
def test_binding_order_does_not_mutate_the_other_stage(self):
with fixture.planning(fixture.parallel_of(attn_dp=1, attn_tp=1)):
attn = declare_attn(terminal=True)
ffn = declare_ffn(previous=attn, terminal=True)
for reverse in (False, True):
a = lambda: make_attn_stage(
declaration=attn, norm=fixture.Norm(), following=ffn
)
f = lambda: make_ffn_stage(declaration=ffn, norm=fixture.Norm())
first, second = (f(), a()) if reverse else (a(), f())
self.assertIsNot(first.plan, second.plan)
self.assertIsNot(first.norm, second.norm)
self.assertFalse(hasattr(first.plan, "layer_facts"))
self.assertFalse(hasattr(first.plan, "previous"))
with self.assertRaises(TypeError):
declare_ffn(previous=first)
with self.assertRaisesRegex(ValueError, "following"):
make_attn_stage(declaration=attn, norm=None, following=declare_ffn())
def test_public_factories_connect_all_stage_pairs(self):
for first_kind in StageKind:
for second_kind in StageKind:
with (
self.subTest(first=first_kind, second=second_kind),
fixture.planning(fixture.parallel_of(attn_dp=1, attn_tp=1)),
):
declare = {
StageKind.ATTENTION: declare_attn,
StageKind.FFN: declare_ffn,
}
build = {
StageKind.ATTENTION: make_attn_stage,
StageKind.FFN: make_ffn_stage,
}
first = declare[first_kind]()
second = declare[second_kind](previous=first)
stages = [
build[first_kind](
declaration=first, norm=fixture.Norm(), following=second
),
build[second_kind](declaration=second, norm=fixture.Norm()),
]
fb = SimpleNamespace()
batch.start(fb)
hidden = torch.ones(2, 4)
for stage in stages:
hidden = stage.prepare(hidden, fb)
if stage.kind is StageKind.ATTENTION:
hidden = stage.finish(hidden, fb)
else:
with stage.exit(fb) as output:
pass
hidden = output.finish(hidden)
torch.testing.assert_close(
batch.snapshot(hidden, fb), torch.full((2, 4), 9.0)
)
def test_prepared_branch_reuses_the_same_placements(self):
from sglang.srt.layers.communicator.factories import (
_connect,
_connections,
_fork_input,
)
for dp in (1, 2):
for tp in (1, 2):
for local_dense in (False, True):
parallel = fixture.parallel_of(
attn_dp=dp,
attn_tp=tp,
moe_dense_tp_size=1 if local_dense else None,
)
with (
self.subTest(dp=dp, tp=tp, local_dense=local_dense),
fixture.planning(parallel),
):
attn = declare_attn(previous=declare_ffn(sparse=True))
moe = declare_ffn(previous=attn, sparse=True)
dense = declare_ffn(
prepared_from=moe,
ordinary_only=True,
)
branch_attn = declare_attn(previous=dense, ordinary_only=True)
last_dense = declare_ffn(
previous=branch_attn,
ordinary_only=True,
)
_, into_moe = _connections(attn, moe)
fork = _fork_input(into_moe, dense)
middle = _connect(dense, branch_attn)
last = _connect(branch_attn, last_dense, residual_from=middle)
for declaration, following, expected in (
(dense, branch_attn, fork),
(branch_attn, last_dense, middle),
(last_dense, None, last),
):
make = (
make_attn_stage
if declaration.kind is StageKind.ATTENTION
else make_ffn_stage
)
stage = make(
declaration=declaration,
norm=fixture.Norm(),
following=following,
)
self.assertEqual(
{
v: edge.incoming
for v, edge in stage.plan.variants.items()
},
expected.entries,
)
self.assertFalse(stage.plan.is_first_layer)
if declaration is dense:
with self.assertRaisesRegex(
RuntimeError, "branch_input"
):
stage.plan._steps.ffn.prepare(None, None, None)
with self.assertRaisesRegex(ValueError, "choose"):
declare_ffn(previous=attn, prepared_from=moe)
def test_sequences_of_one_to_four_stages(self):
for count in range(1, 5):
for kinds in itertools.product(StageKind, repeat=count):
with (
self.subTest(kinds=kinds),
fixture.planning(fixture.parallel_of(attn_dp=1, attn_tp=1)),
):
declarations = [
declare_attn() if kind is StageKind.ATTENTION else declare_ffn()
for kind in kinds
]
boundaries = make_stages(
*((d, fixture.Norm()) for d in declarations), terminal=True
)
self.assertEqual(len(boundaries), count)
fb = SimpleNamespace()
batch.start(fb)
hidden = torch.ones(2, 4)
# The direct factories provide an independent construction
# reference, including repeated kinds and terminal placement.
from dataclasses import replace
previous = None
linked = []
for i, d in enumerate(declarations):
previous = replace(
d, previous=previous, terminal=i == count - 1
)
linked.append(previous)
for i, boundary in enumerate(boundaries):
build = (
make_attn_stage
if kinds[i] is StageKind.ATTENTION
else make_ffn_stage
)
reference = build(
declaration=linked[i],
norm=boundary.norm,
following=linked[i + 1] if i + 1 < count else None,
)
self.assertEqual(
boundary.plan.variants, reference.plan.variants
)
self.assertEqual(boundary.plan.is_last_layer, i == count - 1)
self.assertIsNone(declarations[i].previous)
hidden = boundary.prepare(hidden, fb)
if boundary.kind is StageKind.ATTENTION:
hidden = boundary.finish(hidden, fb)
else:
with boundary.exit(fb) as output:
pass
hidden = output.finish(hidden)
# Norm doubles its input, each later stage adds the prior
# residual. Snapshot adds the final contribution.
torch.testing.assert_close(
batch.snapshot(hidden, fb), torch.full((2, 4), float(3**count))
)
def test_sequence_branch_keeps_source_lineage(self):
for dp, tp, local_dense in itertools.product((1, 2), (1, 2), (False, True)):
with (
self.subTest(dp=dp, tp=tp, local_dense=local_dense),
fixture.planning(
fixture.parallel_of(
attn_dp=dp,
attn_tp=tp,
moe_dense_tp_size=1 if local_dense else None,
)
),
):
_, source = make_stages(
(declare_attn(), fixture.Norm()),
(declare_ffn(sparse=True), fixture.Norm()),
previous=declare_ffn(sparse=True),
)
before = dict(source.plan.variants)
first, attn, last = make_stages(
(
declare_ffn(ordinary_only=True),
fixture.Norm(),
),
(declare_attn(ordinary_only=True), fixture.Norm()),
(
declare_ffn(ordinary_only=True),
fixture.Norm(),
),
prepared_from=source.declaration,
)
self.assertEqual(source.plan.variants, before)
self.assertIs(first.declaration.prepared_from, source.declaration)
self.assertIs(attn.declaration.previous, first.declaration)
self.assertIs(last.declaration.previous, attn.declaration)
for v, edge in first.plan.variants.items():
source_input = source.plan.variants[v].incoming
self.assertEqual(
edge.incoming.produced.layout, source_input.need.layout
)
self.assertEqual(edge.incoming.residual, source_input.residual_to)
with self.assertRaisesRegex(RuntimeError, "branch_input"):
first.plan._steps.ffn.prepare(None, None, None)
def test_mixer_successor_comes_from_the_local_sequence(self):
for following in (declare_attn(mixer_exit=True), declare_ffn()):
with (
self.subTest(kind=following.kind),
fixture.planning(fixture.parallel_of(attn_dp=1, attn_tp=2)),
):
mixer, consumer = make_stages(
(declare_attn(mixer_exit=True), fixture.Norm()),
(following, fixture.Norm()),
terminal=True,
)
self.assertIs(mixer.declaration.next_kind, following.kind)
for variant, edge in mixer.plan.variants.items():
output = edge.outgoing.produced
self.assertEqual(
output.always_leaves, following.kind is StageKind.FFN
)
self.assertEqual(
output.leaves_for_next_layer,
following.kind is StageKind.ATTENTION,
)
incoming = consumer.plan.variants[variant].incoming.produced
self.assertEqual(incoming.group, output.group)
def test_sequence_requires_one_source_and_external_terminal(self):
with self.assertRaisesRegex(ValueError, "at least one"):
make_stages()
with self.assertRaisesRegex(ValueError, "choose"):
make_stages(
(declare_attn(), None),
previous=declare_ffn(),
prepared_from=declare_ffn(),
)
with self.assertRaisesRegex(ValueError, "input sources"):
make_stages((declare_attn(previous=declare_ffn()), None))
with self.assertRaisesRegex(ValueError, "terminal"):
make_stages((declare_ffn(terminal=True), None))
def test_mismatched_declared_edge_is_rejected(self):
rows = Layout(frozenset())
declaration = StageDecl(StageInput(rows), StageOutput(rows))
wrong_input = StageInput(rows, read=object())
with self.assertRaisesRegex(ValueError, "disagrees"):
make_stage(
declaration,
kind=StageKind.FFN,
norm=None,
incoming=EdgeDecl(StageOutput(rows), wrong_input, rows, rows),
outgoing=EdgeDecl(declaration.output, declaration.input, rows, rows),
)
if __name__ == "__main__":
unittest.main()
@@ -8,17 +8,12 @@ import torch
from sglang.srt.layers import communicator as comm
from sglang.srt.layers.communicator import (
LayerCommunicator,
Layout,
StageOutput,
SumGroup,
TokenAxis,
)
from sglang.srt.layers.communicator import boundary as comm_boundary
from sglang.srt.layers.communicator import ops as comm_ops
from sglang.srt.layers.communicator import (
sequence_parallel_layer_sides,
)
from sglang.srt.layers.communicator.exit import OutputBoundary
from sglang.srt.layers.moe import (
can_merge_post_experts_all_reduce,
deferred_post_experts_all_reduce,
@@ -26,6 +21,7 @@ from sglang.srt.layers.moe import (
)
from sglang.srt.layers.moe import utils as moe_utils
from sglang.srt.runtime_context import get_forward, get_parallel
from sglang.test.boundary_fixtures import sp_region_steps, stub_plan
from sglang.test.ci.ci_register import register_cpu_ci
from sglang.test.communicator_patch import patch_communicator
from sglang.test.test_utils import CustomTestCase
@@ -33,19 +29,6 @@ from sglang.test.test_utils import CustomTestCase
register_cpu_ci(est_time=11, suite="base-a-test-cpu")
def sp_region_steps():
"""The steps a layer runs while a LayerNorm SP region is active."""
return comm_boundary._select_boundary_steps(
sequence_parallel_layer_sides(
axis_sizes={
TokenAxis.ATTN_DP: 1,
TokenAxis.ATTN_CP: 1,
TokenAxis.ATTN_TP_SCATTER: 2,
}
)
)
def _steps(*, ffn_output=None, returns_over_dp=False, ffn_sum_is_movable=True):
return comm.BoundarySteps(
attention=comm.StageEntry(
@@ -83,7 +66,7 @@ def _steps(*, ffn_output=None, returns_over_dp=False, ffn_sum_is_movable=True):
def _fake_communicator(ffn_sum_is_movable=True):
communicator = LayerCommunicator.__new__(LayerCommunicator)
communicator = stub_plan()
communicator._speculative_algo = None
# An FFN gathered over MoE-CP or on each rank's own rows has no movable sum.
communicator._steps = _steps(ffn_sum_is_movable=ffn_sum_is_movable)
@@ -91,7 +74,6 @@ def _fake_communicator(ffn_sum_is_movable=True):
communicator._input_scattered_steps = None
communicator._cp_steps = None
communicator.is_last_layer = False
communicator._context = types.SimpleNamespace(tp_size=4)
return communicator
@@ -331,9 +313,9 @@ class TestFuseMlpAllReduceGate(CustomTestCase):
tp_size=moe_ep_size * moe_tp_size * moe_dp_size,
),
):
return LayerCommunicator._ffn_sum_can_move_to_next_layer(
_fake_communicator(ffn_sum_is_movable), forward_batch
)
return OutputBoundary(
_fake_communicator(ffn_sum_is_movable)
)._ffn_sum_can_move_to_next_layer(forward_batch)
def test_hybrid_ep_tp_fuses_when_mergeable(self):
self.assertTrue(self._should_fuse(moe_ep_size=2, moe_tp_size=2))
@@ -487,7 +469,7 @@ class TestDeferFfnReduction(CustomTestCase):
return_value=tp_group_object if tp_group else object(),
),
):
return communicator._ffn_sum_moves_to_next_layer(
return communicator.output._ffn_sum_moves_to_next_layer(
forward_batch, mlp_reduce_scatter=reduce_scatter, dp_step=step
)
@@ -1,3 +1,6 @@
from sglang.srt.layers.communicator import StageKind
from sglang.test.boundary_fixtures import sp_region_steps, stub_plan, stub_stage
"""Unit tests for srt/layers/layernorm_sp (Megatron LayerNorm sequence parallelism).
Covers the pure logic that gates SP -- the Qwen3 allowlist, the config guards, and
@@ -19,14 +22,8 @@ from sglang.srt.layers.communicator import (
Layout,
StageOutput,
SumGroup,
TokenAxis,
)
from sglang.srt.layers.communicator import boundary as comm_boundary
from sglang.srt.layers.communicator import ops as comm_ops
from sglang.srt.layers.communicator import (
sequence_parallel_layer_sides,
)
from sglang.srt.layers.communicator.residual import batch as residual_batch
from sglang.srt.model_executor.forward_batch_info import ForwardMode
from sglang.srt.runtime_context import (
get_flags,
@@ -42,19 +39,6 @@ from sglang.test.test_utils import CustomTestCase
register_cpu_ci(est_time=9, suite="base-a-test-cpu")
def sp_region_steps():
"""The steps a layer runs while a LayerNorm SP region is active."""
return comm_boundary._select_boundary_steps(
sequence_parallel_layer_sides(
axis_sizes={
TokenAxis.ATTN_DP: 1,
TokenAxis.ATTN_CP: 1,
TokenAxis.ATTN_TP_SCATTER: 2,
}
)
)
def _initialize(*, enable=True, arch="Qwen3ForCausalLM"):
publish(
ServerArgs(model_path="dummy", enable_layernorm_sp=enable), role="tokenizer"
@@ -174,19 +158,15 @@ class TestSpRegionSteps(CustomTestCase):
shard: add + norm, no move, no collective."""
def communicator(self, *, first_layer):
c = comm.LayerCommunicator.__new__(comm.LayerCommunicator)
c = stub_plan()
c._sp_steps = sp_region_steps()
c._input_scattered_steps = None
c._cp_steps = None
c.layer_facts = SimpleNamespace(
is_first_layer=first_layer, is_layer_sparse=False
)
c.allow_reduce_scatter = False
c.is_first_layer = first_layer
c.is_sparse = False
c._attn_input_fusions = ()
c.input_layernorm = _Norm()
c.post_attention_layernorm = _Norm()
c.norm = _Norm()
c.qkv_latent_func = None
c._context = SimpleNamespace(cache=None)
c._steps = self.ordinary_steps(MagicMock(side_effect=AssertionError("moved")))
return c
@@ -219,17 +199,8 @@ class TestSpRegionSteps(CustomTestCase):
ffn_sum_is_movable=True,
)
def owned_input(self, hidden, residual, mode=ForwardMode.EXTEND, *, record=True):
batch = SimpleNamespace(forward_mode=mode, residual_stream=None)
stream = residual_batch.start(batch)
if residual is not None:
stream.write(residual)
if record:
hidden = stream.leave(hidden, comm.ADD)
return hidden, stream, batch
def run_prepare_attn(self, communicator, mode, hidden, residual):
hidden, residual, batch = self.owned_input(hidden, residual, mode)
batch = SimpleNamespace(forward_mode=mode)
with (
get_flags().sp.override(enabled=True),
get_forward().scoped(sp_active=False),
@@ -241,7 +212,9 @@ class TestSpRegionSteps(CustomTestCase):
layernorm_sp, "sp_entry_scatter", side_effect=lambda h: h[:1]
) as scatter,
):
out = communicator.prepare_attn(hidden, residual, batch)
out = stub_stage(communicator, StageKind.ATTENTION)._prepare_input(
hidden, residual, batch
)
return out, get_forward().sp_active, scatter
def test_the_first_layer_opens_the_region_on_prefill(self):
@@ -287,21 +260,20 @@ class TestSpRegionSteps(CustomTestCase):
"get_attn_tp_context",
return_value=SimpleNamespace(input_scattered=False),
):
h, r = communicator.prepare_attn(
*self.owned_input(hidden, residual.clone())
h, r = stub_stage(communicator, StageKind.ATTENTION)._prepare_input(
hidden,
residual.clone(),
SimpleNamespace(forward_mode=ForwardMode.EXTEND),
)
torch.testing.assert_close(h, torch.full((1, 4), 8.0))
h, r = communicator.prepare_mlp(
*self.owned_input(hidden, residual.clone(), record=False)
h, r = stub_stage(communicator, StageKind.FFN)._prepare_input(
hidden, residual.clone(), object()
)
torch.testing.assert_close(h, torch.full((1, 4), 8.0))
torch.testing.assert_close(r.residual, torch.full((1, 4), 4.0))
batch = SimpleNamespace(residual_stream=None)
stream = residual_batch.start(batch)
stream.write(residual)
out = communicator.postprocess_layer(hidden, stream, batch)
out = communicator.output.postprocess_layer(hidden, residual, object())
self.assertIs(out[0], hidden)
self.assertIs(out[1], stream)
self.assertIs(out[1], residual)
if __name__ == "__main__":
@@ -1,3 +1,5 @@
from sglang.test.boundary_fixtures import stub_plan
"""Under DP attention with reduce_scatterv, postprocess sums a FULL-mode layer's
output while scattering it back to the local tokens only when the FFN left its
all-reduce out."""
@@ -12,11 +14,7 @@ from unittest.mock import patch
import torch
from sglang.srt.layers import communicator as comm
from sglang.srt.layers.communicator import (
CommunicateSummableTensorPairFn,
LayerCommunicator,
LayerFacts,
)
from sglang.srt.layers.communicator import exit as comm_exit
from sglang.srt.layers.communicator import ops as comm_ops
from sglang.srt.layers.moe import utils as moe_utils
from sglang.srt.layers.moe.utils import (
@@ -76,7 +74,7 @@ def reduce_scatterv_applies():
yield
def postprocess_sums(*, allow_reduce_scatter, is_layer_sparse):
def postprocess_sums(*, reduce_scatter, is_layer_sparse):
calls = []
parallel = types.SimpleNamespace(
tp_size=2,
@@ -97,19 +95,19 @@ def postprocess_sums(*, allow_reduce_scatter, is_layer_sparse):
"dp_scatter", side_effect=lambda *args: calls.append("dp_scatter")
),
):
CommunicateSummableTensorPairFn._scatter_hidden_states(
torch.zeros(2, 4),
None,
step = comm_exit._reduce_and_redistribute_output_step(
forward_batch,
context=None,
allow_reduce_scatter=allow_reduce_scatter,
is_layer_sparse=is_layer_sparse,
leaves_for_reduce_scatter=reduce_scatter,
leaves_for_reduce_scatterv=reduce_scatter,
)
comm_ops._to_local_tokens(
step or comm_ops._redistribute_output, forward_batch, torch.zeros(2, 4)
)
assert len(calls) == 1, calls
return calls[0] == "reduce_scatterv"
def ffn_leaves_the_sum_out(*, allow_reduce_scatter, is_layer_sparse):
def ffn_leaves_the_sum_out(*, reduce_scatter, is_layer_sparse):
# A layer that allows reduce-scatter publishes mlp_reduce_scatter while
# reduce_scatterv applies; a dense MLP reads only that flag.
with (
@@ -133,7 +131,7 @@ def ffn_leaves_the_sum_out(*, allow_reduce_scatter, is_layer_sparse):
"should_use_flashinfer_cutlass_moe_fp4_allgather",
return_value=False,
),
get_forward().scoped(mlp_reduce_scatter=allow_reduce_scatter),
get_forward().scoped(mlp_reduce_scatter=reduce_scatter),
):
if is_layer_sparse:
return should_skip_post_experts_all_reduce(is_tp_path=True)
@@ -143,50 +141,41 @@ def ffn_leaves_the_sum_out(*, allow_reduce_scatter, is_layer_sparse):
class TestPostprocessReduceScatterv(CustomTestCase):
def test_sums_exactly_when_the_ffn_left_the_sum_out(self):
for allow, sparse in itertools.product((False, True), repeat=2):
case = dict(allow_reduce_scatter=allow, is_layer_sparse=sparse)
case = dict(reduce_scatter=allow, is_layer_sparse=sparse)
with self.subTest(**case):
self.assertEqual(
postprocess_sums(**case), ffn_leaves_the_sum_out(**case)
)
def test_a_dense_layer_without_reduce_scatter_is_not_summed_again(self):
self.assertFalse(
postprocess_sums(allow_reduce_scatter=False, is_layer_sparse=False)
)
self.assertFalse(postprocess_sums(reduce_scatter=False, is_layer_sparse=False))
def test_layer_facts_record_whether_the_layer_is_sparse(self):
for sparse in (False, True):
with self.subTest(sparse=sparse):
facts = LayerFacts.init_new(
layer_id=1,
num_layers=4,
is_layer_sparse=sparse,
is_previous_layer_sparse=sparse,
is_next_layer_sparse=sparse,
)
self.assertEqual(facts.is_layer_sparse, sparse)
def test_postprocess_only_scatters_a_complete_output(self):
# The operation-scheduled path supplies a complete output; postprocess
# only scatters it, even when the exit could choose reduce-scatter.
for step in (object(), None):
with self.subTest(reduce_scatter=step is not None):
communicator = stub_plan()
communicator._sp_steps = None
communicator._input_scattered_steps = None
communicator._cp_steps = None
communicator._steps = steps(ffn_output_move=None)
communicator.output._postprocess_dp_step = lambda forward_batch: step
with patch_communicator(
"_to_local_tokens", side_effect=lambda s, fb, h: (s, h)
):
hidden_states, residual = communicator.output.postprocess_layer(
"h", "r", "fb"
)
self.assertEqual(hidden_states, (comm_ops._redistribute_output, "h"))
self.assertEqual(residual, "r")
def test_postprocess_passes_the_layer_sparsity(self):
seen = {}
communicator = LayerCommunicator.__new__(LayerCommunicator)
communicator._sp_steps = None
communicator._input_scattered_steps = None
communicator._cp_steps = None
communicator._context = None
communicator.allow_reduce_scatter = False
communicator.layer_facts = types.SimpleNamespace(is_layer_sparse=True)
communicator._steps = steps(
ffn_output_move=lambda **kwargs: seen.update(kwargs) or (None, None)
)
communicator.postprocess_layer(None, None, None)
self.assertIs(seen["is_layer_sparse"], True)
def test_postprocess_takes_the_output_back_with_the_exit_step(self):
def test_legacy_postprocess_takes_the_output_back_with_the_exit_step(self):
# Under attention DP the base postprocess brings the FFN output back with
# the step the FFN exit chooses.
for step in (object(), None):
with self.subTest(reduce_scatter=step is not None):
communicator = LayerCommunicator.__new__(LayerCommunicator)
communicator = comm.LayerCommunicator.__new__(comm.LayerCommunicator)
communicator._sp_steps = None
communicator._input_scattered_steps = None
communicator._cp_steps = None
@@ -216,18 +205,18 @@ class TestReduceAndRedistributeOutputStep(CustomTestCase):
patch_communicator("should_use_dp_reduce_scatterv", return_value=varlen),
patch_communicator("can_use_dp_reduce_scatter", return_value=tiles),
):
# A MoE block leaves its sum to reduce_scatterv whenever it applies.
return comm_ops._reduce_and_redistribute_output_step(
# Both dense and MoE producers follow the selected output step.
return comm_exit._reduce_and_redistribute_output_step(
forward_batch,
leaves_for_reduce_scatter=allow,
leaves_for_reduce_scatterv=allow or sparse,
leaves_for_reduce_scatterv=allow,
)
def test_every_condition(self):
for varlen, max_len, tiles, allow, sparse in itertools.product(
(False, True), repeat=5
):
if varlen and (allow or sparse):
if varlen and allow:
expected = comm_ops._reduce_and_redistribute_output_varlen
elif allow and max_len and tiles:
expected = comm_ops._reduce_and_redistribute_output_max_len
@@ -1,3 +1,6 @@
from sglang.srt.layers.communicator import StageKind
from sglang.test.boundary_fixtures import stub_plan, stub_stage
"""prepare_attn completes a sum the previous layer left, then adds the residual
and applies the input norm in the form the attention's quant format wants."""
@@ -12,6 +15,7 @@ import torch
from sglang.srt.layers import communicator as comm
from sglang.srt.layers.communicator import ops as comm_ops
from sglang.srt.layers.communicator.fusions.allreduce import attention_fusions
from sglang.test.ci.ci_register import register_cpu_ci
from sglang.test.communicator_patch import patch_communicator
from sglang.test.test_utils import CustomTestCase
@@ -44,19 +48,18 @@ class Norm:
def communicator(norm):
c = comm.LayerCommunicator.__new__(comm.LayerCommunicator)
c.input_layernorm = norm
c = stub_plan()
c.norm = norm
# A layer inside the stack: an absent residual was written back before.
c.layer_facts = SimpleNamespace(is_first_layer=False)
c.is_first_layer = False
c._sp_steps = None
c._input_scattered_steps = None
c._cp_steps = None
c.qkv_latent_func = None
c._context = None
c.enable_fused_ar_quant = False
c.fused_ar_quant_keep_bf16 = False
# Construction picks the fused entries; call it under the platform patches.
c._attn_input_fusions = c._select_attn_input_fusions()
c._attn_input_fusions = attention_fusions(c)
# A layer whose attention takes its input as it is and owes nothing on it.
c._steps = comm.BoundarySteps(
attention=comm.StageEntry(
@@ -132,16 +135,6 @@ def platform(*, use_aiter=False, gfx95=False, fusion=False, kernel_group=True):
yield kernels, all_reduce, group
def prepare_attn(communicator, hidden, residual, forward_batch=None, **kwargs):
from sglang.srt.layers.communicator.residual.add_norm import ADD
from sglang.srt.layers.communicator.residual.stream import ResidualStream
batch = forward_batch or SimpleNamespace()
batch.residual_stream = ResidualStream(residual)
hidden = batch.residual_stream.leave(hidden, ADD)
return communicator.prepare_attn(hidden, batch.residual_stream, batch, **kwargs)
class TestPrepareAttnSteps(CustomTestCase):
def test_each_quant_format_uses_its_kernel(self):
for quant_format, flags, kernel in (
@@ -165,12 +158,8 @@ class TestPrepareAttnSteps(CustomTestCase):
platform(**flags) as (kernels, _, _),
):
norm = Norm()
prepare_attn(
communicator(norm),
torch.ones(2, 4),
residual,
None,
quant_format=quant_format,
stub_stage(communicator(norm), StageKind.ATTENTION)._prepare_input(
torch.ones(2, 4), residual, None, quant_format=quant_format
)
for name, mock in kernels.items():
self.assertEqual(mock.called, name == kernel, name)
@@ -185,13 +174,9 @@ class TestPrepareAttnSteps(CustomTestCase):
):
with self.subTest(quant_format=quant_format), platform(**flags):
hidden_states = torch.ones(2, 4)
_, residual = prepare_attn(
communicator(Norm()),
hidden_states,
None,
None,
quant_format=quant_format,
)
_, residual = stub_stage(
communicator(Norm()), StageKind.ATTENTION
)._prepare_input(hidden_states, None, None, quant_format=quant_format)
self.assertIs(residual.residual, hidden_states)
def test_a_pending_sum_is_completed_once(self):
@@ -202,7 +187,9 @@ class TestPrepareAttnSteps(CustomTestCase):
):
norm = Norm()
hidden_states = comm.UnreducedOutput(torch.ones(2, 4), group=group)
prepare_attn(communicator(norm), hidden_states, torch.zeros(2, 4), None)
stub_stage(communicator(norm), StageKind.ATTENTION)._prepare_input(
hidden_states, torch.zeros(2, 4), None
)
self.assertEqual(all_reduce.call_count, 0 if fusion else 1)
self.assertEqual(
norm.calls, ["fused_all_reduce_norm"] if fusion else ["norm"]
@@ -211,8 +198,7 @@ class TestPrepareAttnSteps(CustomTestCase):
def test_a_pending_sum_keeps_the_quant_format_without_fusion(self):
with platform(use_aiter=True) as (kernels, all_reduce, group):
norm = Norm()
prepare_attn(
communicator(norm),
stub_stage(communicator(norm), StageKind.ATTENTION)._prepare_input(
comm.UnreducedOutput(torch.ones(2, 4), group=group),
torch.zeros(2, 4),
None,
@@ -225,8 +211,7 @@ class TestPrepareAttnSteps(CustomTestCase):
def test_a_pending_sum_over_another_group_skips_the_fused_kernel(self):
with platform(fusion=True, kernel_group=False) as (_, all_reduce, group):
norm = Norm()
prepare_attn(
communicator(norm),
stub_stage(communicator(norm), StageKind.ATTENTION)._prepare_input(
comm.UnreducedOutput(torch.ones(2, 4), group=group),
torch.zeros(2, 4),
None,
@@ -237,8 +222,7 @@ class TestPrepareAttnSteps(CustomTestCase):
def test_a_post_residual_addition_bypasses_the_fused_kernel(self):
with platform(fusion=True) as (_, all_reduce, group):
norm = Norm()
prepare_attn(
communicator(norm),
stub_stage(communicator(norm), StageKind.ATTENTION)._prepare_input(
comm.UnreducedOutput(torch.ones(2, 4), group=group),
torch.zeros(2, 4),
None,
@@ -257,8 +241,9 @@ class TestPrepareAttnSteps(CustomTestCase):
partial = torch.ones(3, 4)
local = torch.full((local_rows, 4), 7.0)
step = MagicMock(return_value=local)
hidden_states, residual = prepare_attn(
communicator(norm),
hidden_states, residual = stub_stage(
communicator(norm), StageKind.ATTENTION
)._prepare_input(
comm.UnreducedOutput(partial, reduce_and_redistribute=step),
torch.zeros(local_rows, 4),
None,
@@ -296,8 +281,7 @@ class TestPrepareAttnSteps(CustomTestCase):
)
partial_sum = torch.ones(3, 4)
step = MagicMock(return_value=torch.full((1, 4), 7.0))
prepare_attn(
c,
stub_stage(c, StageKind.ATTENTION)._prepare_input(
comm.UnreducedOutput(partial_sum, reduce_and_redistribute=step),
torch.zeros(1, 4),
None,
@@ -314,8 +298,7 @@ class TestPrepareAttnSteps(CustomTestCase):
):
norm = Norm()
with self.assertRaises(RuntimeError):
prepare_attn(
communicator(norm),
stub_stage(communicator(norm), StageKind.ATTENTION)._prepare_input(
comm.UnreducedOutput(
torch.ones(2, 4), reduce_and_redistribute=step
),
@@ -1,4 +1,5 @@
import unittest
import weakref
from types import SimpleNamespace
from unittest.mock import Mock, patch
@@ -8,7 +9,6 @@ from sglang.srt.layers.aux_hidden_states import AuxHiddenStateList
from sglang.srt.layers.communicator import (
BoundarySteps,
EdgeDecl,
LayerCommunicator,
Layout,
StageEntry,
StageInput,
@@ -16,7 +16,8 @@ from sglang.srt.layers.communicator import (
SumGroup,
make_boundary,
)
from sglang.srt.layers.communicator.layer import StageCommunicator
from sglang.srt.layers.communicator.boundary import StageKind
from sglang.srt.layers.communicator.construction import StageCommunicator
from sglang.srt.layers.communicator.output import UnreducedOutput
from sglang.srt.layers.communicator.residual.access import add_to_output
from sglang.srt.layers.communicator.residual.add_norm import ADD
@@ -25,6 +26,7 @@ from sglang.srt.layers.communicator.residual.stream import (
OwedOutput,
ResidualStream,
)
from sglang.test.boundary_fixtures import stub_plan, stub_stage
from sglang.test.ci.ci_register import register_cpu_ci
from sglang.test.test_utils import CustomTestCase
@@ -63,16 +65,15 @@ class TestResidualStream(CustomTestCase):
StageEntry(boundary.prepare, rows), None, StageOutput(rows), None, False
)
stage = StageCommunicator(
SimpleNamespace(input_layernorm=None, _context=None),
"attention",
"input_layernorm",
SimpleNamespace(norm=None),
StageKind.ATTENTION,
None,
)
value = torch.ones(2, 4)
batch = SimpleNamespace(residual_stream=ResidualStream())
hidden, stream = stage.prepare(value, batch.residual_stream, batch, steps)
hidden, stream = stage._prepare(value, ResidualStream(), None, steps)
self.assertEqual(read.enters, 1)
torch.testing.assert_close(hidden, value * 6)
hidden, stream = stage.prepare(stream.residual, stream, batch, steps)
hidden, stream = stage._prepare(stream.residual, stream, None, steps)
self.assertEqual(read.enters, 1)
torch.testing.assert_close(hidden, value * 6)
@@ -110,17 +111,16 @@ class TestResidualStream(CustomTestCase):
StageEntry(boundary.prepare, rows), None, StageOutput(rows), None, False
)
stage = StageCommunicator(
SimpleNamespace(input_layernorm=None, _context=None),
"attention",
"input_layernorm",
SimpleNamespace(norm=None),
StageKind.ATTENTION,
None,
)
for increment in (2, 7):
with self.subTest(increment=increment):
actual = Update(increment)
stream = ResidualStream(torch.full((2, 4), 3.0))
hidden = stream.leave(torch.ones(2, 4), actual)
batch = SimpleNamespace(residual_stream=stream)
result, stream = stage.prepare(hidden, stream, batch, steps)
result, stream = stage._prepare(hidden, stream, None, steps)
torch.testing.assert_close(
result, torch.full((2, 4), 2.0 * (4 + increment))
)
@@ -139,7 +139,6 @@ class TestResidualStream(CustomTestCase):
self.residual,
None,
None,
None,
update=SimpleNamespace(adds_plainly=False),
)
@@ -189,7 +188,6 @@ class TestResidualStream(CustomTestCase):
self.residual,
None,
norm,
None,
pending=stream.pending,
update=stream.pending.update,
)
@@ -209,7 +207,6 @@ class TestResidualStream(CustomTestCase):
self.residual,
None,
None,
None,
pending=stream.pending,
update=ADD,
)
@@ -241,7 +238,7 @@ class TestResidualStream(CustomTestCase):
self.stream.input(hidden.clone())
def test_capture_and_compute_share_the_fused_read(self):
boundary = LayerCommunicator.__new__(LayerCommunicator)
boundary = stub_plan()
events = []
outputs = AuxHiddenStateList()
@@ -257,27 +254,28 @@ class TestResidualStream(CustomTestCase):
events.append("capture")
outputs.capture(value, owned=owned)
boundary.prepare_attn = Mock(side_effect=prepare)
boundary.attn = SimpleNamespace(
entry=lambda batch: SimpleNamespace(
capture_move=None,
capture_move_allocates=False,
capture_preserves_residual=None,
)
boundary.norm = None
stub_stage(boundary, StageKind.ATTENTION)._prepare_input = Mock(
side_effect=prepare
)
output, stream = boundary.prepare_attn_and_capture_last_layer_outputs(
stub_stage(boundary, StageKind.ATTENTION).entry = lambda batch: SimpleNamespace(
capture_move=None,
capture_move_allocates=False,
capture_preserves_residual=None,
)
output, stream = stub_stage(boundary, StageKind.ATTENTION)._prepare_attention(
self.hidden, self.stream, None, capture_output=capture
)
self.assertEqual(events, ["add_norm", "capture"])
self.group.all_reduce.assert_called_once()
boundary.prepare_attn.assert_called_once()
stub_stage(boundary, StageKind.ATTENTION)._prepare_input.assert_called_once()
torch.testing.assert_close(output, torch.full((2, 4), 15.0))
torch.testing.assert_close(outputs[0], torch.full((2, 4), 5.0))
stream.residual.zero_()
torch.testing.assert_close(outputs[0], torch.full((2, 4), 5.0))
def test_deepstack_capture_precedes_the_extra_addition(self):
boundary = LayerCommunicator.__new__(LayerCommunicator)
boundary = stub_plan()
extra = torch.full_like(self.partial, 7.0)
outputs = AuxHiddenStateList()
@@ -288,15 +286,16 @@ class TestResidualStream(CustomTestCase):
stream.write(updated)
return updated * 3, stream
boundary.prepare_attn = Mock(side_effect=prepare)
boundary.attn = SimpleNamespace(
entry=lambda batch: SimpleNamespace(
capture_move=None,
capture_move_allocates=False,
capture_preserves_residual=None,
)
boundary.norm = None
stub_stage(boundary, StageKind.ATTENTION)._prepare_input = Mock(
side_effect=prepare
)
output, _ = boundary.prepare_attn_and_capture_last_layer_outputs(
stub_stage(boundary, StageKind.ATTENTION).entry = lambda batch: SimpleNamespace(
capture_move=None,
capture_move_allocates=False,
capture_preserves_residual=None,
)
output, _ = stub_stage(boundary, StageKind.ATTENTION)._prepare_attention(
self.hidden,
self.stream,
None,
@@ -376,5 +375,81 @@ class TestResidualStream(CustomTestCase):
move.assert_called_once_with(self.partial)
class TestBatchStageOwnership(CustomTestCase):
def test_terminal_norm_releases_layer_buffers(self):
from sglang.srt.layers.communicator.residual import batch
fb = SimpleNamespace(residual_stream=None)
batch.start(fb)
residual = torch.ones(2, 4)
contribution = torch.full_like(residual, 2)
refs = [weakref.ref(residual), weakref.ref(contribution)]
batch.current(fb).write(residual)
output = batch.current(fb).leave(contribution, ADD)
result = batch.norm(output, fb, lambda x, r: (x + r, r))
del output, residual, contribution
torch.testing.assert_close(result, torch.full((2, 4), 3.0))
self.assertTrue(all(ref() is None for ref in refs))
self.assertIsNone(fb.residual_stream)
def test_pp_export_transfers_buffer_ownership(self):
from sglang.srt.layers.communicator.residual import batch
fb = SimpleNamespace(residual_stream=None)
batch.start(fb)
residual = torch.ones(2, 4)
contribution = torch.full_like(residual, 2)
refs = [weakref.ref(residual), weakref.ref(contribution)]
batch.current(fb).write(residual)
output = batch.current(fb).leave(contribution, ADD)
proxy = batch.to_pp(output, fb)
del output, residual, contribution
self.assertTrue(all(ref() is not None for ref in refs))
self.assertIsNone(fb.residual_stream)
del proxy
self.assertTrue(all(ref() is None for ref in refs))
def test_interleaved_batches_keep_independent_streams(self):
from sglang.srt.layers.communicator.residual import batch
class Read:
norms_plainly = False
def enter(self, value):
return value
def read(self, value, norm, quant_format="", **kwargs):
return value * 2, value
def update_and_read(self, update, value, residual, norm, **kwargs):
updated = update.update(value, residual)
return updated * 2, updated
rows = Layout(frozenset())
boundary = make_boundary(
EdgeDecl(StageOutput(rows), StageInput(rows, read=Read()), rows, rows)
)
owner = stub_plan()
owner.norm = None
owner._steps = BoundarySteps(
None, StageEntry(boundary.prepare, rows), StageOutput(rows), None, False
)
owner._sp_steps = owner._input_scattered_steps = owner._cp_steps = None
stage = stub_stage(owner, StageKind.FFN)
a, b = SimpleNamespace(), SimpleNamespace()
batch.start(a)
batch.start(b)
first = stage.prepare(torch.ones(2, 4), a)
second = stage.prepare(torch.full((2, 4), 10.0), b)
first = batch.current(a).leave(first * 3, ADD)
second = batch.current(b).leave(second * 5, ADD)
torch.testing.assert_close(stage.prepare(first, a), torch.full((2, 4), 14.0))
torch.testing.assert_close(stage.prepare(second, b), torch.full((2, 4), 220.0))
self.assertIsNot(batch.current(a), batch.current(b))
batch.start(a)
self.assertIsNone(batch.current(a).pending)
self.assertIsNotNone(batch.current(b).residual)
if __name__ == "__main__":
unittest.main()
@@ -1,3 +1,6 @@
from sglang.srt.layers.communicator import StageKind
from sglang.test.boundary_fixtures import stub_plan, stub_stage
"""CPU-only metadata tests for explicit DP-attention LoRA routing."""
import sys
@@ -12,12 +15,11 @@ import torch
from sglang.srt.layers.communicator import (
ADD,
NORM_QUANT_READ,
LayerCommunicator,
Layout,
TokenAxis,
)
from sglang.srt.layers.communicator.boundary import StageEntry
from sglang.srt.layers.communicator.ops import _consumer_step, _read_input
from sglang.srt.layers.communicator.residual import batch as residual_batch
from sglang.srt.layers.dp_attention import DpPaddingMode
from sglang.srt.lora.backend.base_backend import BaseLoRABackend
from sglang.srt.lora.backend.triton_backend import (
@@ -229,13 +231,11 @@ def test_communicator_publishes_layout_at_each_transition(
if not publish_lora_layout:
# Without LoRA under DP attention, the communicator leaves the flag alone.
expected_mlp = expected_attn = initial
communicator = LayerCommunicator.__new__(LayerCommunicator)
communicator = stub_plan()
communicator._publish_lora_layout = publish_lora_layout
communicator.layer_facts = SimpleNamespace(is_first_layer=False)
communicator._context = SimpleNamespace()
communicator.is_first_layer = False
communicator._sp_steps = None
communicator.post_attention_layernorm = None
communicator.input_layernorm = lambda x: x
communicator.norm = lambda x: x
communicator.qkv_latent_func = None
gathered, local = Layout(frozenset()), Layout(frozenset({TokenAxis.ATTN_DP}))
# The rows of the steps the batch runs decide, not the ordinary steps'.
@@ -243,8 +243,8 @@ def test_communicator_publishes_layout_at_each_transition(
ffn=SimpleNamespace(input_rows=local if gathered_over_dp else gathered)
)
selected = SimpleNamespace(
attention=SimpleNamespace(
input_sum=None,
attention=StageEntry(
input_rows=local,
prepare=partial(
_consumer_step,
step=partial(
@@ -260,8 +260,7 @@ def test_communicator_publishes_layout_at_each_transition(
input_move=lambda hidden_states, **kwargs: hidden_states,
handoff=lambda hidden_states, *args: hidden_states,
),
ffn=SimpleNamespace(
input_sum=None,
ffn=StageEntry(
prepare=lambda hidden_states, residual, *args, **kwargs: (
hidden_states,
residual,
@@ -275,13 +274,11 @@ def test_communicator_publishes_layout_at_each_transition(
hidden = torch.zeros(num_tokens, 4)
with get_forward().scoped(lora_batch_layout=initial):
for _ in range(2):
batch = SimpleNamespace(residual_stream=None)
stream = residual_batch.start(batch)
stream.write(hidden)
communicator.prepare_mlp(hidden, stream, batch)
stub_stage(communicator, StageKind.FFN)._prepare_input(hidden, hidden, None)
assert get_forward().lora_batch_layout is expected_mlp
stream = residual_batch.start(batch)
communicator.prepare_attn(hidden, stream, batch)
stub_stage(communicator, StageKind.ATTENTION)._prepare_input(
hidden, None, None
)
assert get_forward().lora_batch_layout is expected_attn
@@ -8,6 +8,7 @@ from torch import nn
from sglang.srt.layers import communicator as comm
from sglang.srt.layers.communicator import ops as comm_ops
from sglang.srt.layers.communicator.residual.stream import ResidualStream
from sglang.srt.model_executor.forward_batch_info import ForwardMode, PPProxyTensors
from sglang.srt.models.bailing_moe import BailingMoEModel
from sglang.srt.models.bailing_moe_v3 import BailingMoELinearModel
@@ -16,6 +17,9 @@ from sglang.srt.models.glm4_moe_lite import Glm4MoeLiteModel
from sglang.srt.models.glm5_next import Glm5NextModel
from sglang.srt.models.gpt_oss import GptOssModel
from sglang.srt.models.laguna import LagunaModel
from sglang.srt.models.llama4 import Llama4Model
from sglang.srt.models.qwen3 import Qwen3Model
from sglang.srt.models.qwen3_vl import Qwen3LLMModel
from sglang.srt.models.qwen3_vl_moe import Qwen3MoeLLMModel
from sglang.test.ci.ci_register import register_cpu_ci
from sglang.test.test_utils import CustomTestCase
@@ -33,6 +37,7 @@ MODELS = (
LagunaModel,
Qwen3MoeLLMModel,
)
CAPTURE_MODELS = (*MODELS, Llama4Model, Qwen3Model, Qwen3LLMModel)
def all_reduce(hidden_states):
@@ -44,10 +49,13 @@ GROUP = SimpleNamespace(all_reduce=all_reduce)
class DeferringLayer(nn.Module):
def __init__(self, defer, return_topk=False):
def __init__(self, defer, return_topk=False, stage_api=False):
super().__init__()
self.return_topk = return_topk
self.stage_api = stage_api
self.layer_communicator = comm.LayerCommunicator.__new__(comm.LayerCommunicator)
self.layer_communicator.input_layernorm = None
self.attn_stage = self.layer_communicator.attn
self.layer_communicator._steps = comm.BoundarySteps(
attention=comm.StageEntry(
prepare=partial(
@@ -106,21 +114,27 @@ class DeferringLayer(nn.Module):
**kwargs,
):
stream = forward_batch.residual_stream
assert residual is stream
hidden_states, residual = stream.input(hidden_states)
if stream is not None:
residual = stream
if isinstance(residual, ResidualStream):
hidden_states, residual = residual.input(hidden_states)
hidden_states = comm.reduce_output(hidden_states)
if residual is None:
residual = hidden_states.clone()
else:
# later norms mutate the residual; captured snapshots must stay intact
residual.add_(hidden_states)
stream.write(residual)
capture = kwargs.get("capture_output")
if capture is not None:
capture(residual.clone())
with self.layer_communicator.ffn_exit(forward_batch) as ffn_exit:
partial = torch.full_like(hidden_states, 0.5)
hidden_states, residual = ffn_exit.finish(partial, stream)
if stream is not None:
stream.write(residual)
residual = stream
hidden_states, residual = ffn_exit.finish(partial, residual)
if self.stage_api:
return (hidden_states, None) if self.return_topk else hidden_states
if self.return_topk:
return hidden_states, residual, None
return hidden_states, residual
@@ -151,7 +165,17 @@ def build_model(model_cls, *, defer, capture):
model.layers_to_capture = [0, 1, 2] if capture else []
model.layers = nn.ModuleList(
DeferringLayer(
defer and i < NUM_LAYERS - 1, return_topk=model_cls is Glm5NextModel
defer and i < NUM_LAYERS - 1,
return_topk=model_cls is Glm5NextModel,
stage_api=model_cls
in (
BailingMoEModel,
BailingMoELinearModel,
GptOssModel,
Llama4Model,
Qwen3Model,
Qwen3LLMModel,
),
)
for i in range(NUM_LAYERS)
)
@@ -184,7 +208,7 @@ class TestAuxCaptureDeferredAllreduce(CustomTestCase):
forward_mode=ForwardMode.DECODE,
capture_hidden_mode=SimpleNamespace(need_capture=lambda: True),
)
for model_cls in MODELS:
for model_cls in CAPTURE_MODELS:
for defer in (False, True):
for capture in (False, True):
with (
@@ -1,3 +1,5 @@
from sglang.srt.layers.communicator.residual.add_norm import ADD
"""Deepstack visual embeddings are added to a decoder layer's output. When the
layer left its FFN all-reduce to the next layer, that output is one rank's
partial sum, and an embedding added to it is counted once per rank."""
@@ -9,7 +11,7 @@ import torch
from torch import nn
from sglang.srt.layers.communicator import LayerCommunicator, UnreducedOutput
from sglang.srt.layers.communicator.residual.add_norm import ADD
from sglang.srt.layers.communicator.residual import batch as residual_batch
from sglang.test.ci.ci_register import register_cpu_ci
from sglang.test.test_utils import CustomTestCase
@@ -56,6 +58,20 @@ class DeferringLayer(nn.Module):
return stream.leave(output, ADD), stream
class DeferringStageLayer(DeferringLayer):
def forward(self, positions=None, hidden_states=None, forward_batch=None, **kwargs):
stream = residual_batch.current(forward_batch)
hidden_states, residual = stream.finish(hidden_states)
residual = hidden_states if residual is None else hidden_states + residual
stream.write(residual)
output = (
torch.ones_like(residual)
if self.is_last_layer
else UnreducedOutput(torch.full_like(residual, 1 / TP_SIZE), group=GROUP)
)
return stream.leave(output, ADD)
class SumNorm(nn.Module):
def forward(self, hidden_states, residual=None, post_residual_addition=None):
if residual is None:
@@ -158,9 +174,14 @@ class TestSplitPrefillCompletion(CustomTestCase):
with self.subTest(model=model_cls.__name__, tokens=tokens):
# Also leave the final output: the stack exit must handle it
# independently of the last layer's fusion decision.
stage_api = model_cls in (
Qwen3ForCausalLM,
SarvamMoEForCausalLM,
)
layer_type = DeferringStageLayer if stage_api else DeferringLayer
wrapper = SimpleNamespace(
model=SimpleNamespace(
layers=[DeferringLayer(False) for _ in range(NUM_LAYERS)],
layers=[layer_type(False) for _ in range(NUM_LAYERS)],
config=SimpleNamespace(num_hidden_layers=NUM_LAYERS),
norm=SumNorm(),
),
@@ -160,13 +160,10 @@ CASES = {
def build(case, num_layers, layer_id, config=None, **kwargs):
"""Construct one decoder layer with its submodules stubbed. Returns the
LayerCommunicator kwargs, the LayerFacts.init_new kwargs, and the
names of the stubbed submodules it built."""
"""Construct a decoder with stubbed compute and inspect its stage declarations."""
module_name, class_name, make_config, stubs, _ = CASES[case]
module = import_model(module_name)
communicator = MagicMock()
facts = MagicMock()
communicator = MagicMock(return_value=(MagicMock(), MagicMock()))
built = []
def recording_stub(name):
@@ -176,7 +173,17 @@ def build(case, num_layers, layer_id, config=None, **kwargs):
return make
patches = dict(LayerCommunicator=communicator, LayerFacts=facts)
from sglang.srt.layers.communicator import declare_attn, declare_ffn
if hasattr(module, "make_stages"):
patches = dict(
declare_attn=declare_attn,
declare_ffn=declare_ffn,
make_stages=communicator,
)
else:
facts = MagicMock()
patches = dict(LayerCommunicator=communicator, LayerFacts=facts)
patches.update({name: recording_stub(name) for name in stubs})
if hasattr(module, "get_parallel"):
patches["get_parallel"] = lambda: PARALLEL
@@ -185,7 +192,29 @@ def build(case, num_layers, layer_id, config=None, **kwargs):
config or make_config(num_layers), layer_id=layer_id, **kwargs
)
communicator.assert_called_once()
return communicator.call_args.kwargs, facts.init_new.call_args.kwargs, built
if not hasattr(module, "make_stages"):
passed = communicator.call_args.kwargs
planned = facts.init_new.call_args.kwargs
return (
passed,
{
"terminal": planned["layer_id"] == planned["num_layers"] - 1,
"enters_stack": planned["layer_id"] == 0,
"is_layer_sparse": planned["is_layer_sparse"],
},
built,
)
declaration = communicator.call_args.args[1][0]
sequence = communicator.call_args.kwargs
return (
{},
{
"terminal": sequence["terminal"],
"enters_stack": sequence["previous"] is None,
"is_layer_sparse": declaration.sparse,
},
built,
)
def planned_as_last(case, num_layers, layer_id, **kwargs):
@@ -193,10 +222,10 @@ def planned_as_last(case, num_layers, layer_id, **kwargs):
fact the communicator reads the last layer from."""
passed, planned, _ = build(case, num_layers, layer_id, **kwargs)
assert "is_last_layer" not in passed, "the plan is the only source"
return planned["layer_id"] == planned["num_layers"] - 1
return planned["terminal"]
class TestLastLayerCommunicator(CustomTestCase):
class TestTerminalStages(CustomTestCase):
def test_only_the_last_layer_is_last(self):
"""With all-reduce fusion on, only the model's last layer is marked last,
so its FFN all-reduce is never left to a next layer that does not exist."""
@@ -241,7 +270,8 @@ class TestLastLayerCommunicator(CustomTestCase):
num_layers, layer_id = draft
with self.subTest(case=case):
_, planned, _ = build(case, num_layers, layer_id, is_nextn=True)
self.assertEqual((planned["layer_id"], planned["num_layers"]), (0, 1))
self.assertTrue(planned["enters_stack"])
self.assertTrue(planned["terminal"])
def test_bailing_draft_layer_builds_an_moe(self):
"""The Bailing V2 NextN checkpoint holds expert weights, so the NextN
@@ -58,25 +58,10 @@ def delegates_to_super(forward):
)
def declares_no_deferral(node):
"""Builds its communicator with allow_deferred_ffn_reduction=False: its FFN
completes its own reduction."""
return any(
keyword.arg == "allow_deferred_ffn_reduction"
and isinstance(keyword.value, ast.Constant)
and keyword.value.value is False
for call in ast.walk(node)
if isinstance(call, ast.Call)
for keyword in call.keywords
)
def defers(node):
"""Asks whether to leave the reduction to the next layer (directly or
through ffn_exit), or wraps its output as unreduced itself."""
if declares_no_deferral(node):
return False
return any(any(calls(node, name)) for name in ("ffn_exit", UNREDUCED))
return any(any(calls(node, name)) for name in ("ffn_exit", "exit", UNREDUCED))
class Census:
@@ -115,7 +100,13 @@ class Census:
and any(
isinstance(call, ast.Call)
and isinstance(call.func, ast.Name)
and call.func.id in {"LayerCommunicator", "MHCLayerCommunicator"}
and call.func.id
in {
"LayerCommunicator",
"MHCLayerCommunicator",
"make_decoder_stages",
"make_stages",
}
for call in ast.walk(node)
)
)
@@ -129,7 +120,6 @@ class Census:
if name not in deferring
for _, node in defs
if set(base_names(node)) & deferring
and (self.include_complete or not declares_no_deferral(node))
and (
method(node, "forward") is None
or delegates_to_super(method(node, "forward"))
@@ -299,6 +289,19 @@ class TestLayerStackExit(CustomTestCase):
for name, (forward, layers) in sorted(self.complete_census.subjects().items()):
exits = list(calls(forward, EXIT)) + list(calls(forward, "take_output"))
if not exits:
batch_exits = [
call
for call in ast.walk(forward)
if isinstance(call, ast.Call)
and isinstance(call.func, ast.Attribute)
and isinstance(call.func.value, ast.Name)
and call.func.value.id == "residual_batch"
and call.func.attr in {"norm", "to_pp", "fold", "finish"}
]
if batch_exits and all(
not inside_loop(forward, call) for call in batch_exits
):
continue
if delegates_to_super(forward):
continue
if layers and all(
@@ -387,6 +390,13 @@ class TestLayerStackExit(CustomTestCase):
exits = list(calls(forward, EXIT)) + list(
calls(forward, "take_output")
)
exits += [
call
for call in calls(forward, "norm")
if isinstance(call.func, ast.Attribute)
and isinstance(call.func.value, ast.Name)
and call.func.value.id == "residual_batch"
]
self.assertEqual(len(exits), 1)
exit_call = exits[0]
self.assertFalse(inside_loop(forward, exit_call))
@@ -412,7 +422,11 @@ class TestLayerStackExit(CustomTestCase):
]
self.assertEqual(len(final_segments) + len(early_returns), 1)
for call in ast.walk(forward):
if isinstance(call, ast.Call) and is_final_norm(call):
if (
isinstance(call, ast.Call)
and call is not exit_call
and is_final_norm(call)
):
self.assertLess(exit_call.lineno, call.lineno)
checked += 1
self.assertGreater(checked, 0)
@@ -55,7 +55,7 @@ def _build(pattern, tp, capture):
layer = cls.__new__(cls)
nn.Module.__init__(layer)
layer.norm = _Norm()
layer._init_layer_communicator(config, i)
layer._init_stage_boundary(config, i)
layer.mixer = _Mixer(0.5 if kind in "M*" else 0.25, tp)
if kind == "M":
layer._forward_mamba = lambda h, batch, mixer=layer.mixer: mixer(h)
@@ -26,9 +26,9 @@ class _RecordingLayer(nn.Module):
super().__init__()
self.inputs_embeds = None
def forward(self, *, inputs_embeds, hidden_states, residual, forward_batch):
def forward(self, *, inputs_embeds, hidden_states, forward_batch):
self.inputs_embeds = inputs_embeds
return hidden_states, residual
return hidden_states
class TestNemotronHMultiTokenPredictor(CustomTestCase):
@@ -5,9 +5,9 @@ from unittest.mock import Mock
import torch
from torch import nn
from sglang.srt.layers.communicator import ADD
from sglang.srt.layers.communicator.layout import SumGroup
from sglang.srt.layers.communicator.residual import batch as residual_batch
from sglang.srt.layers.communicator.residual.add_norm import ADD
from sglang.srt.layers.communicator.residual.stream import ResidualStream
from sglang.srt.model_executor.forward_batch_info import ForwardMode
from sglang.srt.models import nemotron_h_mtp
from sglang.srt.runtime_context import get_context, get_flags, get_parallel
@@ -72,30 +72,28 @@ class TestNemotronMTPReduction(CustomTestCase):
layer.has_end_norm = False
layer.mixer = nn.Identity()
layer.norm = _Norm()
layer._init_layer_communicator(
layer._init_stage_boundary(
SimpleNamespace(hybrid_override_pattern="*E"), 1
)
partial = torch.tensor([[1.0, 2.0], [3.0, 4.0]])
residual = torch.tensor([[7.0, 3.0], [5.0, 9.0]])
expected = partial * tp + residual
stream = ResidualStream(residual)
hidden = stream.leave(
partial, ADD, declared_sum=SumGroup.ATTN_TP if tp > 1 else None
)
batch = SimpleNamespace(
forward_mode=ForwardMode.DECODE,
input_ids=torch.zeros(2, dtype=torch.long),
residual_stream=None,
residual_stream=stream,
)
stream = residual_batch.start(batch)
stream.write(residual)
partial = stream.leave(
partial, ADD, declared_sum=SumGroup.ATTN_TP if tp > 1 else None
)
hidden, output_residual = layer(
inputs_embeds=torch.zeros_like(residual),
hidden_states=partial,
residual=stream,
hidden = layer(
inputs_embeds=torch.zeros_like(partial),
hidden_states=hidden,
forward_batch=batch,
)
torch.testing.assert_close(hidden, expected)
torch.testing.assert_close(output_residual.residual, expected)
torch.testing.assert_close(batch.residual_stream.residual, expected)
self.assertEqual(reduce.call_count, int(tp > 1))
@@ -10,7 +10,6 @@ from sglang.srt.layers.communicator import (
NORM_QUANT_READ,
NORM_READ,
Layout,
MixerExit,
StageDecl,
StageInput,
StageOutput,
@@ -19,6 +18,7 @@ from sglang.srt.layers.communicator import (
UnreducedOutput,
stage_edges,
)
from sglang.srt.layers.communicator.layer import MixerExit
from sglang.srt.layers.moe.utils import should_skip_mlp_all_reduce
from sglang.srt.models import nemotron_h_utils as utils
from sglang.srt.runtime_context import get_parallel
@@ -164,11 +164,10 @@ class TestMixerExit(CustomTestCase):
)
communicator = SimpleNamespace(
_batch_steps=lambda batch: SimpleNamespace(ffn_output=produced),
_context=SimpleNamespace(tp_size=2),
_ffn_sum_can_move_to_next_layer=MagicMock(return_value=movable),
)
hidden = torch.ones(2, 4)
with get_parallel().override(tp_group=tp_group):
with get_parallel().override(tp_group=tp_group, tp_size=2):
with MixerExit(communicator, None) as mixer_exit:
skipped = should_skip_mlp_all_reduce()
output = mixer_exit.finish(hidden)
@@ -1,3 +1,5 @@
from sglang.srt.layers.communicator.layer import FfnExit
"""A Step-3.5 dense layer must leave its all-reduce out when postprocess
reduce-scatters its output; otherwise the output is summed twice. The layer's
FFN exit publishes the decision while the MLP runs, then completes it."""
@@ -47,7 +49,7 @@ def dense_layer(*, reduce_scatter):
complete=complete,
),
)
communicator.ffn_exit = lambda fb: comm.FfnExit(communicator, fb)
communicator.ffn_exit = lambda fb: FfnExit(communicator, fb)
object.__setattr__(layer, "layer_communicator", communicator)
object.__setattr__(layer, "use_moe", False)
object.__setattr__(layer, "mlp", mlp)
@@ -204,5 +204,39 @@ class TestDraftWeightUpdateRecord(CustomTestCase):
self.assertEqual(get_model().model_path, seeded.model_path)
class TestDraftBoundaryPolicy(unittest.TestCase):
def test_scope_restores_target_policy_after_nested_failure(self):
from sglang.srt.runtime_context import (
get_exec,
get_spec,
publish,
reset_context,
)
from sglang.srt.server_args import ServerArgs
reset_context()
try:
publish(
ServerArgs(
model_path="dummy",
boundary_reduction="rs+rsv",
),
role="test",
)
self.assertEqual(get_exec().comm.boundary_reduction, "rs+rsv")
with (
get_spec().override(speculative_boundary_reduction="ar"),
self.assertRaisesRegex(RuntimeError, "draft failed"),
):
with draft_model_build_scope():
self.assertEqual(get_exec().comm.boundary_reduction, "ar")
with draft_model_build_scope():
self.assertEqual(get_exec().comm.boundary_reduction, "ar")
raise RuntimeError("draft failed")
self.assertEqual(get_exec().comm.boundary_reduction, "rs+rsv")
finally:
reset_context()
if __name__ == "__main__":
unittest.main()
@@ -106,6 +106,8 @@ class TestModelOverridableWhitelist(CustomTestCase):
"fp8_gemm_runner_backend",
"fp4_gemm_runner_backend",
"disable_custom_all_reduce",
"boundary_reduction",
"speculative_boundary_reduction",
"enable_aiter_allreduce_fusion",
"disable_aiter_allreduce_fusion_in_prefill",
"disable_aiter_allreduce_fusion_in_decode",
@@ -121,6 +123,119 @@ class TestModelOverridableWhitelist(CustomTestCase):
)
class TestBoundaryReductionDefaults(CustomTestCase):
def test_defaults_wrappers_and_explicit_policies_reach_runtime(self):
from unittest.mock import patch
from sglang.srt.arg_groups.boundary_reduction import resolve_boundary_reduction
from sglang.srt.arg_groups.overrides import run_post_process_pass
from sglang.srt.runtime_context import (
get_exec,
get_spec,
publish,
reset_context,
)
from sglang.srt.server_args import ServerArgs
cases = [
(SimpleNamespace(architectures=[architecture]), default)
for architecture, default in (
("Qwen3ForCausalLM", "ar"),
("Qwen3Model", "ar"),
("MossVLForConditionalGeneration", "ar"),
("BailingMoELinearForCausalLM", "rsv"),
("BailingMoeV2_5ForCausalLM", "rsv"),
("LongcatFlashForCausalLM", "rsv"),
("LongcatFlashForCausalLMNextN", "rsv"),
("Step3VLForConditionalGeneration", "rsv"),
("MiMoV2ForCausalLM", "rs+rsv"),
("Qwen3MoeForCausalLM", "rs+rsv"),
("OtherModel", "rs+rsv"),
)
]
for key in ("text_config", "llm_config"):
for backbone, expected in (("qwen3", "ar"), ("qwen3_moe", "rs+rsv")):
cases.append(
(
SimpleNamespace(**{key: SimpleNamespace(model_type=backbone)}),
expected,
)
)
cases.append(
(
SimpleNamespace(
get_text_config=lambda: SimpleNamespace(model_type="qwen3")
),
"ar",
)
)
for config, default in cases:
for requested in ("auto", "ar", "rs", "rsv", "rs+rsv"):
with self.subTest(config=config, requested=requested):
reset_context()
try:
args = ServerArgs(
model_path="local-model", boundary_reduction=requested
)
args._model_config = SimpleNamespace(hf_config=config)
with patch(
"sglang.srt.arg_groups.pipeline.run_resolution_pipeline",
side_effect=lambda record: run_post_process_pass(
record, resolve_boundary_reduction
),
):
publish(args, role="test")
expected = default if requested == "auto" else requested
self.assertEqual(get_exec().comm.boundary_reduction, expected)
self.assertEqual(
get_spec().speculative_boundary_reduction, expected
)
self.assertEqual(args.boundary_reduction, requested)
finally:
reset_context()
def test_independent_draft_and_mtp_resolution(self):
from unittest.mock import patch
from sglang.srt.arg_groups.boundary_reduction import resolve_boundary_reduction
from sglang.srt.arg_groups.overrides import (
resolving_view,
run_post_process_pass,
)
from sglang.srt.server_args import ServerArgs
for same_model in (False, True):
for requested in ("auto", "ar", "rs", "rsv", "rs+rsv"):
args = ServerArgs(
model_path="target",
boundary_reduction=requested,
speculative_algorithm="EAGLE",
speculative_draft_model_path="target" if same_model else "draft",
)
args._model_config = SimpleNamespace(
hf_config=SimpleNamespace(architectures=["MiMoV2ForCausalLM"])
)
with patch(
"sglang.srt.utils.hf_transformers_utils.get_config",
return_value=SimpleNamespace(model_type="qwen3"),
) as load:
run_post_process_pass(args, resolve_boundary_reduction)
resolved = resolving_view(args)
self.assertEqual(
resolved.boundary_reduction,
"rs+rsv" if requested == "auto" else requested,
)
self.assertEqual(
resolved.speculative_boundary_reduction,
("rs+rsv" if same_model else "ar")
if requested == "auto"
else requested,
)
self.assertEqual(
load.call_count, int(requested == "auto" and not same_model)
)
class TestDSparkCheckpointConfig(CustomTestCase):
def test_sample_from_anchor_is_read_from_checkpoint_config(self):
from sglang.srt.speculative.dspark_components.dspark_config import (