[Refactor] Send producer-written residual streams through to_pp (#41753)

This commit is contained in:
Cheng Wan
2026-09-29 21:11:02 -07:00
committed by GitHub
parent 843f50f5c1
commit faa5064f6e
3 changed files with 9 additions and 5 deletions
+1 -1
View File
@@ -195,7 +195,7 @@ hidden_states = residual_batch.final_norm(
`skip_empty=True` still completes owed work on an empty batch, which may be a collective that other ranks join, and skips only the norm kernel. `final_norm()` and `to_pp()` release the batch's stream after they hand on its output.
This is a non-pipeline stack fragment. Pipeline reception uses the first boundary's `from_pp(tensors, forward_batch)` to reconstruct a stream; the sender uses `residual_batch.to_pp()`. Dynamic completion work is resolved before transport. By default, `to_pp()` preserves a statically declared partial sum for the receiver's incoming contract to reconstruct and complete. Do not call `complete_output()` before that handoff: it would complete a sum that the receiver still expects to perform.
This is a non-pipeline stack fragment. Pipeline reception uses the first boundary's `from_pp(tensors, forward_batch)` to reconstruct a stream; the sender uses `residual_batch.to_pp()`. A stream the producer already wrote, such as MHC's expanded streams, travels as `hidden_states` alone and arrives written. Dynamic completion work is resolved before transport. By default, `to_pp()` preserves a statically declared partial sum for the receiver's incoming contract to reconstruct and complete. Do not call `complete_output()` before that handoff: it would complete a sum that the receiver still expects to perform.
A terminal FFN normally completes its output before the final norm. The exception is a MoE finalize handoff. When the stage's fusion provider can defer the finalize at the terminal layer (for CuteDSL, `install_cutedsl_fusion(..., terminal_finalize=True)`), the exit may leave a `DeferredFinalize` for the final norm. Pass the handle straight to `residual_batch.final_norm(hidden_states, forward_batch, self.norm, finalize_norm=service)`, where `service` is the object `install_cutedsl_fusion()` returned; its `finalize(handoff, residual, gamma)` fuses finalize, all-reduce, residual add, and norm. The fused path reads `layernorm.gemma_weight`, so it currently expects a `GemmaRMSNorm` final norm. Without `finalize_norm`, `final_norm()` first completes the handoff unfused, and `capture` is then allowed. With `finalize_norm`, `final_norm()` rejects `capture` when a handoff arrives. Calling `complete_output()` first runs the producer's unfused finalize, and `snapshot()` raises on a handoff. See `python/sglang/srt/models/qwen3_5.py`.
@@ -115,13 +115,19 @@ def to_pp(hidden_states, forward_batch, *, preserve_declared=True):
partial sum unreduced for the receiver's from_pp to complete; pass
False only when the receiver does not declare that sum.
Runtime-selected completion work is finished before transport.
Runtime-selected completion work is finished before transport. A stream
the producer already wrote (MHC writes its streams at the FFN exit) has no
separate residual and is sent as hidden_states alone; the receiver's
from_pp reconstructs it as written.
"""
hidden_states, residual = stream_of(forward_batch).export(
hidden_states, preserve_declared=preserve_declared
)
forward_batch.residual_stream = None
return PPProxyTensors({"hidden_states": hidden_states, "residual": residual})
tensors = {"hidden_states": hidden_states}
if residual is not None:
tensors["residual"] = residual
return PPProxyTensors(tensors)
def snapshot(hidden_states, forward_batch):
-2
View File
@@ -1185,8 +1185,6 @@ class Glm5NextModel(nn.Module):
)
if not self.pp_group.is_last_rank:
if self.config.mhc:
return PPProxyTensors({"hidden_states": hidden_states})
return residual_batch.to_pp(hidden_states, forward_batch)
else:
hidden_states = residual_batch.complete_output(hidden_states, forward_batch)