From 64e404263e8d895e082ba01617cf10d6409f3831 Mon Sep 17 00:00:00 2001 From: qiaonan02 <167996578+QNan0513@users.noreply.github.com> Date: Wed, 19 Aug 2026 08:15:03 +0800 Subject: [PATCH] [Doc] Fix TP and attention-TP group layout in initialize_model_parallel docstring (#34862) Co-authored-by: NanoByte0513 <167996578+NanoByte0513@users.noreply.github.com> Co-authored-by: Xiaoyu Zhang <1182563586@qq.com> --- python/sglang/srt/distributed/parallel_state.py | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/python/sglang/srt/distributed/parallel_state.py b/python/sglang/srt/distributed/parallel_state.py index b131a48e6..75dfdd0ce 100644 --- a/python/sglang/srt/distributed/parallel_state.py +++ b/python/sglang/srt/distributed/parallel_state.py @@ -2372,12 +2372,16 @@ def initialize_model_parallel( Let's say we use 2 GPUs for attention context parallelism (attn_cp_size=2) and 4 GPUs for attention tensor parallelism (attn_tp_size=4). As for MoE part, we use 2 GPUs for moe data - parallelism (moe_dp_size=2) and 4 GPUs for moe expert parallelism (moe_ep_size=4). The present + parallelism (moe_dp_size=2) and 4 GPUs for moe expert parallelism (moe_ep_size=4). Note that + this implies tensor_model_parallel_size=8 (attn_tp_size = tp_size // attn_cp_size // + attn_dp_size), so all 8 GPUs form a single tensor model-parallel group. The present function will create the following groups: - 2 tensor model-parallel groups: - [g0, g1, g2, g3], [g4, g5, g6, g7] + 1 tensor model-parallel group: + [g0, g1, g2, g3, g4, g5, g6, g7] 4 attention context-parallel groups: [g0, g4], [g1, g5], [g2, g6], [g3, g7] + 2 attention tensor-parallel groups: + [g0, g1, g2, g3], [g4, g5, g6, g7] 2 moe expert-parallel groups: [g0, g1, g2, g3], [g4, g5, g6, g7] 4 moe data-parallel groups: