minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

config: spell the parallel config tier at the call site (#36250)

Browse Source
This commit is contained in:
Cheng Wan
2026-08-26 03:00:28 -07:00
committed by GitHub
parent 689ade69d1
commit 8005df61d3
136 changed files with 1033 additions and 795 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/skills/sglang-runtime-context/SKILL.md
+58 -45
View File
@@ -15,7 +15,7 @@ One container owns process-static runtime state: `sglang.srt.runtime_context.Run
| runtime flags | `get_flags()` | state that is *not* a pure function of config: `capture` (cuda-graph lifecycle), `moe` (ACTIVE backends, swappable), `dp` (DP-attention runtime flags) | materialized at subsystem init; groups offer `override()` for tests | | runtime flags | `get_flags()` | state that is *not* a pure function of config: `capture` (cuda-graph lifecycle), `moe` (ACTIVE backends, swappable), `dp` (DP-attention runtime flags) | materialized at subsystem init; groups offer `override()` for tests |
| resources | `get_resources()`, `get_stream(name)`, `get_buffer(name, factory)` | process-level handles: graph pools, EPLB state, EP dispatcher state, named side streams, workspace buffers | lazy; cleared by `reset_context()` | | resources | `get_resources()`, `get_stream(name)`, `get_buffer(name, factory)` | process-level handles: graph pools, EPLB state, EP dispatcher state, named side streams, workspace buffers | lazy; cleared by `reset_context()` |
| per-forward | `get_forward()` | forward-scoped flags (multi-stream switch, MoE output buffer, attn-TP inputs, extend-in-batch) | contextvar-backed; `scoped(**kw)` restores on exit; new threads see defaults | | per-forward | `get_forward()` | forward-scoped flags (multi-stream switch, MoE output buffer, attn-TP inputs, extend-in-batch) | contextvar-backed; `scoped(**kw)` restores on exit; new threads see defaults |
| parallel | `get_parallel()` | **dual**: live topology (tp/pp/moe/attn sizes, ranks, groups — `@property`, read-through) *plus* parallel config-bag leaves via `__getattr__` | live: after dist init; config leaves: after publish | | parallel | `get_parallel()` | **dual, spelled**: bare names are the live topology (tp/pp/moe/attn sizes, ranks, groups — `@property`, read-through); `get_parallel().config.<leaf>` is the parallel config bag | live: after dist init; `config`: after publish |
`reset_context()` (unit-test teardown) drops the published config and installs fresh `reset_context()` (unit-test teardown) drops the published config and installs fresh
flags/resources/forward tiers. flags/resources/forward tiers.
@@ -214,23 +214,29 @@ row for a method the Ray actor does not have, and an effective-field set
without `load_format` -- and both were invisible because the assertion had without `load_format` -- and both were invisible because the assertion had
slack (`>= len(...) - 1`) or compared key names instead of value sources. slack (`>= len(...) - 1`) or compared key names instead of value sources.
### `get_parallel()`: config leaves vs live topology ### `get_parallel()`: live topology bare, configuration under `config`
Config leaves (`nccl_port`, `enable_dp_attention`, `dp_size`, `ep_size`, **Bare is the live group, `config` is what was configured.** `get_parallel().tp_size`
`dwdp_size`, ...) resolve through the parallel bag; live topology (`tp_size`, and its size / rank / group siblings are `@property` read-through over the canonical
`attn_tp_group`, ranks) are `@property` and **win on name collisions**. Five topology getters; `get_parallel().config.<leaf>` reads the published `parallel` bag
sizes are live-shadowed (`tp/pp/dcp/attn_cp/moe_dp_size`): the live property always (`nccl_port`, `enable_dp_attention`, `dp_size`, `ep_size`, `dwdp_size`, ... and the
wins on the accessor, so a config-intent read of those goes through five sizes that also have a live property). A bare read of a config-only leaf raises
`configured_tp_size()` / `configured_pp_size()` / `configured_moe_dp_size()` / an `AttributeError` naming the `.config` spelling — the tier is never guessed from
`configured_attn_cp_size()` (DCP: the live `get_parallel().attn_dcp_size` / whether a property happens to exist.
`.dcp_enabled`, which are safe with no group installed — they answer `1` /
`False` — but report the *effective* topology, never the requested size; The two tiers are **not** two spellings of one number. Live diverges from configured
a config-intent read would need its own accessor, which no call site wherever elastic EP scales the world away from the launch shape, and wherever
requires today). A process-global seed field-read of one of these `initialize_model_parallel` aliases `_MOE_DP` to `_ATTN_CP` (`attn_cp_size >
sizes (`get_server_args().tp_size`, or an alias of it) is a read-ratchet failure; the moe_dp_size`), which makes a live comparison of that pair degenerate. The five
sites that legitimately go around the live property are the `configured_*_size()` live-shadowed sizes (`tp/pp/dcp/attn_cp/moe_dp_size`) are where the choice matters,
readers, and those are what the ratchet registers, each with its reason and every business read of `get_parallel().config.<one of them>` is registered with
(`_CONFIGURED_SIZE_CALL_SITES` in `test_global_config_read_ratchet.py`). A its reason in `_CONFIGURED_SIZE_CALL_SITES` (`test_global_config_read_ratchet.py`).
DCP has a third shape: the live `get_parallel().attn_dcp_size` / `.dcp_enabled`
answer the *effective* topology (`1` / `False` with no group installed), never the
requested size — `.config.dcp_size` is the requested one.
A process-global seed field-read of one of these sizes
(`get_server_args().tp_size`, or an alias of it) is a read-ratchet failure. A
`server_args` the object was *handed* is a different thing and not a ratchet `server_args` the object was *handed* is a different thing and not a ratchet
matter — see "Reads that legitimately stay on a ServerArgs instance". matter — see "Reads that legitimately stay on a ServerArgs instance".
Fail-loud is narrower: before dist init, a live size/group read raises — except Fail-loud is narrower: before dist init, a live size/group read raises — except
@@ -238,25 +244,31 @@ the DCP pair, which degrades instead (`dcp_enabled` → `False`,
`attn_dcp_size` → `1` when no group is installed; `attn_dcp_size` → `1` when no group is installed;
`test_attn_dcp_defaults_when_group_is_uninitialized` pins this). After init, `test_attn_dcp_defaults_when_group_is_uninitialized` pins this). After init,
only the DCP group is optional (`_DCP` exists only when `dcp_size > 1`; attn-CP and only the DCP group is optional (`_DCP` exists only when `dcp_size > 1`; attn-CP and
moe-DP always install, as size-1 aliases if unused). `ParallelContext.__getattr__` is deliberately dynamo-traceable (no moe-DP always install, as size-1 aliases if unused). The `config` hop is
deliberately dynamo-traceable (a plain property over a slot, no
`object.__getattribute__`); gate helpers like `enable_moe_dense_fully_dp()` run inside `object.__getattribute__`); gate helpers like `enable_moe_dense_fully_dp()` run inside
compiled model forwards (`test_parallel_config_leaves_trace_under_torch_compile` pins compiled model forwards (`test_parallel_config_leaves_trace_under_torch_compile` pins
this). this).
A third surface carries the same names: `ParallelState` (`self.ps` / `mr.ps`), the
frozen per-process snapshot built once in `Scheduler.__init__` from these configured
sizes plus this process's ranks, and handed down (draft runners included). Prefer it
where an object was handed one; it is not a global accessor.
### Reading config: the seed is off limits ### Reading config: the seed is off limits
`get_server_args().field` in business code is a ratchet failure. Read: `get_server_args().field` in business code is a ratchet failure. Read:
- **a resolved leaf** → its namespace bag (`get_exec().moe.moe_runner_backend`, - **a resolved leaf** → its namespace bag (`get_exec().moe.moe_runner_backend`,
`get_schedule().chunked_prefill_size`, …). Bag-backed reads — a leaf directly, or `get_schedule().chunked_prefill_size`, …). Bag-backed reads — a leaf directly, or
a bag-derived accessor below, including `configured_*_size()` which reads the a bag-derived accessor below, including the `get_parallel().config` hop — are
parallel bag's own leaf — are what see post-publish overrides. Only the what see post-publish overrides. Only the
instance-derived accessors (the ones with no leaf to read) answer from the instance-derived accessors (the ones with no leaf to read) answer from the
startup record and therefore do not. startup record and therefore do not.
- **a leaf the caller names at runtime** (a readback reporting a list of fields) - **a leaf the caller names at runtime** (a readback reporting a list of fields)
→ `get_context().config_leaf(name)`; it resolves the name through `NS` and → `get_context().config_leaf(name)`; it resolves the name through `NS` and
raises on a non-leaf. A call site that knows its field reads the bag leaf. raises on a non-leaf. A call site that knows its field reads the bag leaf.
- **the live topology** → `get_parallel()`. - **the live topology** → `get_parallel()` (bare names).
- **a value derived from published leaves** → an accessor in `runtime_context` that - **a value derived from published leaves** → an accessor in `runtime_context` that
derives it *from the bags*: `mamba_extra_buffer_enabled()` / derives it *from the bags*: `mamba_extra_buffer_enabled()` /
`mamba_extra_buffer_lazy_enabled()` read `get_memory()` and `get_exec()`, so `mamba_extra_buffer_lazy_enabled()` read `get_memory()` and `get_exec()`, so
@@ -275,24 +287,18 @@ this).
property with no bag of its own. A new derived member gets an accessor here property with no bag of its own. A new derived member gets an accessor here
rather than call sites reaching for the record, and only when the bag-derived rather than call sites reaching for the record, and only when the bag-derived
shape above cannot express it. shape above cannot express it.
- **what was *configured*, where `get_parallel()` shadows it with the live value** - **what was *configured*, where the bare name is the live value**
→ `configured_{tp,pp,moe_dp,attn_cp}_size()` — the full names are → `get_parallel().config.{tp,pp,moe_dp,attn_cp,dcp}_size`. It reads the parallel
`configured_tp_size`, `configured_pp_size`, `configured_moe_dp_size`, bag's own leaf, so it answers with the resolved configuration and follows a
`configured_attn_cp_size`. They read the parallel bag's own leaf (going post-publish override. The DCP live pair (`get_parallel().attn_dcp_size` /
around the live property that shadows those four names), so they answer with `.dcp_enabled`) is a different question again: it answers the effective topology
the resolved configuration and follow a post-publish override. DCP has no configured accessor because no (`1` / `False` when no group is installed), never the requested size, and it does
config-intent DCP call site exists today — the live reads go through not *need* dist init to answer. Every (file, size) pair is registered
`get_parallel().attn_dcp_size` / `.dcp_enabled`, which answer the effective
topology (`1` / `False` when no group is installed). A site
that must know the *requested* DCP size before dist init needs its own
`configured_dcp_size()` (and an entry in `_CONFIGURED_SIZE_CALL_SITES`, which lives
in the ratchet test, not in this skill); note the live pair does not *need*
dist init — with no group it answers `1` / `False` — it just cannot answer
with the requested size. Every (file, accessor) pair is registered
with its reason in `test_global_config_read_ratchet.py` with its reason in `test_global_config_read_ratchet.py`
(`_CONFIGURED_SIZE_CALL_SITES`), and that test fails if the code and the list (`_CONFIGURED_SIZE_CALL_SITES`), and that test fails if the code and the list
disagree — a new file, or a new accessor in a listed file, has to be added — so a new site needs both an answer the live property cannot give and disagree — a new file, or a new size in a listed file, has to be added — so a new
an entry saying what it is. site needs both an answer the live property cannot give and an entry saying what
it is.
- **this runner's resolved value** → the runner - **this runner's resolved value** → the runner
(`prefill_attention_backend_str`, `kv_cache_dtype_str`, (`prefill_attention_backend_str`, `kv_cache_dtype_str`,
`draft_attention_backend`, `num_fused_shared_experts` on the model). `draft_attention_backend`, `num_fused_shared_experts` on the model).
@@ -498,13 +504,19 @@ ONE thread — do not design for TBO threads that don't exist.
— including local copies of an alias, `cfg = sa` — module-level, or parked on an — including local copies of an alias, `cfg = sa` — module-level, or parked on an
instance attribute, plus the `getattr(..., "field")` spelling of each; a name instance attribute, plus the `getattr(..., "field")` spelling of each; a name
computed at runtime or indirection deeper than a local name copy is census-tool computed at runtime or indirection deeper than a local name copy is census-tool
territory, per the test's docstring). The scanners match `get_server_args` and territory, per the test's docstring). The scanner matches `get_server_args` by its
`configured_*_size` by their literal names, and the same file *bans* literal name, and the same file *bans* `import ... as` renames of it so that
`import ... as` renames of them so that matching stays sound. Exempt by owner matching stays sound. Exempt by owner
module only (`runtime_context.py`, `server_args.py`, `arg_groups/`). The same file module only (`runtime_context.py`, `server_args.py`, `arg_groups/`). The same file
carries `_CONFIGURED_SIZE_CALL_SITES`, the (file, accessor) map of every carries `_CONFIGURED_SIZE_CALL_SITES`, the (file, size) map of every
`configured_*_size()` reader with the reason the live property cannot serve it — a new `get_parallel().config.<live-shadowed size>` reader with the reason the live property
file or a new accessor in a listed file must be added there. cannot serve it — a new file or a new size in a listed file must be added there. Its
subject set is *derived* (property names ∩ `parallel` NS leaves), and it resolves
every spelling of the call itself — an aliased import, a module-qualified receiver
(including the whole dotted path an unaliased `import` binds), a local bound to either
hop — so neither a rename nor a new shadowed size escapes it.
`TestParallelConfigReadSpellings` in that file runs each spelling, because a spelling
the scanner cannot resolve drops the read instead of failing anything.
6. **Module-state ratchet** (`test_module_state_ratchet.py`): `global` statements in the 6. **Module-state ratchet** (`test_module_state_ratchet.py`): `global` statements in the
flag-owning layers are pinned by name. A new module-level runtime global belongs on a flag-owning layers are pinned by name. A new module-level runtime global belongs on a
flags group / resources slot instead; migrating a pinned survivor must shrink the pin. flags group / resources slot instead; migrating a pinned survivor must shrink the pin.
@@ -538,8 +550,9 @@ Never module-skip a test "until the migration settles" — seed the context inst
form** (attribute-source ints get automatic-dynamic after the first size form** (attribute-source ints get automatic-dynamic after the first size
change). Bools (≤2 values) are tolerable in any form — see change). Bools (≤2 values) are tolerable in any form — see
`ForwardFlags._GRAPH_VISIBLE`. Config-bag leaves are real instance attributes for `ForwardFlags._GRAPH_VISIBLE`. Config-bag leaves are real instance attributes for
exactly this reason, and `ParallelContext.__getattr__` must stay free of exactly this reason, and the parallel config tier is read through the plain
`object.__getattribute__` (dynamo graph-breaks on it). Before moving such state, `ParallelContext.config` property for the same reason (`__getattr__` is
error-only, and `object.__getattribute__` graph-breaks). Before moving such state,
prove its readers sit outside compile coverage; a piecewise-prefill boot of a small prove its readers sit outside compile coverage; a piecewise-prefill boot of a small
model is the fast check (recompile storms show as `torch._dynamo hit model is the fast check (recompile storms show as `torch._dynamo hit
config.recompile_limit` during the compile pass). config.recompile_limit` during the compile pass).
python/sglang/benchmark/one_batch.py
+1 -1
View File
@@ -542,7 +542,7 @@ def _maybe_prepare_mlp_sync_batch(batch: ScheduleBatch, model_runner):
prepare_mlp_sync_batch_raw( prepare_mlp_sync_batch_raw(
batch, batch,
model_runner=model_runner, model_runner=model_runner,
dp_size=get_parallel().dp_size, dp_size=get_parallel().config.dp_size,
attn_tp_size=get_parallel().attn_tp_size, attn_tp_size=get_parallel().attn_tp_size,
attn_cp_size=model_runner.ps.attn_cp_size, attn_cp_size=model_runner.ps.attn_cp_size,
tp_group=model_runner.tp_group, tp_group=model_runner.tp_group,
python/sglang/srt/batch_overlap/two_batch_overlap.py
+1 -1
View File
@@ -769,7 +769,7 @@ class TboForwardBatchPreparer:
# TODO improve, e.g. unify w/ `init_raw` # TODO improve, e.g. unify w/ `init_raw`
if ( if (
get_parallel().moe_dense_tp_size == 1 get_parallel().config.moe_dense_tp_size == 1
and batch.global_dp_buffer_len is not None and batch.global_dp_buffer_len is not None
): ):
sum_len = end_token_index - start_token_index sum_len = end_token_index - start_token_index
python/sglang/srt/disaggregation/common/conn.py
+17 -14
View File
@@ -37,7 +37,6 @@ from sglang.srt.layers.dp_attention import (
get_attention_dp_size, get_attention_dp_size,
) )
from sglang.srt.runtime_context import ( from sglang.srt.runtime_context import (
configured_pp_size,
get_disagg, get_disagg,
get_parallel, get_parallel,
get_serving, get_serving,
@@ -171,7 +170,7 @@ class CommonKVManager(BaseKVManager):
# for p/d multi node infer # for p/d multi node infer
self.bootstrap_host = get_serving().host self.bootstrap_host = get_serving().host
self.bootstrap_port = get_disagg().disaggregation_bootstrap_port self.bootstrap_port = get_disagg().disaggregation_bootstrap_port
self.dist_init_addr = get_parallel().dist_init_addr self.dist_init_addr = get_parallel().config.dist_init_addr
parallel = get_parallel() parallel = get_parallel()
self.attn_tp_size = parallel.attn_tp_size self.attn_tp_size = parallel.attn_tp_size
self.attn_tp_rank = parallel.attn_tp_rank self.attn_tp_rank = parallel.attn_tp_rank
@@ -182,16 +181,19 @@ class CommonKVManager(BaseKVManager):
self.attn_dp_size = get_attention_dp_size() self.attn_dp_size = get_attention_dp_size()
self.attn_dp_rank = get_attention_dp_rank() self.attn_dp_rank = get_attention_dp_rank()
self.system_dp_size = ( self.system_dp_size = (
1 if get_parallel().enable_dp_attention else get_parallel().dp_size 1
if get_parallel().config.enable_dp_attention
else get_parallel().config.dp_size
) )
self.system_dp_rank = ( self.system_dp_rank = (
self.kv_args.system_dp_rank if self.kv_args.system_dp_rank else 0 self.kv_args.system_dp_rank if self.kv_args.system_dp_rank else 0
) )
self.pp_size = configured_pp_size() self.pp_size = get_parallel().config.pp_size
self.pp_rank = self.kv_args.pp_rank self.pp_rank = self.kv_args.pp_rank
self.local_ip = get_local_ip_auto() self.local_ip = get_local_ip_auto()
cp_sharded_prefill = self.attn_cp_size > 1 and ( cp_sharded_prefill = self.attn_cp_size > 1 and (
self.is_hybrid_mla_backend or get_parallel().enable_dsa_cache_layer_split self.is_hybrid_mla_backend
or get_parallel().config.enable_dsa_cache_layer_split
) )
hybrid_decode_pulls_all_ranks = ( hybrid_decode_pulls_all_ranks = (
@@ -304,7 +306,7 @@ class CommonKVManager(BaseKVManager):
return ( return (
self.attn_cp_size > 1 self.attn_cp_size > 1
and self.attn_cp_rank != 0 and self.attn_cp_rank != 0
and not get_parallel().enable_dsa_cache_layer_split and not get_parallel().config.enable_dsa_cache_layer_split
) )
def requires_dcp_relayout(self, dst_dcp_size: int, dst_dcp_rank: int) -> bool: def requires_dcp_relayout(self, dst_dcp_size: int, dst_dcp_rank: int) -> bool:
@@ -749,7 +751,7 @@ class CommonKVManager(BaseKVManager):
`Connection refused`, and the leader's `prefill_port_table` ends `Connection refused`, and the leader's `prefill_port_table` ends
up missing rows. up missing rows.
""" """
if not self.dist_init_addr or get_parallel().nnodes == 1: if not self.dist_init_addr or get_parallel().config.nnodes == 1:
return local_port return local_port
if not (dist.is_available() and dist.is_initialized()): if not (dist.is_available() and dist.is_initialized()):
@@ -801,8 +803,8 @@ class CommonKVManager(BaseKVManager):
"rank_port": self.rank_port, "rank_port": self.rank_port,
"page_size": self.kv_args.page_size, "page_size": self.kv_args.page_size,
"kv_cache_dtype": self.kv_cache_dtype_str, "kv_cache_dtype": self.kv_cache_dtype_str,
"load_balance_method": get_parallel().load_balance_method, "load_balance_method": get_parallel().config.load_balance_method,
"enable_dsa_cache_layer_split": get_parallel().enable_dsa_cache_layer_split, "enable_dsa_cache_layer_split": get_parallel().config.enable_dsa_cache_layer_split,
# Self-register the HTTP API port so the decode can derive the PD # Self-register the HTTP API port so the decode can derive the PD
# retract rebootstrap /generate URL from bootstrap info instead of a # retract rebootstrap /generate URL from bootstrap info instead of a
# router-injected pd_rebootstrap_prefill_url. # router-injected pd_rebootstrap_prefill_url.
@@ -1184,11 +1186,12 @@ class CommonKVSender(BaseKVSender):
return return
self.kv_mgr.update_status(self.bootstrap_room, KVPoll.Bootstrapping) self.kv_mgr.update_status(self.bootstrap_room, KVPoll.Bootstrapping)
if get_parallel().dp_size > 1 and not req_has_disagg_prefill_dp_rank: if get_parallel().config.dp_size > 1 and not req_has_disagg_prefill_dp_rank:
if get_parallel().load_balance_method != "follow_bootstrap_room": if get_parallel().config.load_balance_method != "follow_bootstrap_room":
self._register_prefill_dp_rank() self._register_prefill_dp_rank()
elif ( elif (
self.kv_mgr.attn_dp_rank != self.bootstrap_room % get_parallel().dp_size self.kv_mgr.attn_dp_rank
!= self.bootstrap_room % get_parallel().config.dp_size
): ):
# follow_bootstrap_room was overridden by external routed_dp_rank # follow_bootstrap_room was overridden by external routed_dp_rank
if envs.SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK.get(): if envs.SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK.get():
@@ -1199,7 +1202,7 @@ class CommonKVSender(BaseKVSender):
f"follow_bootstrap_room conflict: dispatched to dp_rank " f"follow_bootstrap_room conflict: dispatched to dp_rank "
f"{self.kv_mgr.attn_dp_rank} but bootstrap_room " f"{self.kv_mgr.attn_dp_rank} but bootstrap_room "
f"{self.bootstrap_room} implies dp_rank " f"{self.bootstrap_room} implies dp_rank "
f"{self.bootstrap_room % get_parallel().dp_size}. " f"{self.bootstrap_room % get_parallel().config.dp_size}. "
f"Set SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK=1 " f"Set SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK=1 "
f"to allow mixed routing.", f"to allow mixed routing.",
) )
@@ -1273,7 +1276,7 @@ class CommonKVSender(BaseKVSender):
if ( if (
self.kv_mgr.enable_all_cp_ranks_for_transfer self.kv_mgr.enable_all_cp_ranks_for_transfer
and not get_parallel().enable_dsa_cache_layer_split and not get_parallel().config.enable_dsa_cache_layer_split
): ):
kv_indices, index_slice = filter_kv_indices_for_cp_rank( kv_indices, index_slice = filter_kv_indices_for_cp_rank(
self.kv_mgr, self.kv_mgr,
python/sglang/srt/disaggregation/encoder/http_server.py
+1 -1
View File
@@ -207,7 +207,7 @@ def launch_server(server_args: ServerArgs):
configure_logger(server_args, prefix=" encode_server") configure_logger(server_args, prefix=" encode_server")
publish(server_args, role="encoder") publish(server_args, role="encoder")
if get_parallel().dp_size > 1: if get_parallel().config.dp_size > 1:
dp_dispatcher = launch_dp_runtime(server_args) dp_dispatcher = launch_dp_runtime(server_args)
# runtime initializes multiprocess metrics before spawning; # runtime initializes multiprocess metrics before spawning;
# HTTP only exposes their endpoint. # HTTP only exposes their endpoint.
python/sglang/srt/disaggregation/encoder/runtime.py
+10 -9
View File
@@ -50,7 +50,6 @@ from sglang.srt.observability.trace import (
trace_set_thread_info, trace_set_thread_info,
) )
from sglang.srt.runtime_context import ( from sglang.srt.runtime_context import (
configured_tp_size,
get_observability, get_observability,
get_parallel, get_parallel,
get_serving, get_serving,
@@ -1491,10 +1490,10 @@ def launch_local_runtime(server_args: ServerArgs) -> EncoderRuntime:
This function owns backend construction only. HTTP/gRPC middleware, This function owns backend construction only. HTTP/gRPC middleware,
service registration, and network serving remain Transport concerns. service registration, and network serving remain Transport concerns.
""" """
if get_parallel().dp_size > 1: if get_parallel().config.dp_size > 1:
raise ValueError( raise ValueError(
"launch_local_runtime requires --dp-size 1; got " "launch_local_runtime requires --dp-size 1; got "
f"dp_size={get_parallel().dp_size}." f"dp_size={get_parallel().config.dp_size}."
) )
# Set up prometheus metrics. # Set up prometheus metrics.
@@ -1512,8 +1511,10 @@ def launch_local_runtime(server_args: ServerArgs) -> EncoderRuntime:
zmq_context = zmq.Context(10) zmq_context = zmq.Context(10)
ipc_path_prefix = random_uuid() ipc_path_prefix = random_uuid()
port_args = PortArgs.init_new(server_args) port_args = PortArgs.init_new(server_args)
if get_parallel().dist_init_addr: if get_parallel().config.dist_init_addr:
dist_init_method = NetworkAddress.parse(get_parallel().dist_init_addr).to_tcp() dist_init_method = NetworkAddress.parse(
get_parallel().config.dist_init_addr
).to_tcp()
else: else:
dist_init_method = NetworkAddress( dist_init_method = NetworkAddress(
get_serving().host or "127.0.0.1", port_args.nccl_port get_serving().host or "127.0.0.1", port_args.nccl_port
@@ -1529,7 +1530,7 @@ def launch_local_runtime(server_args: ServerArgs) -> EncoderRuntime:
send_sockets: List[zmq.Socket] = [] send_sockets: List[zmq.Socket] = []
tp_processes: List[mp.Process] = [] tp_processes: List[mp.Process] = []
for rank in range(1, configured_tp_size()): for rank in range(1, get_parallel().config.tp_size):
schedule_path = f"ipc:///tmp/{ipc_path_prefix}_schedule_{rank}" schedule_path = f"ipc:///tmp/{ipc_path_prefix}_schedule_{rank}"
send_sockets.append( send_sockets.append(
get_zmq_socket(zmq_context, zmq.PUSH, schedule_path, bind=False) get_zmq_socket(zmq_context, zmq.PUSH, schedule_path, bind=False)
@@ -1569,12 +1570,12 @@ def launch_dp_runtime(server_args: ServerArgs) -> DPDispatcher:
HTTP uses this entry point today. gRPC can reuse it later without HTTP uses this entry point today. gRPC can reuse it later without
importing HTTP application state or Uvicorn. importing HTTP application state or Uvicorn.
""" """
if get_parallel().dp_size <= 1 or server_args.tp_size != 1: if get_parallel().config.dp_size <= 1 or server_args.tp_size != 1:
raise ValueError( raise ValueError(
"Encoder DP mode requires --dp-size > 1 and --tp-size 1; got " "Encoder DP mode requires --dp-size > 1 and --tp-size 1; got "
f"dp_size={get_parallel().dp_size}, tp_size={server_args.tp_size}." f"dp_size={get_parallel().config.dp_size}, tp_size={server_args.tp_size}."
) )
dp_size = get_parallel().dp_size dp_size = get_parallel().config.dp_size
logger.info(f"Launching encoder in DP mode: dp_size={dp_size}") logger.info(f"Launching encoder in DP mode: dp_size={dp_size}")
# DP mode: workers (subprocesses) write metrics to the shared multiproc dir; # DP mode: workers (subprocesses) write metrics to the shared multiproc dir;
python/sglang/srt/disaggregation/prefill.py
+1 -1
View File
@@ -180,7 +180,7 @@ class PrefillBootstrapQueue:
"SGLANG_DISAGG_STAGING_BUFFER with pp_size > 1 is only " "SGLANG_DISAGG_STAGING_BUFFER with pp_size > 1 is only "
"supported by Mooncake." "supported by Mooncake."
) )
if get_parallel().enable_prefill_context_parallel: if get_parallel().config.enable_prefill_context_parallel:
# CP rewrites index_slice per rank, breaking the chunk grid. # CP rewrites index_slice per rank, breaking the chunk grid.
raise RuntimeError( raise RuntimeError(
"SGLANG_DISAGG_STAGING_BUFFER does not support " "SGLANG_DISAGG_STAGING_BUFFER does not support "
python/sglang/srt/distributed/bootstrap.py
+2 -2
View File
@@ -128,7 +128,7 @@ def init_torch_distributed(
# included in later KV-cache sizing instead of appearing during capture. # included in later KV-cache sizing instead of appearing during capture.
if ( if (
device == "cuda" device == "cuda"
and get_parallel().enable_tp_lm_head_all_to_all and get_parallel().config.enable_tp_lm_head_all_to_all
and ps.tp_size > 1 and ps.tp_size > 1
): ):
_prewarm_tp_lm_head_all_to_all() _prewarm_tp_lm_head_all_to_all()
@@ -266,7 +266,7 @@ def _init_parallel_groups(
duplicate_attn_cp_group=( duplicate_attn_cp_group=(
is_hip() is_hip()
and server_args.enable_two_batch_overlap and server_args.enable_two_batch_overlap
and get_parallel().enable_dsa_prefill_context_parallel and get_parallel().config.enable_dsa_prefill_context_parallel
), ),
enable_symm_mem=get_exec().comm.enable_symm_mem, enable_symm_mem=get_exec().comm.enable_symm_mem,
recovered_rank=is_ep_joiner, recovered_rank=is_ep_joiner,
python/sglang/srt/distributed/device_communicators/triton_symm_mem_ag.py
+1 -1
View File
@@ -477,7 +477,7 @@ class MultimemAllGatherer:
# EP/mooncake setups, and keep multimem enabled. # EP/mooncake setups, and keep multimem enabled.
if ( if (
tp_group.world_size > 1 tp_group.world_size > 1
and get_parallel().nnodes > 1 and get_parallel().config.nnodes > 1
and not all(in_the_same_node_as(tp_group.cpu_group, source_rank=0)) and not all(in_the_same_node_as(tp_group.cpu_group, source_rank=0))
): ):
logger.warning( logger.warning(
python/sglang/srt/elastic_ep/elastic_ep.py
+7 -7
View File
@@ -12,7 +12,6 @@ from sglang.srt.distributed.utils import get_global_tcp_store
from sglang.srt.eplb.expert_location import broadcast_global_expert_location_metadata from sglang.srt.eplb.expert_location import broadcast_global_expert_location_metadata
from sglang.srt.managers.schedule_batch import ServerArgs from sglang.srt.managers.schedule_batch import ServerArgs
from sglang.srt.runtime_context import ( from sglang.srt.runtime_context import (
configured_tp_size,
get_exec, get_exec,
get_parallel, get_parallel,
) )
@@ -93,7 +92,7 @@ class ElasticEPStateManager:
if get_exec().moe.elastic_ep_backend is not None: if get_exec().moe.elastic_ep_backend is not None:
world_size = torch.distributed.get_world_size() world_size = torch.distributed.get_world_size()
active_rank_capacity = get_parallel().max_ep_size or world_size active_rank_capacity = get_parallel().config.max_ep_size or world_size
assert active_rank_capacity >= world_size, ( assert active_rank_capacity >= world_size, (
f"--max-ep-size ({active_rank_capacity}) must be >= " f"--max-ep-size ({active_rank_capacity}) must be >= "
f"world_size ({world_size})." f"world_size ({world_size})."
@@ -110,7 +109,7 @@ class ElasticEPStateManager:
if get_exec().moe.moe_a2a_backend == "nixl": if get_exec().moe.moe_a2a_backend == "nixl":
cls._on_scale = cls._on_scale_nixl cls._on_scale = cls._on_scale_nixl
inst.ep_join_rank_offset = get_parallel().ep_join_rank_offset inst.ep_join_rank_offset = get_parallel().config.ep_join_rank_offset
if server_args.is_ep_joiner: if server_args.is_ep_joiner:
cls._init_joiner_state(inst, server_args) cls._init_joiner_state(inst, server_args)
@@ -128,11 +127,12 @@ class ElasticEPStateManager:
if get_exec().moe.ep_join_mode == "scale": if get_exec().moe.ep_join_mode == "scale":
inst.effective_ep_size = ( inst.effective_ep_size = (
get_parallel().ep_join_rank_offset + configured_tp_size() get_parallel().config.ep_join_rank_offset
+ get_parallel().config.tp_size
) )
inst.original_ep_size = ( inst.original_ep_size = (
get_parallel().elastic_ep_initial_size get_parallel().config.elastic_ep_initial_size
or get_parallel().ep_join_rank_offset or get_parallel().config.ep_join_rank_offset
) )
inst.has_scaled = True inst.has_scaled = True
else: else:
@@ -317,7 +317,7 @@ def elastic_expanded_world_enabled() -> bool:
inst = ElasticEPStateManager.instance() inst = ElasticEPStateManager.instance()
if inst is None: if inst is None:
return False return False
if get_parallel().max_ep_size is None: if get_parallel().config.max_ep_size is None:
return False return False
active_target_size = inst.effective_ep_size active_target_size = inst.effective_ep_size
if inst.pending_ep_size is not None and inst.scale_phase in ( if inst.pending_ep_size is not None and inst.scale_phase in (
python/sglang/srt/elastic_ep/expert_backup_manager.py
+5 -6
View File
@@ -18,7 +18,6 @@ from sglang.srt.managers.io_struct import (
from sglang.srt.model_loader.loader import DefaultModelLoader, get_model_loader from sglang.srt.model_loader.loader import DefaultModelLoader, get_model_loader
from sglang.srt.model_loader.utils import set_default_torch_dtype from sglang.srt.model_loader.utils import set_default_torch_dtype
from sglang.srt.runtime_context import ( from sglang.srt.runtime_context import (
configured_tp_size,
get_disagg, get_disagg,
get_exec, get_exec,
get_model, get_model,
@@ -51,8 +50,8 @@ class ExpertBackupManager:
self.weight_pointer_map = {} self.weight_pointer_map = {}
self.transfer_engine = None self.transfer_engine = None
self.session_id = None self.session_id = None
self.engine_num = get_parallel().nnodes self.engine_num = get_parallel().config.nnodes
self.engine_rank = get_parallel().node_rank self.engine_rank = get_parallel().config.node_rank
self.expert_num = self.model_config.hf_config.n_routed_experts self.expert_num = self.model_config.hf_config.n_routed_experts
self.idmn = (self.expert_num // self.engine_num) * self.engine_rank self.idmn = (self.expert_num // self.engine_num) * self.engine_rank
self.idmx = (self.expert_num // self.engine_num) * (self.engine_rank + 1) self.idmx = (self.expert_num // self.engine_num) * (self.engine_rank + 1)
@@ -60,11 +59,11 @@ class ExpertBackupManager:
# Synchronization socket to avoid PUB/SUB slow joiner issues. # Synchronization socket to avoid PUB/SUB slow joiner issues.
self.recv_from_expert_backup_client = context.socket(zmq.PULL) self.recv_from_expert_backup_client = context.socket(zmq.PULL)
self.recv_from_expert_backup_client.bind( self.recv_from_expert_backup_client.bind(
f"tcp://{get_local_ip_auto()}:{PORT_BASE + get_parallel().node_rank * 2}" f"tcp://{get_local_ip_auto()}:{PORT_BASE + get_parallel().config.node_rank * 2}"
) )
self.send_to_expert_backup_client = context.socket(zmq.PUB) self.send_to_expert_backup_client = context.socket(zmq.PUB)
self.send_to_expert_backup_client.bind( self.send_to_expert_backup_client.bind(
f"tcp://{get_local_ip_auto()}:{PORT_BASE + get_parallel().node_rank * 2 + 1}" f"tcp://{get_local_ip_auto()}:{PORT_BASE + get_parallel().config.node_rank * 2 + 1}"
) )
self.backup_weights_from_disk() self.backup_weights_from_disk()
self.start_transfer_server() self.start_transfer_server()
@@ -73,7 +72,7 @@ class ExpertBackupManager:
# losing the initial PUB message due to slow joiners. # losing the initial PUB message due to slow joiners.
num_ready_clients = 0 num_ready_clients = 0
while num_ready_clients < configured_tp_size(): while num_ready_clients < get_parallel().config.tp_size:
sock_recv(self.recv_from_expert_backup_client) sock_recv(self.recv_from_expert_backup_client)
num_ready_clients += 1 num_ready_clients += 1
python/sglang/srt/entrypoints/engine.py
+12 -11
View File
@@ -98,9 +98,6 @@ from sglang.srt.parser.template_detection import resolve_auto_parsers
from sglang.srt.parser.template_manager import TemplateManager from sglang.srt.parser.template_manager import TemplateManager
from sglang.srt.plugins import load_plugins from sglang.srt.plugins import load_plugins
from sglang.srt.runtime_context import ( from sglang.srt.runtime_context import (
configured_attn_cp_size,
configured_moe_dp_size,
configured_pp_size,
get_exec, get_exec,
get_model, get_model,
get_parallel, get_parallel,
@@ -345,7 +342,7 @@ class Engine(EngineScoreMixin, EngineBase):
routed_dp_rank = data_parallel_rank routed_dp_rank = data_parallel_rank
if routed_dp_rank is not None: if routed_dp_rank is not None:
dp_size = get_parallel().dp_size dp_size = get_parallel().config.dp_size
if dp_size <= 1 and routed_dp_rank == 0: if dp_size <= 1 and routed_dp_rank == 0:
logger.debug( logger.debug(
f"routed_dp_rank={routed_dp_rank} is ignored because dp_size={dp_size}" f"routed_dp_rank={routed_dp_rank} is ignored because dp_size={dp_size}"
@@ -684,7 +681,7 @@ class Engine(EngineScoreMixin, EngineBase):
pp_rank_range, tp_rank_range, pp_size_per_node, tp_size_per_node = ( pp_rank_range, tp_rank_range, pp_size_per_node, tp_size_per_node = (
_calculate_rank_ranges( _calculate_rank_ranges(
server_args.nnodes, server_args.nnodes,
configured_pp_size(), get_parallel().config.pp_size,
tp_size, tp_size,
server_args.node_rank, server_args.node_rank,
) )
@@ -832,7 +829,7 @@ class Engine(EngineScoreMixin, EngineBase):
""" """
scheduler_procs = [] scheduler_procs = []
use_dp_controller = ( use_dp_controller = (
get_parallel().dp_size > 1 or get_exec().moe.ep_join_mode == "scale" get_parallel().config.dp_size > 1 or get_exec().moe.ep_join_mode == "scale"
) )
if not use_dp_controller: if not use_dp_controller:
@@ -845,7 +842,7 @@ class Engine(EngineScoreMixin, EngineBase):
pp_rank_range, tp_rank_range, pp_size_per_node, tp_size_per_node = ( pp_rank_range, tp_rank_range, pp_size_per_node, tp_size_per_node = (
_calculate_rank_ranges( _calculate_rank_ranges(
server_args.nnodes, server_args.nnodes,
configured_pp_size(), get_parallel().config.pp_size,
server_args.tp_size, server_args.tp_size,
server_args.node_rank, server_args.node_rank,
) )
@@ -1845,10 +1842,14 @@ def _compute_parallelism_ranks(
Called while the launcher is deciding what to spawn, so the sizes are the Called while the launcher is deciding what to spawn, so the sizes are the
configured ones -- the groups this is laying out do not exist yet. configured ones -- the groups this is laying out do not exist yet.
""" """
attn_dp_size = get_parallel().dp_size if get_parallel().enable_dp_attention else 1 attn_dp_size = (
get_parallel().config.dp_size
if get_parallel().config.enable_dp_attention
else 1
)
tp_size = server_args.tp_size tp_size = server_args.tp_size
attn_cp_size = configured_attn_cp_size() attn_cp_size = get_parallel().config.attn_cp_size
moe_dp_size = configured_moe_dp_size() moe_dp_size = get_parallel().config.moe_dp_size
# Parallelism hierarchy (outermost to innermost): # Parallelism hierarchy (outermost to innermost):
# - Attention: Global(TP) -> DP -> ATTN_CP -> ATTN_TP (innermost) # - Attention: Global(TP) -> DP -> ATTN_CP -> ATTN_TP (innermost)
@@ -1859,6 +1860,6 @@ def _compute_parallelism_ranks(
moe_ep_rank = ( moe_ep_rank = (
tp_rank tp_rank
% (tp_size // moe_dp_size) % (tp_size // moe_dp_size)
// (tp_size // moe_dp_size // get_parallel().ep_size) // (tp_size // moe_dp_size // get_parallel().config.ep_size)
) )
return attn_cp_rank, moe_dp_rank, moe_ep_rank return attn_cp_rank, moe_dp_rank, moe_ep_rank
python/sglang/srt/entrypoints/http_server.py
+10 -6
View File
@@ -2177,7 +2177,7 @@ async def _send_disaggregation_warmup_requests(
return await asyncio.gather( return await asyncio.gather(
*( *(
send_request(session, dp_rank) send_request(session, dp_rank)
for dp_rank in range(get_parallel().dp_size) for dp_rank in range(get_parallel().config.dp_size)
) )
) )
@@ -2236,9 +2236,11 @@ def _execute_server_warmup(server_args: ServerArgs):
}, },
} }
if server_args.skip_tokenizer_init: if server_args.skip_tokenizer_init:
json_data["input_ids"] = [[10, 11, 12] for _ in range(get_parallel().dp_size)] json_data["input_ids"] = [
[10, 11, 12] for _ in range(get_parallel().config.dp_size)
]
# TODO Workaround the bug that embedding errors for list of size 1 # TODO Workaround the bug that embedding errors for list of size 1
if get_parallel().dp_size == 1: if get_parallel().config.dp_size == 1:
json_data["input_ids"] = json_data["input_ids"][0] json_data["input_ids"] = json_data["input_ids"][0]
elif ( elif (
is_vlm is_vlm
@@ -2282,9 +2284,11 @@ def _execute_server_warmup(server_args: ServerArgs):
"temperature": 0.0, "temperature": 0.0,
} }
else: else:
json_data["text"] = ["The capital city of France is"] * get_parallel().dp_size json_data["text"] = [
"The capital city of France is"
] * get_parallel().config.dp_size
# TODO Workaround the bug that embedding errors for list of size 1 # TODO Workaround the bug that embedding errors for list of size 1
if get_parallel().dp_size == 1: if get_parallel().config.dp_size == 1:
json_data["text"] = json_data["text"][0] json_data["text"] = json_data["text"][0]
# Config debug dumping # Config debug dumping
@@ -2326,7 +2330,7 @@ def _execute_server_warmup(server_args: ServerArgs):
if not failed_status_codes: if not failed_status_codes:
logger.info( logger.info(
"Disaggregation warmup requests completed for all %s DP ranks", "Disaggregation warmup requests completed for all %s DP ranks",
get_parallel().dp_size, get_parallel().config.dp_size,
) )
logger.info("End of disaggregation warmup") logger.info("End of disaggregation warmup")
else: else:
python/sglang/srt/entrypoints/v1_loads.py
+3 -4
View File
@@ -27,7 +27,6 @@ from fastapi import APIRouter, Depends, HTTPException
from fastapi.responses import Response from fastapi.responses import Response
from sglang.srt.runtime_context import ( from sglang.srt.runtime_context import (
configured_pp_size,
get_parallel, get_parallel,
) )
from sglang.srt.utils import get_device_name from sglang.srt.utils import get_device_name
@@ -148,9 +147,9 @@ async def get_loads(
"accelerator": _accelerator_name(), "accelerator": _accelerator_name(),
"num_accelerators": _num_accelerators_per_dp_rank( "num_accelerators": _num_accelerators_per_dp_rank(
tokenizer_manager.server_args.tp_size, tokenizer_manager.server_args.tp_size,
configured_pp_size(), get_parallel().config.pp_size,
get_parallel().dp_size, get_parallel().config.dp_size,
get_parallel().enable_dp_attention, get_parallel().config.enable_dp_attention,
), ),
"loads": loads, "loads": loads,
} }
python/sglang/srt/eplb/expert_location.py
+9 -7
View File
@@ -26,7 +26,6 @@ import torch.distributed
import torch.nn.functional as F import torch.nn.functional as F
from sglang.srt.runtime_context import ( from sglang.srt.runtime_context import (
configured_tp_size,
get_device, get_device,
get_exec, get_exec,
) )
@@ -202,7 +201,7 @@ class ExpertLocationMetadata:
model_config_for_expert_location = common["model_config_for_expert_location"] model_config_for_expert_location = common["model_config_for_expert_location"]
num_physical_experts = common["num_physical_experts"] num_physical_experts = common["num_physical_experts"]
num_groups = model_config_for_expert_location.num_groups num_groups = model_config_for_expert_location.num_groups
num_nodes = 1 if use_flat_topology else get_parallel().nnodes num_nodes = 1 if use_flat_topology else get_parallel().config.nnodes
from sglang.srt.eplb import eplb_algorithms from sglang.srt.eplb import eplb_algorithms
@@ -245,14 +244,15 @@ class ExpertLocationMetadata:
+ get_exec().moe.ep_num_redundant_experts + get_exec().moe.ep_num_redundant_experts
) )
# elastic-EP scale-up rewrites ep_size on the published config # elastic-EP scale-up rewrites ep_size on the published config
ep_size = get_parallel().ep_size ep_size = get_parallel().config.ep_size
num_physical_experts = base_num_physical_experts num_physical_experts = base_num_physical_experts
initial_ep_size = get_parallel().elastic_ep_initial_size initial_ep_size = get_parallel().config.elastic_ep_initial_size
if initial_ep_size is not None: if initial_ep_size is not None:
if get_exec().moe.ep_join_mode == "scale": if get_exec().moe.ep_join_mode == "scale":
ep_size = max( ep_size = max(
ep_size, ep_size,
get_parallel().ep_join_rank_offset + configured_tp_size(), get_parallel().config.ep_join_rank_offset
+ get_parallel().config.tp_size,
) )
num_physical_experts, num_local_physical_experts = ( num_physical_experts, num_local_physical_experts = (
_compute_elastic_expert_layout( _compute_elastic_expert_layout(
@@ -580,7 +580,7 @@ def _compute_logical_to_all_physical_map(
num_local_gpu_physical_experts = num_physical_experts // ep_size num_local_gpu_physical_experts = num_physical_experts // ep_size
prefer_same_node = _prefer_same_node_experts() prefer_same_node = _prefer_same_node_experts()
num_gpus_per_node = ( num_gpus_per_node = (
get_parallel().ep_size // get_parallel().nnodes get_parallel().config.ep_size // get_parallel().config.nnodes
if prefer_same_node if prefer_same_node
else None else None
) )
@@ -644,7 +644,9 @@ def compute_logical_to_rank_dispatch_physical_map(
num_local_gpu_physical_experts = num_physical_experts // ep_size num_local_gpu_physical_experts = num_physical_experts // ep_size
prefer_same_node = _prefer_same_node_experts() prefer_same_node = _prefer_same_node_experts()
num_gpus_per_node = ( num_gpus_per_node = (
get_parallel().ep_size // get_parallel().nnodes if prefer_same_node else None get_parallel().config.ep_size // get_parallel().config.nnodes
if prefer_same_node
else None
) )
num_local_node_physical_experts = ( num_local_node_physical_experts = (
num_local_gpu_physical_experts * num_gpus_per_node num_local_gpu_physical_experts * num_gpus_per_node
python/sglang/srt/layers/attention/dsa/dsa_indexer.py
+1 -2
View File
@@ -40,7 +40,6 @@ from sglang.srt.model_executor.runner_backend_utils.tc_piecewise_cuda_graph impo
is_in_tc_piecewise_cuda_graph, is_in_tc_piecewise_cuda_graph,
) )
from sglang.srt.runtime_context import ( from sglang.srt.runtime_context import (
configured_pp_size,
get_device, get_device,
get_exec, get_exec,
get_parallel, get_parallel,
@@ -249,7 +248,7 @@ class Indexer(DSANPUIndexerMixin, BaseFusedOp):
if _is_cuda: if _is_cuda:
self.sm_count = deep_gemm.get_num_sms() self.sm_count = deep_gemm.get_num_sms()
self.half_device_sm_count = ceil_align(self.sm_count // 2, 8) self.half_device_sm_count = ceil_align(self.sm_count // 2, 8)
pp_size = configured_pp_size() pp_size = get_parallel().config.pp_size
self.logits_with_pp_recv = pp_size > 1 and not get_pp_group().is_last_rank self.logits_with_pp_recv = pp_size > 1 and not get_pp_group().is_last_rank
else: else:
self.logits_with_pp_recv = False self.logits_with_pp_recv = False
python/sglang/srt/layers/attention/dsa/utils.py
+3 -3
View File
@@ -106,7 +106,7 @@ def should_use_dsa_fused_topk(seed_dsa_topk_from_draft_extend: bool) -> bool:
def is_dsa_enable_prefill_cp(): def is_dsa_enable_prefill_cp():
if not envs.SGLANG_ENABLE_CP_V2.get(): if not envs.SGLANG_ENABLE_CP_V2.get():
return get_parallel().enable_dsa_prefill_context_parallel return get_parallel().config.enable_dsa_prefill_context_parallel
# Derive from the runtime CP topology + model arch rather than the legacy # Derive from the runtime CP topology + model arch rather than the legacy
# flag under CP-v2: DSA prefill CP is active when the CP group is on for a # flag under CP-v2: DSA prefill CP is active when the CP group is on for a
@@ -122,14 +122,14 @@ def is_dsa_enable_prefill_cp():
def is_dsa_prefill_cp_in_seq_split(): def is_dsa_prefill_cp_in_seq_split():
return ( return (
is_dsa_enable_prefill_cp() is_dsa_enable_prefill_cp()
and get_parallel().dsa_prefill_cp_mode == "in-seq-split" and get_parallel().config.dsa_prefill_cp_mode == "in-seq-split"
) )
def is_dsa_prefill_cp_round_robin_split(): def is_dsa_prefill_cp_round_robin_split():
return ( return (
is_dsa_enable_prefill_cp() is_dsa_enable_prefill_cp()
and get_parallel().dsa_prefill_cp_mode == "round-robin-split" and get_parallel().config.dsa_prefill_cp_mode == "round-robin-split"
) )
python/sglang/srt/layers/attention/flashattention_backend.py
+4 -1
View File
@@ -57,7 +57,10 @@ from sglang.kernels.ops.attention.flash_attention import (
def _should_disable_scheduler_metadata_precompute() -> bool: def _should_disable_scheduler_metadata_precompute() -> bool:
return bool(get_parallel().enable_prefill_cp or get_parallel().enable_dp_attention) return bool(
get_parallel().config.enable_prefill_cp
or get_parallel().config.enable_dp_attention
)
@dataclass @dataclass
python/sglang/srt/layers/communicator.py
+4 -4
View File
@@ -269,7 +269,7 @@ class AttnTpContext:
def init_context(self, q_lora_rank, is_dsa): def init_context(self, q_lora_rank, is_dsa):
self.is_dsa = is_dsa self.is_dsa = is_dsa
self.allow_input_scattered = ( self.allow_input_scattered = (
get_parallel().enable_attn_tp_input_scattered get_parallel().config.enable_attn_tp_input_scattered
and (_is_cuda or _is_npu) and (_is_cuda or _is_npu)
and q_lora_rank is not None and q_lora_rank is not None
and not is_dsa and not is_dsa
@@ -280,7 +280,7 @@ class AttnTpContext:
and not check_cuda_graph_backend(Phase.PREFILL, Backend.TC_PIECEWISE) and not check_cuda_graph_backend(Phase.PREFILL, Backend.TC_PIECEWISE)
and get_spec().speculative_algorithm != "EAGLE3" and get_spec().speculative_algorithm != "EAGLE3"
) )
if get_parallel().enable_attn_tp_input_scattered: if get_parallel().config.enable_attn_tp_input_scattered:
if not self.allow_input_scattered: if not self.allow_input_scattered:
logging.info( logging.info(
"attn_tp_input_scattered is not enabled while other conditions are not met" "attn_tp_input_scattered is not enabled while other conditions are not met"
@@ -438,11 +438,11 @@ class LayerScatterModes:
def enable_moe_dense_fully_dp(): def enable_moe_dense_fully_dp():
return get_parallel().moe_dense_tp_size == 1 return get_parallel().config.moe_dense_tp_size == 1
def enable_dwdp(): def enable_dwdp():
return get_parallel().dwdp_size > 1 return get_parallel().config.dwdp_size > 1
class LayerCommunicator: class LayerCommunicator:
python/sglang/srt/layers/cp/base.py
+1 -1
View File
@@ -283,7 +283,7 @@ def get_cp_strategy() -> Optional[ContextParallelStrategy]:
server_args = get_server_args() server_args = get_server_args()
except ValueError: except ValueError:
return None return None
if server_args is not None and get_parallel().enable_prefill_cp: if server_args is not None and get_parallel().config.enable_prefill_cp:
init_cp_strategy(server_args) init_cp_strategy(server_args)
return _STRATEGY return _STRATEGY
python/sglang/srt/layers/cp/cp_decode_attn_tp.py
+1 -1
View File
@@ -51,7 +51,7 @@ class CpDecodeAttnTpContext:
"""Slices replicated attention weights across CP ranks during decode.""" """Slices replicated attention weights across CP ranks during decode."""
def __init__(self): def __init__(self):
enable_attn_tp = get_parallel().enable_cp_decode_attn_tp enable_attn_tp = get_parallel().config.enable_cp_decode_attn_tp
if enable_attn_tp and get_parallel().attn_cp_size > 1: if enable_attn_tp and get_parallel().attn_cp_size > 1:
self.decode_tp_rank = get_parallel().attn_cp_rank self.decode_tp_rank = get_parallel().attn_cp_rank
python/sglang/srt/layers/dp_attention.py
+9 -8
View File
@@ -29,8 +29,6 @@ from sglang.srt.distributed.device_communicators.pynccl_allocator import (
use_symmetric_memory, use_symmetric_memory,
) )
from sglang.srt.runtime_context import ( from sglang.srt.runtime_context import (
configured_attn_cp_size,
configured_moe_dp_size,
get_device, get_device,
get_exec, get_exec,
get_flags, get_flags,
@@ -349,9 +347,9 @@ def initialize_dp_attention(
dp.max_len_with_idle = ( dp.max_len_with_idle = (
getattr(model_config.hf_config, "hybrid_override_pattern", None) is not None getattr(model_config.hf_config, "hybrid_override_pattern", None) is not None
) )
enable_dp_attention = get_parallel().enable_dp_attention enable_dp_attention = get_parallel().config.enable_dp_attention
dp_size = get_parallel().dp_size dp_size = get_parallel().config.dp_size
attn_cp_size = configured_attn_cp_size() attn_cp_size = get_parallel().config.attn_cp_size
dp.enabled = enable_dp_attention dp.enabled = enable_dp_attention
@@ -363,8 +361,11 @@ def initialize_dp_attention(
) )
_ATTN_DP_SIZE = dp_size if enable_dp_attention else 1 _ATTN_DP_SIZE = dp_size if enable_dp_attention else 1
if get_exec().moe.elastic_ep_backend is not None and get_parallel().max_ep_size: if (
_ATTN_DP_RANK = tp_rank + get_parallel().ep_join_rank_offset get_exec().moe.elastic_ep_backend is not None
and get_parallel().config.max_ep_size
):
_ATTN_DP_RANK = tp_rank + get_parallel().config.ep_join_rank_offset
if server_args.is_ep_scale_joiner: if server_args.is_ep_scale_joiner:
dp.joiner_skip_all_gather = True dp.joiner_skip_all_gather = True
@@ -1033,7 +1034,7 @@ def is_enable_moe_cp_allgather() -> bool:
(``parallel_state.py``), so the live sizes are equal and the comparison would (``parallel_state.py``), so the live sizes are equal and the comparison would
always be false. always be false.
""" """
return configured_attn_cp_size() > configured_moe_dp_size() return get_parallel().config.attn_cp_size > get_parallel().config.moe_dp_size
def moe_cp_all_gather_into_tensor(output: torch.Tensor, input: torch.Tensor): def moe_cp_all_gather_into_tensor(output: torch.Tensor, input: torch.Tensor):
python/sglang/srt/layers/logits_processor.py
+4 -2
View File
@@ -296,8 +296,10 @@ class LogitsProcessor(nn.Module):
self.config = config self.config = config
self.vocab_size = config.vocab_size self.vocab_size = config.vocab_size
self.logit_scale = logit_scale self.logit_scale = logit_scale
self.use_attn_tp_group = get_parallel().enable_dp_lm_head self.use_attn_tp_group = get_parallel().config.enable_dp_lm_head
self.use_tp_lm_head_all_to_all = get_parallel().enable_tp_lm_head_all_to_all self.use_tp_lm_head_all_to_all = (
get_parallel().config.enable_tp_lm_head_all_to_all
)
self.use_fp32_lm_head = get_exec().features.enable_fp32_lm_head self.use_fp32_lm_head = get_exec().features.enable_fp32_lm_head
if self.use_attn_tp_group: if self.use_attn_tp_group:
self.attn_tp_size = get_parallel().attn_tp_size self.attn_tp_size = get_parallel().attn_tp_size
python/sglang/srt/layers/moe/fused_moe_triton/layer.py
+2 -2
View File
@@ -293,10 +293,10 @@ class FusedMoE(torch.nn.Module):
self._num_global_routed = num_experts - num_shared_slots self._num_global_routed = num_experts - num_shared_slots
if get_exec().moe.ep_join_mode == "scale": if get_exec().moe.ep_join_mode == "scale":
storage_ep_size = get_parallel().elastic_ep_initial_size storage_ep_size = get_parallel().config.elastic_ep_initial_size
assert storage_ep_size is not None assert storage_ep_size is not None
self._expert_storage_rank = ( self._expert_storage_rank = (
get_parallel().ep_join_rank_offset + self.moe_ep_rank get_parallel().config.ep_join_rank_offset + self.moe_ep_rank
) )
else: else:
storage_ep_size = self.moe_ep_size storage_ep_size = self.moe_ep_size
python/sglang/srt/layers/moe/moe_runner/flashinfer_cutedsl.py
+1 -1
View File
@@ -337,7 +337,7 @@ def ensure_cutedsl_wrapper(layer: torch.nn.Module) -> None:
else: else:
# Standard allgather path: the MoE sees up to dp_size local forwards # Standard allgather path: the MoE sees up to dp_size local forwards
# gathered together, so scale the per-rank forward bound by dp_size. # gathered together, so scale the per-rank forward bound by dp_size.
max_num_tokens = get_parallel().dp_size * cutedsl_moe_max_num_tokens() max_num_tokens = get_parallel().config.dp_size * cutedsl_moe_max_num_tokens()
top_k = layer.top_k if layer.top_k is not None else layer.moe_runner_config.top_k top_k = layer.top_k if layer.top_k is not None else layer.moe_runner_config.top_k
# inference_mode(False) ensures the wrapper's pre-allocated CUDA-graph # inference_mode(False) ensures the wrapper's pre-allocated CUDA-graph
# buffers are normal tensors. This call typically happens inside # buffers are normal tensors. This call typically happens inside
python/sglang/srt/layers/moe/token_dispatcher/nixl.py
+2 -2
View File
@@ -135,7 +135,7 @@ class NixlEPBuffer:
offset = ElasticEPStateManager.get_ep_join_rank_offset() offset = ElasticEPStateManager.get_ep_join_rank_offset()
global_rank = rank + offset global_rank = rank + offset
max_ep_size = get_parallel().max_ep_size or world_size max_ep_size = get_parallel().config.max_ep_size or world_size
nixl_max_ranks = max_ep_size nixl_max_ranks = max_ep_size
num_rdma_bytes = 0 num_rdma_bytes = 0
@@ -233,7 +233,7 @@ class _NixlEPDispatcherImplBase:
) )
self._active_world_size = dist.get_world_size(group) self._active_world_size = dist.get_world_size(group)
_max_ep = get_parallel().max_ep_size or self._active_world_size _max_ep = get_parallel().config.max_ep_size or self._active_world_size
self._mask_buffer = ( self._mask_buffer = (
torch.zeros(_max_ep, dtype=torch.int32, device="cuda") torch.zeros(_max_ep, dtype=torch.int32, device="cuda")
if self.active_ranks is not None if self.active_ranks is not None
python/sglang/srt/layers/moe/token_dispatcher/pplx.py
+1 -1
View File
@@ -155,7 +155,7 @@ class PplxAllToAllManager:
# pplx forces ep_size == world_size # pplx forces ep_size == world_size
# with pp_size == 1 (enforced in _ensure_nvshmem), so the EP group spans # with pp_size == 1 (enforced in _ensure_nvshmem), so the EP group spans
# a single node iff the whole job runs on one node. # a single node iff the whole job runs on one node.
is_internode = get_parallel().nnodes > 1 is_internode = get_parallel().config.nnodes > 1
if is_internode: if is_internode:
cls._all_to_all = AllToAll.internode( cls._all_to_all = AllToAll.internode(
python/sglang/srt/layers/moe/utils.py
+1 -1
View File
@@ -607,7 +607,7 @@ def should_skip_post_experts_all_reduce(*, is_tp_path: bool) -> bool:
""" """
if should_skip_mlp_all_reduce(): if should_skip_mlp_all_reduce():
return True return True
if get_parallel().dwdp_size > 1: if get_parallel().config.dwdp_size > 1:
return True return True
if should_use_dp_reduce_scatterv(): if should_use_dp_reduce_scatterv():
return True return True
python/sglang/srt/layers/utils/cp_utils.py
+3 -3
View File
@@ -63,18 +63,18 @@ class ContextParallelMetadata:
def is_prefill_context_parallel_enabled(): def is_prefill_context_parallel_enabled():
return get_parallel().enable_prefill_context_parallel return get_parallel().config.enable_prefill_context_parallel
def is_prefill_cp_in_seq_split(): def is_prefill_cp_in_seq_split():
return ( return (
is_prefill_context_parallel_enabled() is_prefill_context_parallel_enabled()
and get_parallel().prefill_cp_mode == "in-seq-split" and get_parallel().config.prefill_cp_mode == "in-seq-split"
) )
def is_mla_prefill_cp_enabled() -> bool: def is_mla_prefill_cp_enabled() -> bool:
return get_parallel().enable_prefill_context_parallel and uses_mla_backend() return get_parallel().config.enable_prefill_context_parallel and uses_mla_backend()
def mla_use_prefill_cp(forward_batch, mla_enable_prefill_cp=None): def mla_use_prefill_cp(forward_batch, mla_enable_prefill_cp=None):
python/sglang/srt/lora/lora_manager.py
+1 -1
View File
@@ -99,7 +99,7 @@ class LoRAManager:
self.pending_lora_load_events = {} self.pending_lora_load_events = {}
self.eviction_policy = server_args.lora_eviction_policy self.eviction_policy = server_args.lora_eviction_policy
self.enable_dp_attention: bool = get_parallel().enable_dp_attention self.enable_dp_attention: bool = get_parallel().config.enable_dp_attention
self._experts_shared_outer_override: Optional[bool] = ( self._experts_shared_outer_override: Optional[bool] = (
server_args.experts_shared_outer_loras server_args.experts_shared_outer_loras
) )
python/sglang/srt/managers/data_parallel_controller.py
+34 -27
View File
@@ -50,9 +50,6 @@ from sglang.srt.observability.req_time_stats import DPControllerReqTimeStats
from sglang.srt.observability.startup_time import aggregate_scheduler_startup_times from sglang.srt.observability.startup_time import aggregate_scheduler_startup_times
from sglang.srt.observability.trace import process_tracing_init, trace_set_thread_info from sglang.srt.observability.trace import process_tracing_init, trace_set_thread_info
from sglang.srt.runtime_context import ( from sglang.srt.runtime_context import (
configured_attn_cp_size,
configured_moe_dp_size,
configured_pp_size,
get_device, get_device,
get_disagg, get_disagg,
get_exec, get_exec,
@@ -151,12 +148,12 @@ class DataParallelController:
self.server_args = server_args self.server_args = server_args
self.port_args = port_args self.port_args = port_args
self.load_balance_method = LoadBalanceMethod.from_str( self.load_balance_method = LoadBalanceMethod.from_str(
get_parallel().load_balance_method get_parallel().config.load_balance_method
) )
self.run_scheduler_process_func = run_scheduler_process_func self.run_scheduler_process_func = run_scheduler_process_func
# Init inter-process communication # Init inter-process communication
self.context = zmq.Context(1 + get_parallel().dp_size) self.context = zmq.Context(1 + get_parallel().config.dp_size)
if server_args.node_rank == 0: if server_args.node_rank == 0:
self.recv_from_tokenizer = get_zmq_socket( self.recv_from_tokenizer = get_zmq_socket(
self.context, zmq.PULL, port_args.scheduler_input_ipc_name, False self.context, zmq.PULL, port_args.scheduler_input_ipc_name, False
@@ -176,8 +173,8 @@ class DataParallelController:
LoadBalanceMethod.TOTAL_TOKENS, LoadBalanceMethod.TOTAL_TOKENS,
) )
self.launch_dp_size: int = get_parallel().dp_size self.launch_dp_size: int = get_parallel().config.dp_size
self.max_dp_size: int = server_args.max_ep_size or get_parallel().dp_size self.max_dp_size: int = server_args.max_ep_size or get_parallel().config.dp_size
assert self.max_dp_size >= self.launch_dp_size, ( assert self.max_dp_size >= self.launch_dp_size, (
f"--max-ep-size ({self.max_dp_size}) must be >= " f"--max-ep-size ({self.max_dp_size}) must be >= "
f"--dp ({self.launch_dp_size})." f"--dp ({self.launch_dp_size})."
@@ -187,7 +184,7 @@ class DataParallelController:
self.max_dp_size - self.launch_dp_size self.max_dp_size - self.launch_dp_size
) )
self.dp_budget = DPBudget(get_parallel().dp_size) self.dp_budget = DPBudget(get_parallel().config.dp_size)
self.load_snapshot_reader = create_load_snapshot_reader( self.load_snapshot_reader = create_load_snapshot_reader(
port_args, port_args,
caller="DataParallelController", caller="DataParallelController",
@@ -204,14 +201,16 @@ class DataParallelController:
self._active_workers: List[int] = list(range(self.launch_dp_size)) self._active_workers: List[int] = list(range(self.launch_dp_size))
self._active_count_cache: int = self.launch_dp_size self._active_count_cache: int = self.launch_dp_size
if get_parallel().enable_dp_attention: if get_parallel().config.enable_dp_attention:
self.launch_dp_attention_schedulers(server_args, port_args) self.launch_dp_attention_schedulers(server_args, port_args)
# When local control broadcast is enabled, send control messages to # When local control broadcast is enabled, send control messages to
# every DP group leader (attn_tp_rank=0) so each leader broadcasts # every DP group leader (attn_tp_rank=0) so each leader broadcasts
# within its own attn_tp_group instead of the full tp_group. # within its own attn_tp_group instead of the full tp_group.
# Otherwise fall back to the original behaviour: send to only the # Otherwise fall back to the original behaviour: send to only the
# first leader, which then broadcasts over the full tp_group. # first leader, which then broadcasts over the full tp_group.
local_ctrl = get_parallel().enable_dp_attention_local_control_broadcast local_ctrl = (
get_parallel().config.enable_dp_attention_local_control_broadcast
)
self.control_message_step = 1 if local_ctrl else server_args.tp_size self.control_message_step = 1 if local_ctrl else server_args.tp_size
else: else:
self.launch_dp_schedulers(server_args, port_args) self.launch_dp_schedulers(server_args, port_args)
@@ -375,7 +374,7 @@ class DataParallelController:
threads = [] threads = []
sockets = [] sockets = []
ready_events = [] ready_events = []
for dp_rank in range(get_parallel().dp_size): for dp_rank in range(get_parallel().config.dp_size):
tmp_port_args = PortArgs.init_new(server_args) tmp_port_args = PortArgs.init_new(server_args)
tmp_port_args.tokenizer_ipc_name = port_args.tokenizer_ipc_name tmp_port_args.tokenizer_ipc_name = port_args.tokenizer_ipc_name
tmp_port_args.detokenizer_ipc_name = port_args.detokenizer_ipc_name tmp_port_args.detokenizer_ipc_name = port_args.detokenizer_ipc_name
@@ -395,7 +394,9 @@ class DataParallelController:
) )
threads.append(thread) threads.append(thread)
base_gpu_id += ( base_gpu_id += (
server_args.tp_size * configured_pp_size() * server_args.gpu_id_step server_args.tp_size
* get_parallel().config.pp_size
* server_args.gpu_id_step
) )
if server_args.node_rank == 0: if server_args.node_rank == 0:
@@ -577,7 +578,7 @@ class DataParallelController:
bind_count = ( bind_count = (
self.max_dp_size self.max_dp_size
if server_args.elastic_ep_backend is not None if server_args.elastic_ep_backend is not None
else get_parallel().dp_size else get_parallel().config.dp_size
) )
for slot in range(bind_count): for slot in range(bind_count):
worker_port, worker_socket = get_zmq_socket_on_host( worker_port, worker_socket = get_zmq_socket_on_host(
@@ -607,7 +608,7 @@ class DataParallelController:
dp_rank: Optional[int], dp_rank: Optional[int],
worker_ports: Optional[List[int]] = None, worker_ports: Optional[List[int]] = None,
): ):
if not get_parallel().enable_dp_attention: if not get_parallel().config.enable_dp_attention:
logger.info(f"Launch DP{dp_rank} starting at GPU #{base_gpu_id}.") logger.info(f"Launch DP{dp_rank} starting at GPU #{base_gpu_id}.")
memory_saver_adapter = TorchMemorySaverAdapter.create( memory_saver_adapter = TorchMemorySaverAdapter.create(
@@ -616,8 +617,8 @@ class DataParallelController:
scheduler_pipe_readers = [] scheduler_pipe_readers = []
pp_size_per_node = max(configured_pp_size() // server_args.nnodes, 1) pp_size_per_node = max(get_parallel().config.pp_size // server_args.nnodes, 1)
nnodes_per_pp_rank = max(server_args.nnodes // configured_pp_size(), 1) nnodes_per_pp_rank = max(server_args.nnodes // get_parallel().config.pp_size, 1)
pp_rank_range = range( pp_rank_range = range(
pp_size_per_node * (server_args.node_rank // nnodes_per_pp_rank), pp_size_per_node * (server_args.node_rank // nnodes_per_pp_rank),
pp_size_per_node * (server_args.node_rank // nnodes_per_pp_rank + 1), pp_size_per_node * (server_args.node_rank // nnodes_per_pp_rank + 1),
@@ -641,14 +642,14 @@ class DataParallelController:
for tp_rank in tp_rank_range: for tp_rank in tp_rank_range:
rank_port_args = port_args rank_port_args = port_args
if get_parallel().enable_dp_attention: if get_parallel().config.enable_dp_attention:
# dp attention has different sharding logic # dp attention has different sharding logic
_, _, dp_rank, _ = compute_dp_attention_world_info( _, _, dp_rank, _ = compute_dp_attention_world_info(
get_parallel().enable_dp_attention, get_parallel().config.enable_dp_attention,
tp_rank, tp_rank,
server_args.tp_size, server_args.tp_size,
get_parallel().dp_size, get_parallel().config.dp_size,
configured_attn_cp_size(), get_parallel().config.attn_cp_size,
) )
# compute zmq ports for this dp rank # compute zmq ports for this dp rank
rank_port_args = PortArgs.init_new( rank_port_args = PortArgs.init_new(
@@ -677,26 +678,32 @@ class DataParallelController:
+ (tp_rank % tp_size_per_node) * server_args.gpu_id_step + (tp_rank % tp_size_per_node) * server_args.gpu_id_step
) )
attn_dp_size = ( attn_dp_size = (
get_parallel().dp_size if get_parallel().enable_dp_attention else 1 get_parallel().config.dp_size
if get_parallel().config.enable_dp_attention
else 1
) )
# Parallelism hierarchy (outermost to innermost): # Parallelism hierarchy (outermost to innermost):
# - Attention: Global(TP) -> DP -> ATTN_CP -> ATTN_TP (innermost) # - Attention: Global(TP) -> DP -> ATTN_CP -> ATTN_TP (innermost)
# - MoE: Global(TP) -> MOE_DP -> EP -> MOE_TP (innermost) # - MoE: Global(TP) -> MOE_DP -> EP -> MOE_TP (innermost)
attn_tp_size = ( attn_tp_size = (
server_args.tp_size // attn_dp_size // configured_attn_cp_size() server_args.tp_size
// attn_dp_size
// get_parallel().config.attn_cp_size
) )
attn_cp_rank = (tp_rank // attn_tp_size) % configured_attn_cp_size() attn_cp_rank = (
tp_rank // attn_tp_size
) % get_parallel().config.attn_cp_size
moe_dp_rank = tp_rank // ( moe_dp_rank = tp_rank // (
server_args.tp_size // configured_moe_dp_size() server_args.tp_size // get_parallel().config.moe_dp_size
) )
moe_ep_rank = ( moe_ep_rank = (
tp_rank tp_rank
% (server_args.tp_size // configured_moe_dp_size()) % (server_args.tp_size // get_parallel().config.moe_dp_size)
// ( // (
server_args.tp_size server_args.tp_size
// configured_moe_dp_size() // get_parallel().config.moe_dp_size
// get_parallel().ep_size // get_parallel().config.ep_size
) )
) )
python/sglang/srt/managers/disagg_service.py
+2 -1
View File
@@ -49,7 +49,8 @@ def maybe_create_ascend_config_store(
which the rust registry ports verbatim), leaving this store as the only which the rust registry ports verbatim), leaving this store as the only
``start_disagg_service`` duty left to perform.""" ``start_disagg_service`` duty left to perform."""
if not ( if not (
get_parallel().node_rank == 0 and transfer_backend == TransferBackend.ASCEND get_parallel().config.node_rank == 0
and transfer_backend == TransferBackend.ASCEND
): ):
return return
try: try:
python/sglang/srt/managers/load_snapshot.py
+7 -7
View File
@@ -71,7 +71,7 @@ def should_use_zmq() -> bool:
``SGLANG_LOAD_SNAPSHOT_USE_ZMQ`` forces zmq mode for testing. ``SGLANG_LOAD_SNAPSHOT_USE_ZMQ`` forces zmq mode for testing.
""" """
return ( return (
get_parallel().enable_dp_attention and get_parallel().nnodes > 1 get_parallel().config.enable_dp_attention and get_parallel().config.nnodes > 1
) or envs.SGLANG_LOAD_SNAPSHOT_USE_ZMQ.get() ) or envs.SGLANG_LOAD_SNAPSHOT_USE_ZMQ.get()
@@ -116,15 +116,15 @@ def zmq_reader_owner(caller: str) -> bool:
""" """
if not should_use_zmq(): if not should_use_zmq():
return False return False
if get_parallel().node_rank != 0: if get_parallel().config.node_rank != 0:
return False return False
if caller == "DataParallelController": if caller == "DataParallelController":
return ( return (
get_parallel().dp_size > 1 get_parallel().config.dp_size > 1
and get_parallel().load_balance_method.lower() in _LOAD_AWARE_METHODS and get_parallel().config.load_balance_method.lower() in _LOAD_AWARE_METHODS
) )
if get_parallel().dp_size > 1 and ( if get_parallel().config.dp_size > 1 and (
get_parallel().load_balance_method.lower() in _LOAD_AWARE_METHODS get_parallel().config.load_balance_method.lower() in _LOAD_AWARE_METHODS
): ):
return False return False
return caller == _tokenizer_load_snapshot_owner_caller() return caller == _tokenizer_load_snapshot_owner_caller()
@@ -658,7 +658,7 @@ def create_load_snapshot_reader(port_args, caller: str):
``"MultiTokenizerRouter"`` -- determines who binds the zmq PULL ``"MultiTokenizerRouter"`` -- determines who binds the zmq PULL
socket when zmq mode is active. socket when zmq mode is active.
""" """
dp_size = get_parallel().dp_size dp_size = get_parallel().config.dp_size
if zmq_reader_owner(caller): if zmq_reader_owner(caller):
return ZmqShmLoadSnapshotReader( return ZmqShmLoadSnapshotReader(
_zmq_addr_for(port_args), shm_path_for(port_args.instance_id), dp_size _zmq_addr_for(port_args), shm_path_for(port_args.instance_id), dp_size
python/sglang/srt/managers/prefill_delayer.py
+1 -1
View File
@@ -108,7 +108,7 @@ class PrefillDelayer:
f"queue_trigger_enabled={self._queue_trigger_enabled}" f"queue_trigger_enabled={self._queue_trigger_enabled}"
) )
self.dp_size = dp_size self.dp_size = dp_size
self.enable_dp_attention = get_parallel().enable_dp_attention self.enable_dp_attention = get_parallel().config.enable_dp_attention
dp_size_dim = dp_size if self.enable_dp_attention else 1 dp_size_dim = dp_size if self.enable_dp_attention else 1
# Mirror scheduler_dp_attn_mixin's NCCL all-gather path: when the # Mirror scheduler_dp_attn_mixin's NCCL all-gather path: when the
python/sglang/srt/managers/scheduler.py
+29 -31
View File
@@ -29,11 +29,6 @@ from typing import TYPE_CHECKING, Any, Deque, Dict, List, Optional, Set, Tuple,
from sglang.srt.runtime_context import ( from sglang.srt.runtime_context import (
attention_backends, attention_backends,
configured_attn_cp_size,
configured_dcp_size,
configured_moe_dp_size,
configured_pp_size,
configured_tp_size,
get_device, get_device,
get_disagg, get_disagg,
get_exec, get_exec,
@@ -463,38 +458,38 @@ class Scheduler(
self.max_recv_per_poll = envs.SGLANG_SCHEDULER_MAX_RECV_PER_POLL.get() self.max_recv_per_poll = envs.SGLANG_SCHEDULER_MAX_RECV_PER_POLL.get()
self.max_new_tokens_limit = envs.SGLANG_MAX_NEW_TOKENS_LIMIT.get() self.max_new_tokens_limit = envs.SGLANG_MAX_NEW_TOKENS_LIMIT.get()
self.enable_hisparse = get_memory().enable_hisparse self.enable_hisparse = get_memory().enable_hisparse
self.enable_dp_attention = get_parallel().enable_dp_attention self.enable_dp_attention = get_parallel().config.enable_dp_attention
self.enable_unified_memory = get_memory().enable_unified_memory self.enable_unified_memory = get_memory().enable_unified_memory
# Distributed rank info # Distributed rank info
attn_tp_rank, attn_tp_size, attn_dp_rank, attn_dp_size = ( attn_tp_rank, attn_tp_size, attn_dp_rank, attn_dp_size = (
compute_dp_attention_world_info( compute_dp_attention_world_info(
get_parallel().enable_dp_attention, get_parallel().config.enable_dp_attention,
tp_rank, tp_rank,
configured_tp_size(), get_parallel().config.tp_size,
get_parallel().dp_size, get_parallel().config.dp_size,
configured_attn_cp_size(), get_parallel().config.attn_cp_size,
) )
) )
self.ps = ParallelState( self.ps = ParallelState(
tp_rank=tp_rank, tp_rank=tp_rank,
tp_size=configured_tp_size(), tp_size=get_parallel().config.tp_size,
pp_rank=pp_rank, pp_rank=pp_rank,
pp_size=configured_pp_size(), pp_size=get_parallel().config.pp_size,
dp_rank=dp_rank, dp_rank=dp_rank,
dp_size=get_parallel().dp_size, dp_size=get_parallel().config.dp_size,
attn_tp_rank=attn_tp_rank, attn_tp_rank=attn_tp_rank,
attn_tp_size=attn_tp_size, attn_tp_size=attn_tp_size,
attn_cp_rank=attn_cp_rank, attn_cp_rank=attn_cp_rank,
attn_cp_size=configured_attn_cp_size(), attn_cp_size=get_parallel().config.attn_cp_size,
attn_dcp_rank=tp_rank % configured_dcp_size(), attn_dcp_rank=tp_rank % get_parallel().config.dcp_size,
attn_dcp_size=configured_dcp_size(), attn_dcp_size=get_parallel().config.dcp_size,
attn_dp_rank=attn_dp_rank, attn_dp_rank=attn_dp_rank,
attn_dp_size=attn_dp_size, attn_dp_size=attn_dp_size,
moe_ep_rank=moe_ep_rank, moe_ep_rank=moe_ep_rank,
moe_ep_size=get_parallel().ep_size, moe_ep_size=get_parallel().config.ep_size,
moe_dp_rank=moe_dp_rank, moe_dp_rank=moe_dp_rank,
moe_dp_size=configured_moe_dp_size(), moe_dp_size=get_parallel().config.moe_dp_size,
gpu_id=gpu_id, gpu_id=gpu_id,
) )
@@ -1069,7 +1064,7 @@ class Scheduler(
self.min_free_slots_delayer = MinFreeSlotsDelayer( self.min_free_slots_delayer = MinFreeSlotsDelayer(
min_free_slots=min_free_slots min_free_slots=min_free_slots
) )
if not get_parallel().pp_max_micro_batch_size: if not get_parallel().config.pp_max_micro_batch_size:
get_context().override( get_context().override(
"scheduler.pp_max_micro_batch_size_default", "scheduler.pp_max_micro_batch_size_default",
pp_max_micro_batch_size=max( pp_max_micro_batch_size=max(
@@ -1414,7 +1409,7 @@ class Scheduler(
gloo_group=self.attn_tp_cpu_group, gloo_group=self.attn_tp_cpu_group,
tp_rank=self.ps.tp_rank, tp_rank=self.ps.tp_rank,
tp_size=self.ps.tp_size, tp_size=self.ps.tp_size,
dp_size=get_parallel().dp_size, dp_size=get_parallel().config.dp_size,
gpu_id=self.ps.gpu_id, gpu_id=self.ps.gpu_id,
bootstrap_port=get_disagg().disaggregation_bootstrap_port, bootstrap_port=get_disagg().disaggregation_bootstrap_port,
max_total_num_tokens=self.max_total_num_tokens, max_total_num_tokens=self.max_total_num_tokens,
@@ -3233,7 +3228,7 @@ class Scheduler(
return NextBatchPlan(batch_to_run=ret, running_batch=running_batch) return NextBatchPlan(batch_to_run=ret, running_batch=running_batch)
def get_num_allocatable_reqs(self, running_bs): def get_num_allocatable_reqs(self, running_bs):
res = get_parallel().pp_max_micro_batch_size - running_bs res = get_parallel().config.pp_max_micro_batch_size - running_bs
res = min(res, self.req_to_token_pool.available_size()) res = min(res, self.req_to_token_pool.available_size())
return res return res
@@ -4890,7 +4885,7 @@ class Scheduler(
old_ep_size = ElasticEPStateManager.get_effective_ep_size() old_ep_size = ElasticEPStateManager.get_effective_ep_size()
new_ep_size = recv_req.new_ep_size new_ep_size = recv_req.new_ep_size
max_ep_size = get_parallel().max_ep_size or old_ep_size max_ep_size = get_parallel().config.max_ep_size or old_ep_size
logger.debug( logger.debug(
"[Elastic EP][scale] request received: new_ep_size=%d " "[Elastic EP][scale] request received: new_ep_size=%d "
@@ -5095,7 +5090,7 @@ def dispatch_event_loop(scheduler: Scheduler):
if disaggregation_mode == DisaggregationMode.NULL: if disaggregation_mode == DisaggregationMode.NULL:
if scheduler.enable_pdmux: if scheduler.enable_pdmux:
scheduler.event_loop_pdmux() scheduler.event_loop_pdmux()
elif configured_pp_size() > 1: elif get_parallel().config.pp_size > 1:
scheduler.event_loop_pp() scheduler.event_loop_pp()
elif scheduler.enable_overlap_mlx: elif scheduler.enable_overlap_mlx:
scheduler.event_loop_overlap_mlx() scheduler.event_loop_overlap_mlx()
@@ -5104,14 +5099,14 @@ def dispatch_event_loop(scheduler: Scheduler):
else: else:
scheduler.event_loop_normal() scheduler.event_loop_normal()
elif disaggregation_mode == DisaggregationMode.PREFILL: elif disaggregation_mode == DisaggregationMode.PREFILL:
if configured_pp_size() > 1: if get_parallel().config.pp_size > 1:
scheduler.event_loop_pp_disagg_prefill() scheduler.event_loop_pp_disagg_prefill()
elif scheduler.enable_overlap: elif scheduler.enable_overlap:
scheduler.event_loop_overlap_disagg_prefill() scheduler.event_loop_overlap_disagg_prefill()
else: else:
scheduler.event_loop_normal_disagg_prefill() scheduler.event_loop_normal_disagg_prefill()
elif disaggregation_mode == DisaggregationMode.DECODE: elif disaggregation_mode == DisaggregationMode.DECODE:
if configured_pp_size() > 1: if get_parallel().config.pp_size > 1:
scheduler.event_loop_pp_disagg_decode() scheduler.event_loop_pp_disagg_decode()
elif scheduler.enable_overlap: elif scheduler.enable_overlap:
scheduler.event_loop_overlap_disagg_decode() scheduler.event_loop_overlap_disagg_decode()
@@ -5152,15 +5147,15 @@ def configure_scheduler_process(
prefix = "" prefix = ""
if shown_dp is not None: if shown_dp is not None:
prefix += f" DP{shown_dp}" prefix += f" DP{shown_dp}"
if configured_pp_size() > 1: if get_parallel().config.pp_size > 1:
prefix += f" PP{pp_rank}" prefix += f" PP{pp_rank}"
if configured_attn_cp_size() > 1: if get_parallel().config.attn_cp_size > 1:
prefix += f" ATTN_CP{attn_cp_rank}" prefix += f" ATTN_CP{attn_cp_rank}"
if configured_moe_dp_size() > 1: if get_parallel().config.moe_dp_size > 1:
prefix += f" MOE_DP{moe_dp_rank}" prefix += f" MOE_DP{moe_dp_rank}"
if configured_tp_size() > 1: if get_parallel().config.tp_size > 1:
prefix += f" TP{shown_tp}" prefix += f" TP{shown_tp}"
if get_parallel().ep_size > 1: if get_parallel().config.ep_size > 1:
prefix += f" EP{shown_moe_ep}" prefix += f" EP{shown_moe_ep}"
# Config the process # Config the process
@@ -5174,7 +5169,10 @@ def configure_scheduler_process(
# Set cpu affinity to this gpu process # Set cpu affinity to this gpu process
if envs.SGLANG_SET_CPU_AFFINITY.get(): if envs.SGLANG_SET_CPU_AFFINITY.get():
set_gpu_proc_affinity( set_gpu_proc_affinity(
configured_pp_size(), configured_tp_size(), get_parallel().nnodes, gpu_id get_parallel().config.pp_size,
get_parallel().config.tp_size,
get_parallel().config.nnodes,
gpu_id,
) )
if not envs.SGLANG_NUMA_BIND_V2.get(): if not envs.SGLANG_NUMA_BIND_V2.get():
numa_node = get_numa_node_if_available(server_args, gpu_id) numa_node = get_numa_node_if_available(server_args, gpu_id)
python/sglang/srt/managers/scheduler_components/dp_attn.py
+2 -2
View File
@@ -411,7 +411,7 @@ class SchedulerDPAttnAdapter:
return prepare_mlp_sync_batch_raw( return prepare_mlp_sync_batch_raw(
local_batch, local_batch,
model_runner=self.model_runner, model_runner=self.model_runner,
dp_size=get_parallel().dp_size, dp_size=get_parallel().config.dp_size,
attn_tp_size=self.ps.attn_tp_size, attn_tp_size=self.ps.attn_tp_size,
attn_cp_size=self.ps.attn_cp_size, attn_cp_size=self.ps.attn_cp_size,
tp_group=self.tp_group, tp_group=self.tp_group,
@@ -420,7 +420,7 @@ class SchedulerDPAttnAdapter:
require_mlp_tp_gather=require_mlp_tp_gather(self.server_args), require_mlp_tp_gather=require_mlp_tp_gather(self.server_args),
disable_overlap_schedule=get_schedule().disable_overlap_schedule, disable_overlap_schedule=get_schedule().disable_overlap_schedule,
offload_tags=self.offload_tags, offload_tags=self.offload_tags,
dwdp=get_parallel().dwdp_size > 1, dwdp=get_parallel().config.dwdp_size > 1,
) )
def maybe_prepare_mlp_sync_batch( def maybe_prepare_mlp_sync_batch(
python/sglang/srt/managers/scheduler_components/metrics_reporter.py
+2 -2
View File
@@ -23,10 +23,10 @@ from sglang.srt.observability.metrics_collector import (
compute_routing_key_stats, compute_routing_key_stats,
) )
from sglang.srt.runtime_context import ( from sglang.srt.runtime_context import (
configured_pp_size,
get_context, get_context,
get_disagg, get_disagg,
get_observability, get_observability,
get_parallel,
get_spec, get_spec,
) )
from sglang.srt.utils.device_timer import DeviceTimer from sglang.srt.utils.device_timer import DeviceTimer
@@ -1117,7 +1117,7 @@ class SchedulerMetricsReporter:
active_lora_ids = set() active_lora_ids = set()
# For PP mode, check all running micro batches # For PP mode, check all running micro batches
if configured_pp_size() > 1: if get_parallel().config.pp_size > 1:
for batch in self.scheduler.running_mbs: for batch in self.scheduler.running_mbs:
if batch and hasattr(batch, "reqs"): if batch and hasattr(batch, "reqs"):
for req in batch.reqs: for req in batch.reqs:
python/sglang/srt/managers/scheduler_components/recv_skipper.py
+1 -1
View File
@@ -34,7 +34,7 @@ class SchedulerRecvSkipper:
return ForwardMode.DECODE return ForwardMode.DECODE
def __init__(self): def __init__(self):
self._use_synced_mode = get_parallel().enable_dp_attention self._use_synced_mode = get_parallel().config.enable_dp_attention
self._counter = 0 self._counter = 0
self._threshold = get_schedule().scheduler_recv_interval self._threshold = get_schedule().scheduler_recv_interval
# All can be tuned if needed # All can be tuned if needed
python/sglang/srt/managers/scheduler_components/request_receiver.py
+3 -3
View File
@@ -151,7 +151,7 @@ class SchedulerRequestReceiver:
return recv_reqs return recv_reqs
def _broadcast_reqs_across_ranks(self, recv_reqs: Optional[List]) -> List: def _broadcast_reqs_across_ranks(self, recv_reqs: Optional[List]) -> List:
if get_parallel().enable_dp_attention: if get_parallel().config.enable_dp_attention:
if self.ps.attn_tp_rank == 0 and self.ps.attn_cp_rank == 0: if self.ps.attn_tp_rank == 0 and self.ps.attn_cp_rank == 0:
work_reqs, control_reqs = self._split_work_and_control_reqs(recv_reqs) work_reqs, control_reqs = self._split_work_and_control_reqs(recv_reqs)
else: else:
@@ -180,7 +180,7 @@ class SchedulerRequestReceiver:
# instead of the full tp_group. This avoids an expensive # instead of the full tp_group. This avoids an expensive
# all-ranks gloo sync. # all-ranks gloo sync.
_local_ctrl = ( _local_ctrl = (
get_parallel().enable_dp_attention_local_control_broadcast get_parallel().config.enable_dp_attention_local_control_broadcast
or is_ep_scale_joiner() or is_ep_scale_joiner()
) )
if _local_ctrl: if _local_ctrl:
@@ -258,7 +258,7 @@ class SchedulerRequestReceiver:
# peer ranks may still be unpickling ShmPointerMMData # peer ranks may still be unpickling ShmPointerMMData
# (-> shm_open). Synchronize the same CPU groups that carried # (-> shm_open). Synchronize the same CPU groups that carried
# SHM-backed work requests before materialize() unlinks them. # SHM-backed work requests before materialize() unlinks them.
if get_parallel().enable_dp_attention: if get_parallel().config.enable_dp_attention:
if self.ps.attn_tp_size > 1: if self.ps.attn_tp_size > 1:
barrier(group=self.attn_tp_cpu_group) barrier(group=self.attn_tp_cpu_group)
if self.ps.attn_cp_size > 1: if self.ps.attn_cp_size > 1:
python/sglang/srt/managers/scheduler_pp_mixin.py
+10 -8
View File
@@ -128,7 +128,7 @@ class SchedulerPPMixin:
next_pp_outputs = None next_pp_outputs = None
next_batch_result = None next_batch_result = None
d2h_event = None d2h_event = None
if get_parallel().pp_async_batch_depth > 0: if get_parallel().config.pp_async_batch_depth > 0:
next_pp_outputs, next_batch_result, d2h_event = ( next_pp_outputs, next_batch_result, d2h_event = (
self._pp_commit_send_output_work_and_preprocess_output_tensors( self._pp_commit_send_output_work_and_preprocess_output_tensors(
next_first_rank_mb_id, next_first_rank_mb_id,
@@ -144,7 +144,7 @@ class SchedulerPPMixin:
self.mb_metadata, self.mb_metadata,
self.last_rank_comm_queue, self.last_rank_comm_queue,
) )
if get_parallel().pp_async_batch_depth == 0: if get_parallel().config.pp_async_batch_depth == 0:
next_pp_outputs, next_batch_result, d2h_event = ( next_pp_outputs, next_batch_result, d2h_event = (
self._pp_commit_send_output_work_and_preprocess_output_tensors( self._pp_commit_send_output_work_and_preprocess_output_tensors(
next_first_rank_mb_id, next_first_rank_mb_id,
@@ -274,7 +274,7 @@ class SchedulerPPMixin:
server_is_idle = False server_is_idle = False
pp_proxy_tensors = self._pp_recv_proxy_tensors() pp_proxy_tensors = self._pp_recv_proxy_tensors()
if get_parallel().pp_async_batch_depth > 0: if get_parallel().config.pp_async_batch_depth > 0:
next_pp_outputs, next_batch_result, d2h_event = ( next_pp_outputs, next_batch_result, d2h_event = (
self._pp_commit_send_output_work_and_preprocess_output_tensors( self._pp_commit_send_output_work_and_preprocess_output_tensors(
next_first_rank_mb_id, next_first_rank_mb_id,
@@ -292,7 +292,7 @@ class SchedulerPPMixin:
self.mb_metadata, self.mb_metadata,
self.last_rank_comm_queue, self.last_rank_comm_queue,
) )
if get_parallel().pp_async_batch_depth == 0: if get_parallel().config.pp_async_batch_depth == 0:
next_pp_outputs, next_batch_result, d2h_event = ( next_pp_outputs, next_batch_result, d2h_event = (
self._pp_commit_send_output_work_and_preprocess_output_tensors( self._pp_commit_send_output_work_and_preprocess_output_tensors(
next_first_rank_mb_id, next_first_rank_mb_id,
@@ -435,7 +435,7 @@ class SchedulerPPMixin:
pp_proxy_tensors = self._pp_recv_proxy_tensors() pp_proxy_tensors = self._pp_recv_proxy_tensors()
# early send output if possible # early send output if possible
if get_parallel().pp_async_batch_depth > 0: if get_parallel().config.pp_async_batch_depth > 0:
next_pp_outputs, next_batch_result, d2h_event = ( next_pp_outputs, next_batch_result, d2h_event = (
self._pp_commit_send_output_work_and_preprocess_output_tensors( self._pp_commit_send_output_work_and_preprocess_output_tensors(
next_first_rank_mb_id, next_first_rank_mb_id,
@@ -453,7 +453,7 @@ class SchedulerPPMixin:
self.last_rank_comm_queue, self.last_rank_comm_queue,
) )
if get_parallel().pp_async_batch_depth == 0: if get_parallel().config.pp_async_batch_depth == 0:
next_pp_outputs, next_batch_result, d2h_event = ( next_pp_outputs, next_batch_result, d2h_event = (
self._pp_commit_send_output_work_and_preprocess_output_tensors( self._pp_commit_send_output_work_and_preprocess_output_tensors(
next_first_rank_mb_id, next_first_rank_mb_id,
@@ -564,10 +564,12 @@ class SchedulerPPMixin:
self.on_idle() self.on_idle()
def init_pp_loop_state(self: Scheduler): def init_pp_loop_state(self: Scheduler):
self.pp_loop_size: int = self.ps.pp_size + get_parallel().pp_async_batch_depth self.pp_loop_size: int = (
self.ps.pp_size + get_parallel().config.pp_async_batch_depth
)
# In CP mode, attention weights are duplicated, eliminating the need for the attention TP all-gather operation. # In CP mode, attention weights are duplicated, eliminating the need for the attention TP all-gather operation.
self.require_attn_tp_allgather = ( self.require_attn_tp_allgather = (
not get_parallel().enable_dsa_prefill_context_parallel not get_parallel().config.enable_dsa_prefill_context_parallel
) )
self.mbs = [None] * self.pp_loop_size self.mbs = [None] * self.pp_loop_size
self.last_mbs = [None] * self.pp_loop_size self.last_mbs = [None] * self.pp_loop_size
python/sglang/srt/managers/tokenizer_control_mixin.py
+22 -14
View File
@@ -165,7 +165,7 @@ class TokenizerControlMixin:
mode = spec[2] if len(spec) > 2 else "queueing" mode = spec[2] if len(spec) > 2 else "queueing"
comm = FanOutCommunicator( comm = FanOutCommunicator(
self._dispatch_to_scheduler, self._dispatch_to_scheduler,
get_parallel().dp_size, get_parallel().config.dp_size,
mode, mode,
) )
setattr(self, f"{name}_communicator", comm) setattr(self, f"{name}_communicator", comm)
@@ -174,8 +174,8 @@ class TokenizerControlMixin:
def update_control_communicator_fan_out(self: TokenizerManager, worker_count: int): def update_control_communicator_fan_out(self: TokenizerManager, worker_count: int):
primary_group_control = ( primary_group_control = (
get_parallel().enable_dp_attention get_parallel().config.enable_dp_attention
and not get_parallel().enable_dp_attention_local_control_broadcast and not get_parallel().config.enable_dp_attention_local_control_broadcast
) )
if primary_group_control: if primary_group_control:
control_fan_out = ( control_fan_out = (
@@ -428,7 +428,8 @@ class TokenizerControlMixin:
) -> Tuple[bool, str]: ) -> Tuple[bool, str]:
self.auto_create_handle_loop() self.auto_create_handle_loop()
assert ( assert (
get_parallel().dp_size == 1 or get_parallel().enable_dp_attention get_parallel().config.dp_size == 1
or get_parallel().config.enable_dp_attention
), "dp_size must be 1 or dp attention must be enabled for update weights from distributed" ), "dp_size must be 1 or dp attention must be enabled for update weights from distributed"
results = await self.init_weights_update_group_communicator(obj) results = await self.init_weights_update_group_communicator(obj)
@@ -441,7 +442,8 @@ class TokenizerControlMixin:
) -> Tuple[bool, str]: ) -> Tuple[bool, str]:
self.auto_create_handle_loop() self.auto_create_handle_loop()
assert ( assert (
get_parallel().dp_size == 1 or get_parallel().enable_dp_attention get_parallel().config.dp_size == 1
or get_parallel().config.enable_dp_attention
), "dp_size must be 1 or dp attention must be enabled for destroy parameter update group" ), "dp_size must be 1 or dp attention must be enabled for destroy parameter update group"
results = await self.destroy_weights_update_group_communicator(obj) results = await self.destroy_weights_update_group_communicator(obj)
@@ -454,7 +456,8 @@ class TokenizerControlMixin:
) -> Tuple[bool, str]: ) -> Tuple[bool, str]:
self.auto_create_handle_loop() self.auto_create_handle_loop()
assert ( assert (
get_parallel().dp_size == 1 or get_parallel().enable_dp_attention get_parallel().config.dp_size == 1
or get_parallel().config.enable_dp_attention
), "dp_size must be 1 or dp attention must be enabled for update weights from distributed" ), "dp_size must be 1 or dp attention must be enabled for update weights from distributed"
if obj.abort_all_requests: if obj.abort_all_requests:
@@ -487,7 +490,7 @@ class TokenizerControlMixin:
self.auto_create_handle_loop() self.auto_create_handle_loop()
# TODO: support DP # TODO: support DP
assert ( assert (
get_parallel().dp_size == 1 get_parallel().config.dp_size == 1
), "dp_size must be 1 for init_weights_send_group_for_remote_instance" ), "dp_size must be 1 for init_weights_send_group_for_remote_instance"
result = ( result = (
await self.init_weights_send_group_for_remote_instance_communicator(obj) await self.init_weights_send_group_for_remote_instance_communicator(obj)
@@ -502,7 +505,7 @@ class TokenizerControlMixin:
self.auto_create_handle_loop() self.auto_create_handle_loop()
# TODO: support DP # TODO: support DP
assert ( assert (
get_parallel().dp_size == 1 get_parallel().config.dp_size == 1
), "dp_size must be 1 for send_weights_to_remote_instance" ), "dp_size must be 1 for send_weights_to_remote_instance"
result = (await self.send_weights_to_remote_instance_communicator(obj))[0] result = (await self.send_weights_to_remote_instance_communicator(obj))[0]
return result.success, result.message return result.success, result.message
@@ -514,7 +517,8 @@ class TokenizerControlMixin:
) -> Tuple[bool, str]: ) -> Tuple[bool, str]:
self.auto_create_handle_loop() self.auto_create_handle_loop()
assert ( assert (
get_parallel().dp_size == 1 or get_parallel().enable_dp_attention get_parallel().config.dp_size == 1
or get_parallel().config.enable_dp_attention
), "dp_size must be 1 or dp attention must be enabled for update weights from tensor" ), "dp_size must be 1 or dp attention must be enabled for update weights from tensor"
if obj.abort_all_requests: if obj.abort_all_requests:
@@ -552,7 +556,8 @@ class TokenizerControlMixin:
try: try:
# For now, we only support single data parallel instance # For now, we only support single data parallel instance
assert ( assert (
get_parallel().dp_size == 1 or get_parallel().enable_dp_attention get_parallel().config.dp_size == 1
or get_parallel().config.enable_dp_attention
), "dp_size must be 1 or dp attention must be enabled for update weights from IPC" ), "dp_size must be 1 or dp attention must be enabled for update weights from IPC"
logger.info("Starting IPC weight update") logger.info("Starting IPC weight update")
@@ -615,7 +620,8 @@ class TokenizerControlMixin:
) )
assert ( assert (
get_parallel().dp_size == 1 or get_parallel().enable_dp_attention get_parallel().config.dp_size == 1
or get_parallel().config.enable_dp_attention
), "dp_size must be 1 or dp attention must be enabled for dynamic lora loading" ), "dp_size must be 1 or dp attention must be enabled for dynamic lora loading"
logger.info( logger.info(
"Start load Lora adapter. Lora name=%s, path=%s", "Start load Lora adapter. Lora name=%s, path=%s",
@@ -693,7 +699,8 @@ class TokenizerControlMixin:
) )
assert ( assert (
get_parallel().dp_size == 1 or get_parallel().enable_dp_attention get_parallel().config.dp_size == 1
or get_parallel().config.enable_dp_attention
), "dp_size must be 1 or dp attention must be enabled for dynamic lora loading" ), "dp_size must be 1 or dp attention must be enabled for dynamic lora loading"
logger.info( logger.info(
"Start load Lora adapter from tensors. Lora name=%s", "Start load Lora adapter from tensors. Lora name=%s",
@@ -773,7 +780,8 @@ class TokenizerControlMixin:
), "lora_name must be provided to unload LoRA adapter" ), "lora_name must be provided to unload LoRA adapter"
assert ( assert (
get_parallel().dp_size == 1 or get_parallel().enable_dp_attention get_parallel().config.dp_size == 1
or get_parallel().config.enable_dp_attention
), "dp_size must be 1 or dp attention must be enabled for dynamic lora loading" ), "dp_size must be 1 or dp attention must be enabled for dynamic lora loading"
logger.info( logger.info(
"Start unload Lora adapter. Lora name=%s", "Start unload Lora adapter. Lora name=%s",
@@ -793,7 +801,7 @@ class TokenizerControlMixin:
self.auto_create_handle_loop() self.auto_create_handle_loop()
results = await self.get_weights_by_name_communicator(obj) results = await self.get_weights_by_name_communicator(obj)
all_parameters = [r.parameter for r in results] all_parameters = [r.parameter for r in results]
if get_parallel().dp_size == 1: if get_parallel().config.dp_size == 1:
return all_parameters[0] return all_parameters[0]
else: else:
return all_parameters return all_parameters
python/sglang/srt/managers/tokenizer_manager.py
+2 -2
View File
@@ -411,7 +411,7 @@ class TokenizerManager(TokenizerControlMixin, TokenizerManagerScoreMixin):
self.server_args = server_args self.server_args = server_args
ensure_published(server_args, role="tokenizer") ensure_published(server_args, role="tokenizer")
self.startup_time: Optional[Dict[str, Any]] = None self.startup_time: Optional[Dict[str, Any]] = None
self.elastic_worker_count = get_parallel().dp_size self.elastic_worker_count = get_parallel().config.dp_size
self.elastic_pending_ep_size = None self.elastic_pending_ep_size = None
self.elastic_scale_phase = "idle" self.elastic_scale_phase = "idle"
self.elastic_last_error = None self.elastic_last_error = None
@@ -1548,7 +1548,7 @@ class TokenizerManager(TokenizerControlMixin, TokenizerManagerScoreMixin):
return batch_size > 0 and ( return batch_size > 0 and (
get_serving().enable_tokenizer_batch_encode get_serving().enable_tokenizer_batch_encode
or ( or (
(not get_parallel().enable_dp_attention) (not get_parallel().config.enable_dp_attention)
and (not self._batch_has_text(batch_size, requests)) and (not self._batch_has_text(batch_size, requests))
) )
) )
python/sglang/srt/mem_cache/kv_cache_builder.py
+3 -1
View File
@@ -275,7 +275,9 @@ def build_kv_cache(
), ),
is_eagle=spec_algorithm.is_eagle(), is_eagle=spec_algorithm.is_eagle(),
tp_cache_group=( tp_cache_group=(
attn_tp_cpu_group if get_parallel().enable_dp_attention else tp_cpu_group attn_tp_cpu_group
if get_parallel().config.enable_dp_attention
else tp_cpu_group
), ),
attn_cp_cache_group=attn_cp_cpu_group, attn_cp_cache_group=attn_cp_cpu_group,
attn_tp_cache_group=attn_tp_cpu_group, attn_tp_cache_group=attn_tp_cpu_group,
python/sglang/srt/mem_cache/kv_cache_configurator.py
+1 -2
View File
@@ -65,7 +65,6 @@ from sglang.srt.mem_cache.memory_pool import (
from sglang.srt.mem_cache.swa_memory_pool import SWAKVPool from sglang.srt.mem_cache.swa_memory_pool import SWAKVPool
from sglang.srt.platforms import current_platform from sglang.srt.platforms import current_platform
from sglang.srt.runtime_context import ( from sglang.srt.runtime_context import (
configured_pp_size,
get_context, get_context,
get_disagg, get_disagg,
get_exec, get_exec,
@@ -1922,7 +1921,7 @@ class KVCacheConfigurator:
token_capacity = min(token_capacity, user_limit) token_capacity = min(token_capacity, user_limit)
# Sync across PP ranks (each may have different layer counts) # Sync across PP ranks (each may have different layer counts)
if configured_pp_size() > 1: if get_parallel().config.pp_size > 1:
tensor = torch.tensor(token_capacity, dtype=torch.int64) tensor = torch.tensor(token_capacity, dtype=torch.int64)
torch.distributed.all_reduce( torch.distributed.all_reduce(
tensor, tensor,
python/sglang/srt/mem_cache/pool_host/base.py
+1 -1
View File
@@ -45,7 +45,7 @@ def ranks_per_host() -> int:
return 1 return 1
if world_group.world_size == 1: if world_group.world_size == 1:
return 1 return 1
return max(world_group.world_size // get_parallel().nnodes, 1) return max(world_group.world_size // get_parallel().config.nnodes, 1)
def host_memory_budget_bytes() -> int: def host_memory_budget_bytes() -> int:
python/sglang/srt/model_executor/cpu_graph_runner.py
+2 -3
View File
@@ -40,7 +40,6 @@ from sglang.srt.model_executor.forward_batch_info import (
from sglang.srt.model_executor.forward_context import ForwardContext, forward_context from sglang.srt.model_executor.forward_context import ForwardContext, forward_context
from sglang.srt.model_executor.runner_utils.capture_mode import model_capture_mode from sglang.srt.model_executor.runner_utils.capture_mode import model_capture_mode
from sglang.srt.runtime_context import ( from sglang.srt.runtime_context import (
configured_pp_size,
get_exec, get_exec,
get_flags, get_flags,
get_lora, get_lora,
@@ -611,8 +610,8 @@ class CPUGraphRunner:
model_runner.server_args.enable_profile_cuda_graph model_runner.server_args.enable_profile_cuda_graph
) )
self.tp_size = model_runner.server_args.tp_size self.tp_size = model_runner.server_args.tp_size
self.dp_size = get_parallel().dp_size self.dp_size = get_parallel().config.dp_size
self.pp_size = configured_pp_size() self.pp_size = get_parallel().config.pp_size
self.capture_forward_mode = ForwardMode.DECODE self.capture_forward_mode = ForwardMode.DECODE
self.capture_hidden_mode = self.return_hidden_states_mode self.capture_hidden_mode = self.return_hidden_states_mode
python/sglang/srt/model_executor/model_runner.py
+13 -11
View File
@@ -489,17 +489,19 @@ class ModelRunner:
if not (get_exec().moe.elastic_ep_backend is not None and is_ep_scale_joiner()): if not (get_exec().moe.elastic_ep_backend is not None and is_ep_scale_joiner()):
return return
join_effective_ep_size = get_parallel().ep_join_rank_offset + self.ps.tp_size join_effective_ep_size = (
get_parallel().config.ep_join_rank_offset + self.ps.tp_size
)
dist.barrier(group=self.tp_group.cpu_group) dist.barrier(group=self.tp_group.cpu_group)
if self.ps.tp_rank == 0: if self.ps.tp_rank == 0:
register_scale_cohort( register_scale_cohort(
get_parallel().ep_join_rank_offset, get_parallel().config.ep_join_rank_offset,
join_effective_ep_size, join_effective_ep_size,
) )
join_scale_process_group() join_scale_process_group()
get_context().override("elastic_ep.scale_join", ep_size=join_effective_ep_size) get_context().override("elastic_ep.scale_join", ep_size=join_effective_ep_size)
global_ep_rank = self.ps.tp_rank + get_parallel().ep_join_rank_offset global_ep_rank = self.ps.tp_rank + get_parallel().config.ep_join_rank_offset
broadcast_global_expert_location_metadata( broadcast_global_expert_location_metadata(
model_config=self.model_config, model_config=self.model_config,
moe_ep_rank=global_ep_rank, moe_ep_rank=global_ep_rank,
@@ -699,7 +701,7 @@ class ModelRunner:
if self.is_draft_worker: if self.is_draft_worker:
return return
expert_rank = self.ps.moe_ep_rank + ( expert_rank = self.ps.moe_ep_rank + (
get_parallel().ep_join_rank_offset if is_ep_scale_joiner() else 0 get_parallel().config.ep_join_rank_offset if is_ep_scale_joiner() else 0
) )
set_global_expert_location_metadata( set_global_expert_location_metadata(
compute_initial_expert_location_metadata( compute_initial_expert_location_metadata(
@@ -897,7 +899,7 @@ class ModelRunner:
device=self.device, device=self.device,
tp_group=( tp_group=(
self.attention_tp_group.cpu_group self.attention_tp_group.cpu_group
if get_parallel().enable_dp_attention if get_parallel().config.enable_dp_attention
else self.tp_group.cpu_group else self.tp_group.cpu_group
), ),
host_to_device_ratio=hisparse_cfg.host_to_device_ratio, host_to_device_ratio=hisparse_cfg.host_to_device_ratio,
@@ -933,7 +935,7 @@ class ModelRunner:
def post_capture_elastic_ep_recover(self): def post_capture_elastic_ep_recover(self):
join_process_groups() join_process_groups()
global_ep_rank = self.ps.tp_rank + get_parallel().ep_join_rank_offset global_ep_rank = self.ps.tp_rank + get_parallel().config.ep_join_rank_offset
broadcast_global_expert_location_metadata( broadcast_global_expert_location_metadata(
model_config=self.model_config, model_config=self.model_config,
moe_ep_rank=global_ep_rank, moe_ep_rank=global_ep_rank,
@@ -973,7 +975,7 @@ class ModelRunner:
self.decode_attn_backend = backends.decode_attn_backend self.decode_attn_backend = backends.decode_attn_backend
self.decode_attn_backend_group = backends.decode_attn_backend_group self.decode_attn_backend_group = backends.decode_attn_backend_group
if get_parallel().dcp_enabled and get_parallel().dcp_replicate_q_proj: if get_parallel().dcp_enabled and get_parallel().config.dcp_replicate_q_proj:
self._prepare_replicated_q_proj() self._prepare_replicated_q_proj()
def _prepare_replicated_q_proj(self) -> None: def _prepare_replicated_q_proj(self) -> None:
@@ -1257,7 +1259,7 @@ class ModelRunner:
def maybe_init_dwdp(self): def maybe_init_dwdp(self):
if self.is_draft_worker: if self.is_draft_worker:
return return
if get_parallel().dwdp_size <= 1: if get_parallel().config.dwdp_size <= 1:
return return
from sglang.srt.layers.moe.dwdp import DwdpManager from sglang.srt.layers.moe.dwdp import DwdpManager
@@ -1431,7 +1433,7 @@ class ModelRunner:
# rather than spawning additional processes, so dp_size must not be # rather than spawning additional processes, so dp_size must not be
# multiplied into the process count here (unlike regular DP, where # multiplied into the process count here (unlike regular DP, where
# dp_size * tp_size * pp_size is the true worker count). # dp_size * tp_size * pp_size is the true worker count).
dp_size = 1 if get_parallel().enable_dp_attention else self.ps.dp_size dp_size = 1 if get_parallel().config.enable_dp_attention else self.ps.dp_size
self.local_omp_cpuid = numa_utils.init_threads_binding( self.local_omp_cpuid = numa_utils.init_threads_binding(
numa_index=self.gpu_id, numa_index=self.gpu_id,
world_size=dp_size * self.ps.tp_size * self.ps.pp_size, world_size=dp_size * self.ps.tp_size * self.ps.pp_size,
@@ -1916,7 +1918,7 @@ class ModelRunner:
if added <= 0: if added <= 0:
return return
initial_ep_size = get_parallel().elastic_ep_initial_size initial_ep_size = get_parallel().config.elastic_ep_initial_size
assert initial_ep_size is not None assert initial_ep_size is not None
get_context().override("elastic_ep.scale", ep_size=effective_size) get_context().override("elastic_ep.scale", ep_size=effective_size)
@@ -1935,7 +1937,7 @@ class ModelRunner:
set_global_expert_location_metadata(new_metadata, allow_overwrite=True) set_global_expert_location_metadata(new_metadata, allow_overwrite=True)
def _elastic_global_rank(self) -> int: def _elastic_global_rank(self) -> int:
return self.ps.tp_rank + get_parallel().ep_join_rank_offset return self.ps.tp_rank + get_parallel().config.ep_join_rank_offset
def _rearm_eplb_after_elastic_scale(self) -> None: def _rearm_eplb_after_elastic_scale(self) -> None:
if self.eplb_manager is None: if self.eplb_manager is None:
python/sglang/srt/model_executor/model_runner_components/remote_instance_weight_transporter.py
+4 -2
View File
@@ -78,11 +78,13 @@ class RemoteInstanceWeightTransporter:
""" """
import requests as http_requests import requests as http_requests
if get_parallel().dist_init_addr: if get_parallel().config.dist_init_addr:
# Multi-node: bootstrap server is on the head node (node_rank==0). # Multi-node: bootstrap server is on the head node (node_rank==0).
# Derive host from dist_init_addr (shared across all nodes). # Derive host from dist_init_addr (shared across all nodes).
bootstrap_host = ( bootstrap_host = (
NetworkAddress.parse(get_parallel().dist_init_addr).resolved().host NetworkAddress.parse(get_parallel().config.dist_init_addr)
.resolved()
.host
) )
else: else:
bootstrap_host = "127.0.0.1" bootstrap_host = "127.0.0.1"
python/sglang/srt/model_executor/model_runner_components/spec_aux_hidden_state.py
+2 -2
View File
@@ -7,8 +7,8 @@ import msgspec
from sglang.srt.configs.model_config import ModelConfig from sglang.srt.configs.model_config import ModelConfig
from sglang.srt.runtime_context import ( from sglang.srt.runtime_context import (
configured_tp_size,
get_model, get_model,
get_parallel,
get_spec, get_spec,
) )
@@ -242,7 +242,7 @@ def _resolve_dflash_draft_cell_size(
draft_model_config=draft_model_config, draft_model_config=draft_model_config,
draft_num_layers=draft_num_layers, draft_num_layers=draft_num_layers,
draft_kv_cache_dtype=draft_kv_cache_dtype, draft_kv_cache_dtype=draft_kv_cache_dtype,
tp_size=configured_tp_size(), tp_size=get_parallel().config.tp_size,
) )
except Exception as e: # noqa: BLE001 except Exception as e: # noqa: BLE001
logger.warning( logger.warning(
python/sglang/srt/model_executor/model_runner_components/startup_weight_load.py
+6 -10
View File
@@ -21,10 +21,6 @@ from sglang.srt.model_loader.weight_utils import (
) )
from sglang.srt.platforms import current_platform from sglang.srt.platforms import current_platform
from sglang.srt.runtime_context import ( from sglang.srt.runtime_context import (
configured_attn_cp_size,
configured_dcp_size,
configured_pp_size,
configured_tp_size,
get_device, get_device,
get_exec, get_exec,
get_lora, get_lora,
@@ -120,12 +116,12 @@ class StartupWeightLoadOptions:
prefill_cuda_graph_backend=cuda_graph_config.prefill.backend, prefill_cuda_graph_backend=cuda_graph_config.prefill.backend,
is_draft_worker=is_draft_worker, is_draft_worker=is_draft_worker,
speculative_algorithm=get_spec().speculative_algorithm, speculative_algorithm=get_spec().speculative_algorithm,
tp_size=configured_tp_size(), tp_size=get_parallel().config.tp_size,
attn_cp_size=configured_attn_cp_size(), attn_cp_size=get_parallel().config.attn_cp_size,
dcp_size=configured_dcp_size(), dcp_size=get_parallel().config.dcp_size,
pp_size=configured_pp_size(), pp_size=get_parallel().config.pp_size,
dp_size=get_parallel().dp_size, dp_size=get_parallel().config.dp_size,
ep_size=get_parallel().ep_size, ep_size=get_parallel().config.ep_size,
cpu_offload_gb=get_exec().offload.cpu_offload_gb, cpu_offload_gb=get_exec().offload.cpu_offload_gb,
offload_group_size=get_exec().offload.offload_group_size, offload_group_size=get_exec().offload.offload_group_size,
enable_memory_saver=get_exec().features.enable_memory_saver, enable_memory_saver=get_exec().features.enable_memory_saver,
python/sglang/srt/model_executor/runner/base_runner.py
+8 -9
View File
@@ -47,7 +47,6 @@ from sglang.srt.model_executor.runner.flashinfer_autotune import (
should_run_flashinfer_autotune, should_run_flashinfer_autotune,
) )
from sglang.srt.runtime_context import ( from sglang.srt.runtime_context import (
configured_pp_size,
get_disagg, get_disagg,
get_exec, get_exec,
get_flags, get_flags,
@@ -219,8 +218,8 @@ class BaseRunner(ABC):
self.device_module = torch.get_device_module(self.device) self.device_module = torch.get_device_module(self.device)
self.tp_size = model_runner.server_args.tp_size self.tp_size = model_runner.server_args.tp_size
# elastic-EP scale-up rewrites dp_size on the published config # elastic-EP scale-up rewrites dp_size on the published config
self.dp_size = get_parallel().dp_size self.dp_size = get_parallel().config.dp_size
self.pp_size = configured_pp_size() self.pp_size = get_parallel().config.pp_size
self.enable_pdmux = model_runner.server_args.enable_pdmux self.enable_pdmux = model_runner.server_args.enable_pdmux
self.return_hidden_states_mode = ( self.return_hidden_states_mode = (
CaptureHiddenMode.NULL CaptureHiddenMode.NULL
@@ -290,7 +289,7 @@ class BaseRunner(ABC):
""" """
if ( if (
not get_parallel().dcp_enabled not get_parallel().dcp_enabled
or get_parallel().dcp_comm_backend != "fi_a2a" or get_parallel().config.dcp_comm_backend != "fi_a2a"
): ):
return return
@@ -349,8 +348,8 @@ class BaseRunner(ABC):
hidden_size=mr.model_config.hidden_size, hidden_size=mr.model_config.hidden_size,
vocab_size=mr.model_config.vocab_size, vocab_size=mr.model_config.vocab_size,
dtype=mr.model_config.dtype, dtype=mr.model_config.dtype,
dp_size=get_parallel().dp_size, dp_size=get_parallel().config.dp_size,
pp_size=configured_pp_size(), pp_size=get_parallel().config.pp_size,
is_encoder_decoder=mr.model_config.is_encoder_decoder, is_encoder_decoder=mr.model_config.is_encoder_decoder,
require_mlp_tp_gather=require_mlp_tp_gather(mr.server_args), require_mlp_tp_gather=require_mlp_tp_gather(mr.server_args),
seq_len_fill_value=mr.attn_backend.get_cuda_graph_seq_len_fill_value(), seq_len_fill_value=mr.attn_backend.get_cuda_graph_seq_len_fill_value(),
@@ -522,7 +521,7 @@ class BaseRunner(ABC):
extend_prefix_lens = None extend_prefix_lens = None
extend_start_loc = None extend_start_loc = None
if configured_pp_size() > 1: if get_parallel().config.pp_size > 1:
# PP0 already cp-split hidden_states before send. # PP0 already cp-split hidden_states before send.
pp_hidden_tokens = num_tokens pp_hidden_tokens = num_tokens
if ( if (
@@ -542,7 +541,7 @@ class BaseRunner(ABC):
assert require_mlp_tp_gather_ or require_attn_tp_gather_ assert require_mlp_tp_gather_ or require_attn_tp_gather_
if require_mlp_tp_gather_: if require_mlp_tp_gather_:
global_num_tokens_cpu = [num_tokens] * get_parallel().dp_size global_num_tokens_cpu = [num_tokens] * get_parallel().config.dp_size
elif require_attn_tp_gather_: elif require_attn_tp_gather_:
global_num_tokens_cpu = [num_tokens] global_num_tokens_cpu = [num_tokens]
else: else:
@@ -646,7 +645,7 @@ class BaseRunner(ABC):
kwargs = {} kwargs = {}
if ( if (
configured_pp_size() > 1 get_parallel().config.pp_size > 1
and "pp_proxy_tensors" in inspect.signature(mr.model.forward).parameters and "pp_proxy_tensors" in inspect.signature(mr.model.forward).parameters
): ):
kwargs["pp_proxy_tensors"] = PPProxyTensors( kwargs["pp_proxy_tensors"] = PPProxyTensors(
python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py
+1 -1
View File
@@ -238,7 +238,7 @@ class DecodeCudaGraphRunner(BaseCudaGraphRunner):
self.require_mlp_tp_gather or self.require_attn_tp_gather self.require_mlp_tp_gather or self.require_attn_tp_gather
) )
self.require_mlp_sync = ( self.require_mlp_sync = (
get_parallel().enable_dp_attention or self.require_gathered_buffer get_parallel().config.enable_dp_attention or self.require_gathered_buffer
) )
self.enable_two_batch_overlap = ( self.enable_two_batch_overlap = (
model_runner.server_args.enable_two_batch_overlap model_runner.server_args.enable_two_batch_overlap
python/sglang/srt/model_executor/runner/eager_runner.py
+1 -1
View File
@@ -141,7 +141,7 @@ class EagerRunner(BaseRunner):
encoder_lens_dtype=( encoder_lens_dtype=(
torch.int64 if torch.device(mr.device).type == "cpu" else torch.int32 torch.int64 if torch.device(mr.device).type == "cpu" else torch.int32
), ),
dp_size=get_parallel().dp_size, dp_size=get_parallel().config.dp_size,
) )
# Eager has no capture step, so warm up here (run-once via mr._kernel_warmed_up). # Eager has no capture step, so warm up here (run-once via mr._kernel_warmed_up).
self.warmup() self.warmup()
python/sglang/srt/model_executor/runner/prefill_cuda_graph_runner.py
+1 -1
View File
@@ -348,7 +348,7 @@ class PrefillCudaGraphRunner(BaseCudaGraphRunner):
self.moe_fusions = self.model_runner.moe_fusions self.moe_fusions = self.model_runner.moe_fusions
self.dsa_indexers = getattr(self.model_runner, "dsa_indexers", None) self.dsa_indexers = getattr(self.model_runner, "dsa_indexers", None)
self.dp_size = get_parallel().dp_size self.dp_size = get_parallel().config.dp_size
self.require_mlp_tp_gather = require_mlp_tp_gather(model_runner.server_args) self.require_mlp_tp_gather = require_mlp_tp_gather(model_runner.server_args)
self.require_attn_tp_gather = require_attn_tp_gather(model_runner.server_args) self.require_attn_tp_gather = require_attn_tp_gather(model_runner.server_args)
python/sglang/srt/model_loader/loader.py
+3 -4
View File
@@ -49,7 +49,6 @@ from sglang.srt.model_loader.remote_instance_weight_loader_utils import (
register_memory_region, register_memory_region,
) )
from sglang.srt.runtime_context import ( from sglang.srt.runtime_context import (
configured_moe_dp_size,
get_exec, get_exec,
get_model, get_model,
get_parallel, get_parallel,
@@ -1891,9 +1890,9 @@ class PreshardedModelLoader(DefaultModelLoader):
"dp": _safe(lambda: parallel.moe_dp_size), "dp": _safe(lambda: parallel.moe_dp_size),
"ep": _safe(lambda: parallel.moe_ep_size), "ep": _safe(lambda: parallel.moe_ep_size),
"pp": _safe(lambda: parallel.pp_size), "pp": _safe(lambda: parallel.pp_size),
"moe_dense_tp_size": parallel.moe_dense_tp_size, "moe_dense_tp_size": parallel.config.moe_dense_tp_size,
"moe_dp_size": configured_moe_dp_size(), "moe_dp_size": get_parallel().config.moe_dp_size,
"enable_dp_lm_head": parallel.enable_dp_lm_head, "enable_dp_lm_head": parallel.config.enable_dp_lm_head,
"enable_fp32_lm_head": get_exec().features.enable_fp32_lm_head, "enable_fp32_lm_head": get_exec().features.enable_fp32_lm_head,
"quantization": model_config.quantization, "quantization": model_config.quantization,
"model_dtype": str(model_config.dtype), "model_dtype": str(model_config.dtype),
python/sglang/srt/models/apertus.py
+1 -1
View File
@@ -442,7 +442,7 @@ class ApertusForCausalLM(nn.Module):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
self.pooler = Pooler(pooling_type=PoolingType.LAST, normalize=True) self.pooler = Pooler(pooling_type=PoolingType.LAST, normalize=True)
python/sglang/srt/models/arcee.py
+1 -1
View File
@@ -405,7 +405,7 @@ class ArceeForCausalLM(nn.Module):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
self.pooler = Pooler(pooling_type=PoolingType.LAST, normalize=True) self.pooler = Pooler(pooling_type=PoolingType.LAST, normalize=True)
python/sglang/srt/models/bailing_moe.py
+1 -1
View File
@@ -817,7 +817,7 @@ class BailingMoEForCausalLM(nn.Module):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/bailing_moe_linear.py
+1 -1
View File
@@ -1084,7 +1084,7 @@ class BailingMoELinearForCausalLM(nn.Module):
config.hidden_size, config.hidden_size,
params_dtype=torch.float32, params_dtype=torch.float32,
quant_config=quant_config, quant_config=quant_config,
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/bailing_moe_nextn.py
+1 -1
View File
@@ -208,7 +208,7 @@ class BailingMoeForCausalLMNextN(nn.Module):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("model.shared_head.head", prefix), prefix=add_prefix("model.shared_head.head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
if hasattr(self.config, "model_type") and config.model_type == "bailing_hybrid": if hasattr(self.config, "model_type") and config.model_type == "bailing_hybrid":
python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py
+2 -2
View File
@@ -299,7 +299,7 @@ class DeepseekMLAForwardMixin:
# --dcp-replicate-q-proj: project full-head Q locally from pre-gathered # --dcp-replicate-q-proj: project full-head Q locally from pre-gathered
# weights and skip the per-layer Q all-gather (bf16 decode absorb only). # weights and skip the per-layer Q all-gather (bf16 decode absorb only).
q_replicate_active = ( q_replicate_active = (
get_parallel().dcp_replicate_q_proj get_parallel().config.dcp_replicate_q_proj
and is_dcp_mla_decode_phase(forward_batch) and is_dcp_mla_decode_phase(forward_batch)
and not self.use_deep_gemm_bmm and not self.use_deep_gemm_bmm
and self.w_kc_qrep is not None and self.w_kc_qrep is not None
@@ -778,7 +778,7 @@ class DeepseekMLAForwardMixin:
attn_output, self.num_local_heads attn_output, self.num_local_heads
) )
else: else:
dcp_comm_backend = get_parallel().dcp_comm_backend dcp_comm_backend = get_parallel().config.dcp_comm_backend
is_lse_base_on_e = is_mla_dcp_lse_base_on_e( is_lse_base_on_e = is_mla_dcp_lse_base_on_e(
self.current_attention_backend self.current_attention_backend
) )
python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla_rocm.py
+2 -2
View File
@@ -331,7 +331,7 @@ class DeepseekMLARocmForwardMixin:
from sglang.srt.model_executor.runner import get_is_capture_mode from sglang.srt.model_executor.runner import get_is_capture_mode
q_replicate_active = ( q_replicate_active = (
get_parallel().dcp_replicate_q_proj get_parallel().config.dcp_replicate_q_proj
and is_dcp_mla_decode_phase(forward_batch) and is_dcp_mla_decode_phase(forward_batch)
and not self.use_deep_gemm_bmm and not self.use_deep_gemm_bmm
and self.w_kc_qrep is not None and self.w_kc_qrep is not None
@@ -778,7 +778,7 @@ class DeepseekMLARocmForwardMixin:
attn_output, self.num_local_heads attn_output, self.num_local_heads
) )
else: else:
dcp_comm_backend = get_parallel().dcp_comm_backend dcp_comm_backend = get_parallel().config.dcp_comm_backend
is_lse_base_on_e = is_mla_dcp_lse_base_on_e( is_lse_base_on_e = is_mla_dcp_lse_base_on_e(
self.current_attention_backend self.current_attention_backend
) )
python/sglang/srt/models/deepseek_nextn.py
+1 -1
View File
@@ -366,7 +366,7 @@ class DeepseekV3ForCausalLMNextN(DeepseekV3ForCausalLM):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("model.shared_head.head", prefix), prefix=add_prefix("model.shared_head.head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/deepseek_v2.py
+1 -1
View File
@@ -3003,7 +3003,7 @@ class DeepseekV2ForCausalLM(nn.Module, DeepseekV2WeightLoaderMixin):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
else: else:
# ranks other than the last rank will have a placeholder layer # ranks other than the last rank will have a placeholder layer
python/sglang/srt/models/deepseek_v4.py
+1 -1
View File
@@ -3230,7 +3230,7 @@ class DeepseekV4ForCausalLM(nn.Module):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
else: else:
self.lm_head = PPMissingLayer() self.lm_head = PPMissingLayer()
python/sglang/srt/models/deepseek_v4_dspark.py
+1 -1
View File
@@ -756,7 +756,7 @@ class DeepseekV4ForCausalLMDSpark(nn.Module):
config.vocab_size, config.vocab_size,
config.hidden_size, config.hidden_size,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
else: else:
self.embed_tokens: Optional[nn.Module] = None self.embed_tokens: Optional[nn.Module] = None
python/sglang/srt/models/deepseek_v4_nextn.py
+1 -1
View File
@@ -250,7 +250,7 @@ class DeepseekV4ForCausalLMNextN(DeepseekV4ForCausalLM):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("model.shared_head.head", prefix), prefix=add_prefix("model.shared_head.head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/dots3_common/modeling.py
+1 -1
View File
@@ -1876,7 +1876,7 @@ class Dots3LanguageModelForCausalLM(nn.Module):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/dots3_common/nextn.py
+1 -1
View File
@@ -167,7 +167,7 @@ class Dots3NoteForCausalLMNextN(Dots3LanguageModelForCausalLM):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("model.shared_head.head", prefix), prefix=add_prefix("model.shared_head.head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
self._mtp_loaded_embed = False self._mtp_loaded_embed = False
python/sglang/srt/models/exaone4.py
+1 -1
View File
@@ -439,7 +439,7 @@ class Exaone4ForCausalLM(nn.Module):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/exaone_moe.py
+1 -1
View File
@@ -643,7 +643,7 @@ class ExaoneMoEForCausalLM(nn.Module):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
# For EAGLE3 support # For EAGLE3 support
python/sglang/srt/models/exaone_moe_mtp.py
+1 -1
View File
@@ -63,7 +63,7 @@ class ExaoneMoEForCausalLMMTP(ExaoneMoEForCausalLM):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/falcon_h1.py
+1 -1
View File
@@ -472,7 +472,7 @@ class FalconH1ForCausalLM(nn.Module):
quant_config=quant_config, quant_config=quant_config,
org_num_embeddings=config.vocab_size, org_num_embeddings=config.vocab_size,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.lm_head = self.lm_head.float() self.lm_head = self.lm_head.float()
self.lm_head_multiplier = config.lm_head_multiplier self.lm_head_multiplier = config.lm_head_multiplier
python/sglang/srt/models/glm4_moe.py
+1 -1
View File
@@ -1163,7 +1163,7 @@ class Glm4MoeForCausalLM(nn.Module):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/glm4_moe_lite.py
+1 -1
View File
@@ -905,7 +905,7 @@ class Glm4MoeLiteForCausalLM(nn.Module, DeepseekV2WeightLoaderMixin):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/glm4_moe_lite_nextn.py
+1 -1
View File
@@ -159,7 +159,7 @@ class Glm4MoeLiteForCausalLMNextN(Glm4MoeLiteForCausalLM):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("model.shared_head.head", prefix), prefix=add_prefix("model.shared_head.head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/glm4_moe_nextn.py
+1 -1
View File
@@ -144,7 +144,7 @@ class Glm4MoeForCausalLMNextN(Glm4MoeForCausalLM):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("model.shared_head.head", prefix), prefix=add_prefix("model.shared_head.head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/glm4v_moe.py
+1 -1
View File
@@ -70,7 +70,7 @@ class Glm4vMoeForConditionalGeneration(Glm4vForConditionalGeneration):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
else: else:
# ranks other than the last rank will have a placeholder layer # ranks other than the last rank will have a placeholder layer
python/sglang/srt/models/glm_ocr_nextn.py
+1 -1
View File
@@ -135,7 +135,7 @@ class GlmOcrForConditionalGenerationNextN(GlmOcrForConditionalGeneration):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("model.shared_head.head", prefix), prefix=add_prefix("model.shared_head.head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/gpt_oss.py
+2 -2
View File
@@ -258,7 +258,7 @@ class GptOssSparseMoeBlock(nn.Module):
hidden_states: torch.Tensor, hidden_states: torch.Tensor,
forward_batch: Optional[ForwardBatch] = None, forward_batch: Optional[ForwardBatch] = None,
) -> torch.Tensor: ) -> torch.Tensor:
if get_parallel().dwdp_size > 1: if get_parallel().config.dwdp_size > 1:
return self.forward_dwdp(hidden_states) return self.forward_dwdp(hidden_states)
if not get_moe_a2a_backend().is_deepep(): if not get_moe_a2a_backend().is_deepep():
@@ -786,7 +786,7 @@ class GptOssForCausalLM(nn.Module):
config.hidden_size, config.hidden_size,
# quant_config=quant_config, # quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
self.capture_aux_hidden_states = False self.capture_aux_hidden_states = False
python/sglang/srt/models/kimi_k25.py
+2 -2
View File
@@ -46,9 +46,9 @@ from sglang.srt.multimodal.mm_utils import (
run_dp_sharded_mrope_vision_model, run_dp_sharded_mrope_vision_model,
) )
from sglang.srt.runtime_context import ( from sglang.srt.runtime_context import (
configured_tp_size,
get_exec, get_exec,
get_mm, get_mm,
get_parallel,
) )
from sglang.srt.utils import add_prefix, is_cuda, is_npu from sglang.srt.utils import add_prefix, is_cuda, is_npu
@@ -733,7 +733,7 @@ class KimiK25ForConditionalGeneration(nn.Module):
# Match the configured TP consumer count captured when the # Match the configured TP consumer count captured when the
# tokenizer creates MmItemMemoryPool. A live attention subgroup # tokenizer creates MmItemMemoryPool. A live attention subgroup
# size could leave acknowledgements missing and strand the lease. # size could leave acknowledgements missing and strand the lease.
ipc_consumer_count = max(configured_tp_size(), 1) ipc_consumer_count = max(get_parallel().config.tp_size, 1)
device_index = device.index device_index = device.index
if device.type == "cuda" and device_index is None: if device.type == "cuda" and device_index is None:
device_index = torch.cuda.current_device() device_index = torch.cuda.current_device()
python/sglang/srt/models/kimi_k3.py
+5 -6
View File
@@ -119,7 +119,6 @@ from sglang.srt.multimodal.kimi_k3_image_processing import (
) )
from sglang.srt.multimodal.mm_utils import materialize_multimodal_features from sglang.srt.multimodal.mm_utils import materialize_multimodal_features
from sglang.srt.runtime_context import ( from sglang.srt.runtime_context import (
configured_tp_size,
get_exec, get_exec,
get_parallel, get_parallel,
get_server_args, get_server_args,
@@ -298,7 +297,7 @@ class KimiK3MLP(nn.Module):
# but allow the NPU launcher to retain the proven attention-TP layout # but allow the NPU launcher to retain the proven attention-TP layout
# without a device-type branch in shared model code. # without a device-type branch in shared model code.
self._dense_attn_tp = ( self._dense_attn_tp = (
get_parallel().enable_dense_mlp_attn_tp get_parallel().config.enable_dense_mlp_attn_tp
and is_dp_attention_enabled() and is_dp_attention_enabled()
and tp_rank is None and tp_rank is None
and tp_size is None and tp_size is None
@@ -556,13 +555,13 @@ class KimiK3MoE(nn.Module):
# a TP-sharded partial sum could never be reduced across ranks that # a TP-sharded partial sum could never be reduced across ranks that
# hold different tokens. # hold different tokens.
self._shared_experts_tp1 = ( self._shared_experts_tp1 = (
self._ep_a2a and not get_parallel().enable_shared_experts_attn_tp self._ep_a2a and not get_parallel().config.enable_shared_experts_attn_tp
) )
# NPU compatibility mode keeps DeepEP's DP-local token dispatch but # NPU compatibility mode keeps DeepEP's DP-local token dispatch but
# uses the original TP-sharded shared MLP. Gather only that branch's # uses the original TP-sharded shared MLP. Gather only that branch's
# inputs, then reduce-scatter its output back to the DP-local rows. # inputs, then reduce-scatter its output back to the DP-local rows.
self._shared_experts_attn_tp_comm = ( self._shared_experts_attn_tp_comm = (
get_parallel().enable_shared_experts_attn_tp get_parallel().config.enable_shared_experts_attn_tp
and self._ep_a2a and self._ep_a2a
and self._dp_attention and self._dp_attention
and get_parallel().attn_tp_size > 1 and get_parallel().attn_tp_size > 1
@@ -2872,7 +2871,7 @@ class KimiK3LinearForCausalLM(nn.Module):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=maybe_prefix(prefix, "lm_head"), prefix=maybe_prefix(prefix, "lm_head"),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
else: else:
self.lm_head = PPMissingLayer() self.lm_head = PPMissingLayer()
@@ -3384,7 +3383,7 @@ class KimiK3ForConditionalGeneration(nn.Module):
# Match the configured TP consumer count captured when the # Match the configured TP consumer count captured when the
# tokenizer creates MmItemMemoryPool. A live attention subgroup # tokenizer creates MmItemMemoryPool. A live attention subgroup
# size could leave acknowledgements missing and strand the lease. # size could leave acknowledgements missing and strand the lease.
ipc_consumer_count = max(configured_tp_size(), 1) ipc_consumer_count = max(get_parallel().config.tp_size, 1)
device_index = device.index device_index = device.index
if device.type == "cuda" and device_index is None: if device.type == "cuda" and device_index is None:
device_index = torch.cuda.current_device() device_index = torch.cuda.current_device()
python/sglang/srt/models/laguna.py
+1 -1
View File
@@ -667,7 +667,7 @@ class LagunaForCausalLM(nn.Module):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
else: else:
self.lm_head = PPMissingLayer() self.lm_head = PPMissingLayer()
python/sglang/srt/models/llada2.py
+1 -1
View File
@@ -824,7 +824,7 @@ class LLaDA2MoeModelLM(nn.Module):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config, return_full_logits=True) self.logits_processor = LogitsProcessor(config, return_full_logits=True)
python/sglang/srt/models/llama.py
+1 -1
View File
@@ -536,7 +536,7 @@ class LlamaForCausalLM(nn.Module):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
self.pooler = Pooler(pooling_type=PoolingType.LAST, normalize=True) self.pooler = Pooler(pooling_type=PoolingType.LAST, normalize=True)
python/sglang/srt/models/longcat_flash.py
+1 -1
View File
@@ -721,7 +721,7 @@ class LongcatFlashForCausalLM(nn.Module):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
self.capture_aux_hidden_states = False self.capture_aux_hidden_states = False
python/sglang/srt/models/mellum.py
+1 -1
View File
@@ -520,7 +520,7 @@ class MellumForCausalLM(Qwen3MoeForCausalLM):
cfg.hidden_size, cfg.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(cfg) self.logits_processor = LogitsProcessor(cfg)
self.capture_aux_hidden_states = False self.capture_aux_hidden_states = False
python/sglang/srt/models/mimo_v2.py
+1 -1
View File
@@ -1187,7 +1187,7 @@ class MiMoV2ForCausalLM(nn.Module, AudioEncoderMixin):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
else: else:
self.lm_head = PPMissingLayer() self.lm_head = PPMissingLayer()
python/sglang/srt/models/mimo_v2_nextn.py
+1 -1
View File
@@ -259,7 +259,7 @@ class MiMoV2MTP(MiMoV2ForCausalLM):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/minimax_m3.py
+1 -1
View File
@@ -1574,7 +1574,7 @@ class MiniMaxM3SparseForCausalLM(nn.Module):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/minimax_m3_vl.py
+1 -1
View File
@@ -123,7 +123,7 @@ class MiniMaxM3SparseForConditionalGeneration(nn.Module):
text_config.hidden_size, text_config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("language_model.lm_head", prefix), prefix=add_prefix("language_model.lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
else: else:
self.lm_head = PPMissingLayer() self.lm_head = PPMissingLayer()
python/sglang/srt/models/nemotron_h.py
+1 -1
View File
@@ -964,7 +964,7 @@ class NemotronHForCausalLM(nn.Module):
else lora_config.lora_vocab_padding_size else lora_config.lora_vocab_padding_size
), ),
quant_config=quant_config, quant_config=quant_config,
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
) )
else: else:
python/sglang/srt/models/nemotron_h_mtp.py
+1 -1
View File
@@ -339,7 +339,7 @@ class NemotronHForCausalLMMTP(NemotronHForCausalLM):
self.config.hidden_size, self.config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/qwen2_moe.py
+1 -1
View File
@@ -1119,7 +1119,7 @@ class Qwen2MoeForCausalLM(nn.Module):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
# For EAGLE3 support # For EAGLE3 support
python/sglang/srt/models/qwen3.py
+1 -1
View File
@@ -492,7 +492,7 @@ class Qwen3ForCausalLM(nn.Module):
config.vocab_size, config.vocab_size,
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
) )
else: else:
python/sglang/srt/models/qwen3_5_text.py
+1 -1
View File
@@ -80,7 +80,7 @@ class Qwen3_5ForCausalLM(nn.Module):
quant_config=quant_config, quant_config=quant_config,
org_num_embeddings=config.vocab_size, org_num_embeddings=config.vocab_size,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
else: else:
self.lm_head = PPMissingLayer() self.lm_head = PPMissingLayer()
python/sglang/srt/models/qwen3_moe.py
+1 -1
View File
@@ -961,7 +961,7 @@ class Qwen3MoeForCausalLM(nn.Module):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
self.capture_aux_hidden_states = False self.capture_aux_hidden_states = False
python/sglang/srt/models/qwen3_moe_mtp.py
+1 -1
View File
@@ -63,7 +63,7 @@ class Qwen3MoeForCausalLMMTP(Qwen3MoeForCausalLM):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/qwen3_next.py
+1 -1
View File
@@ -1027,7 +1027,7 @@ class Qwen3NextForCausalLM(nn.Module):
quant_config=quant_config, quant_config=quant_config,
org_num_embeddings=config.vocab_size, org_num_embeddings=config.vocab_size,
prefix=add_prefix("lm_head", prefix), prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
# For EAGLE3 support # For EAGLE3 support
python/sglang/srt/models/qwen3_next_mtp.py
+1 -1
View File
@@ -80,7 +80,7 @@ class Qwen3NextForCausalLMMTP(Qwen3NextForCausalLM):
config.hidden_size, config.hidden_size,
quant_config=quant_config, quant_config=quant_config,
prefix=add_prefix("model.shared_head.head", prefix), prefix=add_prefix("model.shared_head.head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head, use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
) )
self.logits_processor = LogitsProcessor(config) self.logits_processor = LogitsProcessor(config)
# Mirror Qwen3NextForCausalLM.__init__'s shared-expert fusion setup so # Mirror Qwen3NextForCausalLM.__init__'s shared-expert fusion setup so

Some files were not shown because too many files have changed in this diff Show More