minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity

config: spell the parallel config tier at the call site (#36250)

Browse Source
This commit is contained in:
Cheng Wan
2026-08-26 03:00:28 -07:00
committed by GitHub
parent 689ade69d1
commit 8005df61d3
136 changed files with 1033 additions and 795 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/skills/sglang-runtime-context/SKILL.md
+58 -45
View File
@@ -15,7 +15,7 @@ One container owns process-static runtime state: `sglang.srt.runtime_context.Run
| runtime flags | `get_flags()` | state that is *not* a pure function of config: `capture` (cuda-graph lifecycle), `moe` (ACTIVE backends, swappable), `dp` (DP-attention runtime flags) | materialized at subsystem init; groups offer `override()` for tests |
| resources | `get_resources()`, `get_stream(name)`, `get_buffer(name, factory)` | process-level handles: graph pools, EPLB state, EP dispatcher state, named side streams, workspace buffers | lazy; cleared by `reset_context()` |
| per-forward | `get_forward()` | forward-scoped flags (multi-stream switch, MoE output buffer, attn-TP inputs, extend-in-batch) | contextvar-backed; `scoped(**kw)` restores on exit; new threads see defaults |
| parallel | `get_parallel()` | **dual**: live topology (tp/pp/moe/attn sizes, ranks, groups — `@property`, read-through) *plus* parallel config-bag leaves via `__getattr__` | live: after dist init; config leaves: after publish |
| parallel | `get_parallel()` | **dual, spelled**: bare names are the live topology (tp/pp/moe/attn sizes, ranks, groups — `@property`, read-through); `get_parallel().config.<leaf>` is the parallel config bag | live: after dist init; `config`: after publish |
`reset_context()` (unit-test teardown) drops the published config and installs fresh
flags/resources/forward tiers.
@@ -214,23 +214,29 @@ row for a method the Ray actor does not have, and an effective-field set
without `load_format` -- and both were invisible because the assertion had
slack (`>= len(...) - 1`) or compared key names instead of value sources.
### `get_parallel()`: config leaves vs live topology
### `get_parallel()`: live topology bare, configuration under `config`
Config leaves (`nccl_port`, `enable_dp_attention`, `dp_size`, `ep_size`,
`dwdp_size`, ...) resolve through the parallel bag; live topology (`tp_size`,
`attn_tp_group`, ranks) are `@property` and **win on name collisions**. Five topology
sizes are live-shadowed (`tp/pp/dcp/attn_cp/moe_dp_size`): the live property always
wins on the accessor, so a config-intent read of those goes through
`configured_tp_size()` / `configured_pp_size()` / `configured_moe_dp_size()` /
`configured_attn_cp_size()` (DCP: the live `get_parallel().attn_dcp_size` /
`.dcp_enabled`, which are safe with no group installed — they answer `1` /
`False` — but report the *effective* topology, never the requested size;
a config-intent read would need its own accessor, which no call site
requires today). A process-global seed field-read of one of these
sizes (`get_server_args().tp_size`, or an alias of it) is a read-ratchet failure; the
sites that legitimately go around the live property are the `configured_*_size()`
readers, and those are what the ratchet registers, each with its reason
(`_CONFIGURED_SIZE_CALL_SITES` in `test_global_config_read_ratchet.py`). A
**Bare is the live group, `config` is what was configured.** `get_parallel().tp_size`
and its size / rank / group siblings are `@property` read-through over the canonical
getters; `get_parallel().config.<leaf>` reads the published `parallel` bag
(`nccl_port`, `enable_dp_attention`, `dp_size`, `ep_size`, `dwdp_size`, ... and the
five sizes that also have a live property). A bare read of a config-only leaf raises
an `AttributeError` naming the `.config` spelling — the tier is never guessed from
whether a property happens to exist.
The two tiers are **not** two spellings of one number. Live diverges from configured
wherever elastic EP scales the world away from the launch shape, and wherever
`initialize_model_parallel` aliases `_MOE_DP` to `_ATTN_CP` (`attn_cp_size >
moe_dp_size`), which makes a live comparison of that pair degenerate. The five
live-shadowed sizes (`tp/pp/dcp/attn_cp/moe_dp_size`) are where the choice matters,
and every business read of `get_parallel().config.<one of them>` is registered with
its reason in `_CONFIGURED_SIZE_CALL_SITES` (`test_global_config_read_ratchet.py`).
DCP has a third shape: the live `get_parallel().attn_dcp_size` / `.dcp_enabled`
answer the *effective* topology (`1` / `False` with no group installed), never the
requested size — `.config.dcp_size` is the requested one.
A process-global seed field-read of one of these sizes
(`get_server_args().tp_size`, or an alias of it) is a read-ratchet failure. A
`server_args` the object was *handed* is a different thing and not a ratchet
matter — see "Reads that legitimately stay on a ServerArgs instance".
Fail-loud is narrower: before dist init, a live size/group read raises — except
@@ -238,25 +244,31 @@ the DCP pair, which degrades instead (`dcp_enabled` → `False`,
`attn_dcp_size` → `1` when no group is installed;
`test_attn_dcp_defaults_when_group_is_uninitialized` pins this). After init,
only the DCP group is optional (`_DCP` exists only when `dcp_size > 1`; attn-CP and
moe-DP always install, as size-1 aliases if unused). `ParallelContext.__getattr__` is deliberately dynamo-traceable (no
moe-DP always install, as size-1 aliases if unused). The `config` hop is
deliberately dynamo-traceable (a plain property over a slot, no
`object.__getattribute__`); gate helpers like `enable_moe_dense_fully_dp()` run inside
compiled model forwards (`test_parallel_config_leaves_trace_under_torch_compile` pins
this).
A third surface carries the same names: `ParallelState` (`self.ps` / `mr.ps`), the
frozen per-process snapshot built once in `Scheduler.__init__` from these configured
sizes plus this process's ranks, and handed down (draft runners included). Prefer it
where an object was handed one; it is not a global accessor.
### Reading config: the seed is off limits
`get_server_args().field` in business code is a ratchet failure. Read:
- **a resolved leaf** → its namespace bag (`get_exec().moe.moe_runner_backend`,
`get_schedule().chunked_prefill_size`, …). Bag-backed reads — a leaf directly, or
a bag-derived accessor below, including `configured_*_size()` which reads the
parallel bag's own leaf — are what see post-publish overrides. Only the
a bag-derived accessor below, including the `get_parallel().config` hop — are
what see post-publish overrides. Only the
instance-derived accessors (the ones with no leaf to read) answer from the
startup record and therefore do not.
- **a leaf the caller names at runtime** (a readback reporting a list of fields)
→ `get_context().config_leaf(name)`; it resolves the name through `NS` and
raises on a non-leaf. A call site that knows its field reads the bag leaf.
- **the live topology** → `get_parallel()`.
- **the live topology** → `get_parallel()` (bare names).
- **a value derived from published leaves** → an accessor in `runtime_context` that
derives it *from the bags*: `mamba_extra_buffer_enabled()` /
`mamba_extra_buffer_lazy_enabled()` read `get_memory()` and `get_exec()`, so
@@ -275,24 +287,18 @@ this).
property with no bag of its own. A new derived member gets an accessor here
rather than call sites reaching for the record, and only when the bag-derived
shape above cannot express it.
- **what was *configured*, where `get_parallel()` shadows it with the live value**
→ `configured_{tp,pp,moe_dp,attn_cp}_size()` — the full names are
`configured_tp_size`, `configured_pp_size`, `configured_moe_dp_size`,
`configured_attn_cp_size`. They read the parallel bag's own leaf (going
around the live property that shadows those four names), so they answer with
the resolved configuration and follow a post-publish override. DCP has no configured accessor because no
config-intent DCP call site exists today — the live reads go through
`get_parallel().attn_dcp_size` / `.dcp_enabled`, which answer the effective
topology (`1` / `False` when no group is installed). A site
that must know the *requested* DCP size before dist init needs its own
`configured_dcp_size()` (and an entry in `_CONFIGURED_SIZE_CALL_SITES`, which lives
in the ratchet test, not in this skill); note the live pair does not *need*
dist init — with no group it answers `1` / `False` — it just cannot answer
with the requested size. Every (file, accessor) pair is registered
- **what was *configured*, where the bare name is the live value**
→ `get_parallel().config.{tp,pp,moe_dp,attn_cp,dcp}_size`. It reads the parallel
bag's own leaf, so it answers with the resolved configuration and follows a
post-publish override. The DCP live pair (`get_parallel().attn_dcp_size` /
`.dcp_enabled`) is a different question again: it answers the effective topology
(`1` / `False` when no group is installed), never the requested size, and it does
not *need* dist init to answer. Every (file, size) pair is registered
with its reason in `test_global_config_read_ratchet.py`
(`_CONFIGURED_SIZE_CALL_SITES`), and that test fails if the code and the list
disagree — a new file, or a new accessor in a listed file, has to be added — so a new site needs both an answer the live property cannot give and
an entry saying what it is.
disagree — a new file, or a new size in a listed file, has to be added — so a new
site needs both an answer the live property cannot give and an entry saying what
it is.
- **this runner's resolved value** → the runner
(`prefill_attention_backend_str`, `kv_cache_dtype_str`,
`draft_attention_backend`, `num_fused_shared_experts` on the model).
@@ -498,13 +504,19 @@ ONE thread — do not design for TBO threads that don't exist.
— including local copies of an alias, `cfg = sa` — module-level, or parked on an
instance attribute, plus the `getattr(..., "field")` spelling of each; a name
computed at runtime or indirection deeper than a local name copy is census-tool
territory, per the test's docstring). The scanners match `get_server_args` and
`configured_*_size` by their literal names, and the same file *bans*
`import ... as` renames of them so that matching stays sound. Exempt by owner
territory, per the test's docstring). The scanner matches `get_server_args` by its
literal name, and the same file *bans* `import ... as` renames of it so that
matching stays sound. Exempt by owner
module only (`runtime_context.py`, `server_args.py`, `arg_groups/`). The same file
carries `_CONFIGURED_SIZE_CALL_SITES`, the (file, accessor) map of every
`configured_*_size()` reader with the reason the live property cannot serve it — a new
file or a new accessor in a listed file must be added there.
carries `_CONFIGURED_SIZE_CALL_SITES`, the (file, size) map of every
`get_parallel().config.<live-shadowed size>` reader with the reason the live property
cannot serve it — a new file or a new size in a listed file must be added there. Its
subject set is *derived* (property names ∩ `parallel` NS leaves), and it resolves
every spelling of the call itself — an aliased import, a module-qualified receiver
(including the whole dotted path an unaliased `import` binds), a local bound to either
hop — so neither a rename nor a new shadowed size escapes it.
`TestParallelConfigReadSpellings` in that file runs each spelling, because a spelling
the scanner cannot resolve drops the read instead of failing anything.
6. **Module-state ratchet** (`test_module_state_ratchet.py`): `global` statements in the
flag-owning layers are pinned by name. A new module-level runtime global belongs on a
flags group / resources slot instead; migrating a pinned survivor must shrink the pin.
@@ -538,8 +550,9 @@ Never module-skip a test "until the migration settles" — seed the context inst
form** (attribute-source ints get automatic-dynamic after the first size
change). Bools (≤2 values) are tolerable in any form — see
`ForwardFlags._GRAPH_VISIBLE`. Config-bag leaves are real instance attributes for
exactly this reason, and `ParallelContext.__getattr__` must stay free of
`object.__getattribute__` (dynamo graph-breaks on it). Before moving such state,
exactly this reason, and the parallel config tier is read through the plain
`ParallelContext.config` property for the same reason (`__getattr__` is
error-only, and `object.__getattribute__` graph-breaks). Before moving such state,
prove its readers sit outside compile coverage; a piecewise-prefill boot of a small
model is the fast check (recompile storms show as `torch._dynamo hit
config.recompile_limit` during the compile pass).
python/sglang/benchmark/one_batch.py
+1 -1
View File
@@ -542,7 +542,7 @@ def _maybe_prepare_mlp_sync_batch(batch: ScheduleBatch, model_runner):
prepare_mlp_sync_batch_raw(
batch,
model_runner=model_runner,
dp_size=get_parallel().dp_size,
dp_size=get_parallel().config.dp_size,
attn_tp_size=get_parallel().attn_tp_size,
attn_cp_size=model_runner.ps.attn_cp_size,
tp_group=model_runner.tp_group,
python/sglang/srt/batch_overlap/two_batch_overlap.py
+1 -1
View File
@@ -769,7 +769,7 @@ class TboForwardBatchPreparer:
# TODO improve, e.g. unify w/ `init_raw`
if (
get_parallel().moe_dense_tp_size == 1
get_parallel().config.moe_dense_tp_size == 1
and batch.global_dp_buffer_len is not None
):
sum_len = end_token_index - start_token_index
python/sglang/srt/disaggregation/common/conn.py
+17 -14
View File
@@ -37,7 +37,6 @@ from sglang.srt.layers.dp_attention import (
get_attention_dp_size,
)
from sglang.srt.runtime_context import (
configured_pp_size,
get_disagg,
get_parallel,
get_serving,
@@ -171,7 +170,7 @@ class CommonKVManager(BaseKVManager):
# for p/d multi node infer
self.bootstrap_host = get_serving().host
self.bootstrap_port = get_disagg().disaggregation_bootstrap_port
self.dist_init_addr = get_parallel().dist_init_addr
self.dist_init_addr = get_parallel().config.dist_init_addr
parallel = get_parallel()
self.attn_tp_size = parallel.attn_tp_size
self.attn_tp_rank = parallel.attn_tp_rank
@@ -182,16 +181,19 @@ class CommonKVManager(BaseKVManager):
self.attn_dp_size = get_attention_dp_size()
self.attn_dp_rank = get_attention_dp_rank()
self.system_dp_size = (
1 if get_parallel().enable_dp_attention else get_parallel().dp_size
1
if get_parallel().config.enable_dp_attention
else get_parallel().config.dp_size
)
self.system_dp_rank = (
self.kv_args.system_dp_rank if self.kv_args.system_dp_rank else 0
)
self.pp_size = configured_pp_size()
self.pp_size = get_parallel().config.pp_size
self.pp_rank = self.kv_args.pp_rank
self.local_ip = get_local_ip_auto()
cp_sharded_prefill = self.attn_cp_size > 1 and (
self.is_hybrid_mla_backend or get_parallel().enable_dsa_cache_layer_split
self.is_hybrid_mla_backend
or get_parallel().config.enable_dsa_cache_layer_split
)
hybrid_decode_pulls_all_ranks = (
@@ -304,7 +306,7 @@ class CommonKVManager(BaseKVManager):
return (
self.attn_cp_size > 1
and self.attn_cp_rank != 0
and not get_parallel().enable_dsa_cache_layer_split
and not get_parallel().config.enable_dsa_cache_layer_split
)
def requires_dcp_relayout(self, dst_dcp_size: int, dst_dcp_rank: int) -> bool:
@@ -749,7 +751,7 @@ class CommonKVManager(BaseKVManager):
`Connection refused`, and the leader's `prefill_port_table` ends
up missing rows.
"""
if not self.dist_init_addr or get_parallel().nnodes == 1:
if not self.dist_init_addr or get_parallel().config.nnodes == 1:
return local_port
if not (dist.is_available() and dist.is_initialized()):
@@ -801,8 +803,8 @@ class CommonKVManager(BaseKVManager):
"rank_port": self.rank_port,
"page_size": self.kv_args.page_size,
"kv_cache_dtype": self.kv_cache_dtype_str,
"load_balance_method": get_parallel().load_balance_method,
"enable_dsa_cache_layer_split": get_parallel().enable_dsa_cache_layer_split,
"load_balance_method": get_parallel().config.load_balance_method,
"enable_dsa_cache_layer_split": get_parallel().config.enable_dsa_cache_layer_split,
# Self-register the HTTP API port so the decode can derive the PD
# retract rebootstrap /generate URL from bootstrap info instead of a
# router-injected pd_rebootstrap_prefill_url.
@@ -1184,11 +1186,12 @@ class CommonKVSender(BaseKVSender):
return
self.kv_mgr.update_status(self.bootstrap_room, KVPoll.Bootstrapping)
if get_parallel().dp_size > 1 and not req_has_disagg_prefill_dp_rank:
if get_parallel().load_balance_method != "follow_bootstrap_room":
if get_parallel().config.dp_size > 1 and not req_has_disagg_prefill_dp_rank:
if get_parallel().config.load_balance_method != "follow_bootstrap_room":
self._register_prefill_dp_rank()
elif (
self.kv_mgr.attn_dp_rank != self.bootstrap_room % get_parallel().dp_size
self.kv_mgr.attn_dp_rank
!= self.bootstrap_room % get_parallel().config.dp_size
):
# follow_bootstrap_room was overridden by external routed_dp_rank
if envs.SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK.get():
@@ -1199,7 +1202,7 @@ class CommonKVSender(BaseKVSender):
f"follow_bootstrap_room conflict: dispatched to dp_rank "
f"{self.kv_mgr.attn_dp_rank} but bootstrap_room "
f"{self.bootstrap_room} implies dp_rank "
f"{self.bootstrap_room % get_parallel().dp_size}. "
f"{self.bootstrap_room % get_parallel().config.dp_size}. "
f"Set SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK=1 "
f"to allow mixed routing.",
)
@@ -1273,7 +1276,7 @@ class CommonKVSender(BaseKVSender):
if (
self.kv_mgr.enable_all_cp_ranks_for_transfer
and not get_parallel().enable_dsa_cache_layer_split
and not get_parallel().config.enable_dsa_cache_layer_split
):
kv_indices, index_slice = filter_kv_indices_for_cp_rank(
self.kv_mgr,
python/sglang/srt/disaggregation/encoder/http_server.py
+1 -1
View File
@@ -207,7 +207,7 @@ def launch_server(server_args: ServerArgs):
configure_logger(server_args, prefix=" encode_server")
publish(server_args, role="encoder")
if get_parallel().dp_size > 1:
if get_parallel().config.dp_size > 1:
dp_dispatcher = launch_dp_runtime(server_args)
# runtime initializes multiprocess metrics before spawning;
# HTTP only exposes their endpoint.
python/sglang/srt/disaggregation/encoder/runtime.py
+10 -9
View File
@@ -50,7 +50,6 @@ from sglang.srt.observability.trace import (
trace_set_thread_info,
)
from sglang.srt.runtime_context import (
configured_tp_size,
get_observability,
get_parallel,
get_serving,
@@ -1491,10 +1490,10 @@ def launch_local_runtime(server_args: ServerArgs) -> EncoderRuntime:
This function owns backend construction only. HTTP/gRPC middleware,
service registration, and network serving remain Transport concerns.
"""
if get_parallel().dp_size > 1:
if get_parallel().config.dp_size > 1:
raise ValueError(
"launch_local_runtime requires --dp-size 1; got "
f"dp_size={get_parallel().dp_size}."
f"dp_size={get_parallel().config.dp_size}."
)
# Set up prometheus metrics.
@@ -1512,8 +1511,10 @@ def launch_local_runtime(server_args: ServerArgs) -> EncoderRuntime:
zmq_context = zmq.Context(10)
ipc_path_prefix = random_uuid()
port_args = PortArgs.init_new(server_args)
if get_parallel().dist_init_addr:
dist_init_method = NetworkAddress.parse(get_parallel().dist_init_addr).to_tcp()
if get_parallel().config.dist_init_addr:
dist_init_method = NetworkAddress.parse(
get_parallel().config.dist_init_addr
).to_tcp()
else:
dist_init_method = NetworkAddress(
get_serving().host or "127.0.0.1", port_args.nccl_port
@@ -1529,7 +1530,7 @@ def launch_local_runtime(server_args: ServerArgs) -> EncoderRuntime:
send_sockets: List[zmq.Socket] = []
tp_processes: List[mp.Process] = []
for rank in range(1, configured_tp_size()):
for rank in range(1, get_parallel().config.tp_size):
schedule_path = f"ipc:///tmp/{ipc_path_prefix}_schedule_{rank}"
send_sockets.append(
get_zmq_socket(zmq_context, zmq.PUSH, schedule_path, bind=False)
@@ -1569,12 +1570,12 @@ def launch_dp_runtime(server_args: ServerArgs) -> DPDispatcher:
HTTP uses this entry point today. gRPC can reuse it later without
importing HTTP application state or Uvicorn.
"""
if get_parallel().dp_size <= 1 or server_args.tp_size != 1:
if get_parallel().config.dp_size <= 1 or server_args.tp_size != 1:
raise ValueError(
"Encoder DP mode requires --dp-size > 1 and --tp-size 1; got "
f"dp_size={get_parallel().dp_size}, tp_size={server_args.tp_size}."
f"dp_size={get_parallel().config.dp_size}, tp_size={server_args.tp_size}."
)
dp_size = get_parallel().dp_size
dp_size = get_parallel().config.dp_size
logger.info(f"Launching encoder in DP mode: dp_size={dp_size}")
# DP mode: workers (subprocesses) write metrics to the shared multiproc dir;
python/sglang/srt/disaggregation/prefill.py
+1 -1
View File
@@ -180,7 +180,7 @@ class PrefillBootstrapQueue:
"SGLANG_DISAGG_STAGING_BUFFER with pp_size > 1 is only "
"supported by Mooncake."
)
if get_parallel().enable_prefill_context_parallel:
if get_parallel().config.enable_prefill_context_parallel:
# CP rewrites index_slice per rank, breaking the chunk grid.
raise RuntimeError(
"SGLANG_DISAGG_STAGING_BUFFER does not support "
python/sglang/srt/distributed/bootstrap.py
+2 -2
View File
@@ -128,7 +128,7 @@ def init_torch_distributed(
# included in later KV-cache sizing instead of appearing during capture.
if (
device == "cuda"
and get_parallel().enable_tp_lm_head_all_to_all
and get_parallel().config.enable_tp_lm_head_all_to_all
and ps.tp_size > 1
):
_prewarm_tp_lm_head_all_to_all()
@@ -266,7 +266,7 @@ def _init_parallel_groups(
duplicate_attn_cp_group=(
is_hip()
and server_args.enable_two_batch_overlap
and get_parallel().enable_dsa_prefill_context_parallel
and get_parallel().config.enable_dsa_prefill_context_parallel
),
enable_symm_mem=get_exec().comm.enable_symm_mem,
recovered_rank=is_ep_joiner,
python/sglang/srt/distributed/device_communicators/triton_symm_mem_ag.py
+1 -1
View File
@@ -477,7 +477,7 @@ class MultimemAllGatherer:
# EP/mooncake setups, and keep multimem enabled.
if (
tp_group.world_size > 1
and get_parallel().nnodes > 1
and get_parallel().config.nnodes > 1
and not all(in_the_same_node_as(tp_group.cpu_group, source_rank=0))
):
logger.warning(
python/sglang/srt/elastic_ep/elastic_ep.py
+7 -7
View File
@@ -12,7 +12,6 @@ from sglang.srt.distributed.utils import get_global_tcp_store
from sglang.srt.eplb.expert_location import broadcast_global_expert_location_metadata
from sglang.srt.managers.schedule_batch import ServerArgs
from sglang.srt.runtime_context import (
configured_tp_size,
get_exec,
get_parallel,
)
@@ -93,7 +92,7 @@ class ElasticEPStateManager:
if get_exec().moe.elastic_ep_backend is not None:
world_size = torch.distributed.get_world_size()
active_rank_capacity = get_parallel().max_ep_size or world_size
active_rank_capacity = get_parallel().config.max_ep_size or world_size
assert active_rank_capacity >= world_size, (
f"--max-ep-size ({active_rank_capacity}) must be >= "
f"world_size ({world_size})."
@@ -110,7 +109,7 @@ class ElasticEPStateManager:
if get_exec().moe.moe_a2a_backend == "nixl":
cls._on_scale = cls._on_scale_nixl
inst.ep_join_rank_offset = get_parallel().ep_join_rank_offset
inst.ep_join_rank_offset = get_parallel().config.ep_join_rank_offset
if server_args.is_ep_joiner:
cls._init_joiner_state(inst, server_args)
@@ -128,11 +127,12 @@ class ElasticEPStateManager:
if get_exec().moe.ep_join_mode == "scale":
inst.effective_ep_size = (
get_parallel().ep_join_rank_offset + configured_tp_size()
get_parallel().config.ep_join_rank_offset
+ get_parallel().config.tp_size
)
inst.original_ep_size = (
get_parallel().elastic_ep_initial_size
or get_parallel().ep_join_rank_offset
get_parallel().config.elastic_ep_initial_size
or get_parallel().config.ep_join_rank_offset
)
inst.has_scaled = True
else:
@@ -317,7 +317,7 @@ def elastic_expanded_world_enabled() -> bool:
inst = ElasticEPStateManager.instance()
if inst is None:
return False
if get_parallel().max_ep_size is None:
if get_parallel().config.max_ep_size is None:
return False
active_target_size = inst.effective_ep_size
if inst.pending_ep_size is not None and inst.scale_phase in (
python/sglang/srt/elastic_ep/expert_backup_manager.py
+5 -6
View File
@@ -18,7 +18,6 @@ from sglang.srt.managers.io_struct import (
from sglang.srt.model_loader.loader import DefaultModelLoader, get_model_loader
from sglang.srt.model_loader.utils import set_default_torch_dtype
from sglang.srt.runtime_context import (
configured_tp_size,
get_disagg,
get_exec,
get_model,
@@ -51,8 +50,8 @@ class ExpertBackupManager:
self.weight_pointer_map = {}
self.transfer_engine = None
self.session_id = None
self.engine_num = get_parallel().nnodes
self.engine_rank = get_parallel().node_rank
self.engine_num = get_parallel().config.nnodes
self.engine_rank = get_parallel().config.node_rank
self.expert_num = self.model_config.hf_config.n_routed_experts
self.idmn = (self.expert_num // self.engine_num) * self.engine_rank
self.idmx = (self.expert_num // self.engine_num) * (self.engine_rank + 1)
@@ -60,11 +59,11 @@ class ExpertBackupManager:
# Synchronization socket to avoid PUB/SUB slow joiner issues.
self.recv_from_expert_backup_client = context.socket(zmq.PULL)
self.recv_from_expert_backup_client.bind(
f"tcp://{get_local_ip_auto()}:{PORT_BASE + get_parallel().node_rank * 2}"
f"tcp://{get_local_ip_auto()}:{PORT_BASE + get_parallel().config.node_rank * 2}"
)
self.send_to_expert_backup_client = context.socket(zmq.PUB)
self.send_to_expert_backup_client.bind(
f"tcp://{get_local_ip_auto()}:{PORT_BASE + get_parallel().node_rank * 2 + 1}"
f"tcp://{get_local_ip_auto()}:{PORT_BASE + get_parallel().config.node_rank * 2 + 1}"
)
self.backup_weights_from_disk()
self.start_transfer_server()
@@ -73,7 +72,7 @@ class ExpertBackupManager:
# losing the initial PUB message due to slow joiners.
num_ready_clients = 0
while num_ready_clients < configured_tp_size():
while num_ready_clients < get_parallel().config.tp_size:
sock_recv(self.recv_from_expert_backup_client)
num_ready_clients += 1
python/sglang/srt/entrypoints/engine.py
+12 -11
View File
@@ -98,9 +98,6 @@ from sglang.srt.parser.template_detection import resolve_auto_parsers
from sglang.srt.parser.template_manager import TemplateManager
from sglang.srt.plugins import load_plugins
from sglang.srt.runtime_context import (
configured_attn_cp_size,
configured_moe_dp_size,
configured_pp_size,
get_exec,
get_model,
get_parallel,
@@ -345,7 +342,7 @@ class Engine(EngineScoreMixin, EngineBase):
routed_dp_rank = data_parallel_rank
if routed_dp_rank is not None:
dp_size = get_parallel().dp_size
dp_size = get_parallel().config.dp_size
if dp_size <= 1 and routed_dp_rank == 0:
logger.debug(
f"routed_dp_rank={routed_dp_rank} is ignored because dp_size={dp_size}"
@@ -684,7 +681,7 @@ class Engine(EngineScoreMixin, EngineBase):
pp_rank_range, tp_rank_range, pp_size_per_node, tp_size_per_node = (
_calculate_rank_ranges(
server_args.nnodes,
configured_pp_size(),
get_parallel().config.pp_size,
tp_size,
server_args.node_rank,
)
@@ -832,7 +829,7 @@ class Engine(EngineScoreMixin, EngineBase):
"""
scheduler_procs = []
use_dp_controller = (
get_parallel().dp_size > 1 or get_exec().moe.ep_join_mode == "scale"
get_parallel().config.dp_size > 1 or get_exec().moe.ep_join_mode == "scale"
)
if not use_dp_controller:
@@ -845,7 +842,7 @@ class Engine(EngineScoreMixin, EngineBase):
pp_rank_range, tp_rank_range, pp_size_per_node, tp_size_per_node = (
_calculate_rank_ranges(
server_args.nnodes,
configured_pp_size(),
get_parallel().config.pp_size,
server_args.tp_size,
server_args.node_rank,
)
@@ -1845,10 +1842,14 @@ def _compute_parallelism_ranks(
Called while the launcher is deciding what to spawn, so the sizes are the
configured ones -- the groups this is laying out do not exist yet.
"""
attn_dp_size = get_parallel().dp_size if get_parallel().enable_dp_attention else 1
attn_dp_size = (
get_parallel().config.dp_size
if get_parallel().config.enable_dp_attention
else 1
)
tp_size = server_args.tp_size
attn_cp_size = configured_attn_cp_size()
moe_dp_size = configured_moe_dp_size()
attn_cp_size = get_parallel().config.attn_cp_size
moe_dp_size = get_parallel().config.moe_dp_size
# Parallelism hierarchy (outermost to innermost):
# - Attention: Global(TP) -> DP -> ATTN_CP -> ATTN_TP (innermost)
@@ -1859,6 +1860,6 @@ def _compute_parallelism_ranks(
moe_ep_rank = (
tp_rank
% (tp_size // moe_dp_size)
// (tp_size // moe_dp_size // get_parallel().ep_size)
// (tp_size // moe_dp_size // get_parallel().config.ep_size)
)
return attn_cp_rank, moe_dp_rank, moe_ep_rank
python/sglang/srt/entrypoints/http_server.py
+10 -6
View File
@@ -2177,7 +2177,7 @@ async def _send_disaggregation_warmup_requests(
return await asyncio.gather(
*(
send_request(session, dp_rank)
for dp_rank in range(get_parallel().dp_size)
for dp_rank in range(get_parallel().config.dp_size)
)
)
@@ -2236,9 +2236,11 @@ def _execute_server_warmup(server_args: ServerArgs):
},
}
if server_args.skip_tokenizer_init:
json_data["input_ids"] = [[10, 11, 12] for _ in range(get_parallel().dp_size)]
json_data["input_ids"] = [
[10, 11, 12] for _ in range(get_parallel().config.dp_size)
]
# TODO Workaround the bug that embedding errors for list of size 1
if get_parallel().dp_size == 1:
if get_parallel().config.dp_size == 1:
json_data["input_ids"] = json_data["input_ids"][0]
elif (
is_vlm
@@ -2282,9 +2284,11 @@ def _execute_server_warmup(server_args: ServerArgs):
"temperature": 0.0,
}
else:
json_data["text"] = ["The capital city of France is"] * get_parallel().dp_size
json_data["text"] = [
"The capital city of France is"
] * get_parallel().config.dp_size
# TODO Workaround the bug that embedding errors for list of size 1
if get_parallel().dp_size == 1:
if get_parallel().config.dp_size == 1:
json_data["text"] = json_data["text"][0]
# Config debug dumping
@@ -2326,7 +2330,7 @@ def _execute_server_warmup(server_args: ServerArgs):
if not failed_status_codes:
logger.info(
"Disaggregation warmup requests completed for all %s DP ranks",
get_parallel().dp_size,
get_parallel().config.dp_size,
)
logger.info("End of disaggregation warmup")
else:
python/sglang/srt/entrypoints/v1_loads.py
+3 -4
View File
@@ -27,7 +27,6 @@ from fastapi import APIRouter, Depends, HTTPException
from fastapi.responses import Response
from sglang.srt.runtime_context import (
configured_pp_size,
get_parallel,
)
from sglang.srt.utils import get_device_name
@@ -148,9 +147,9 @@ async def get_loads(
"accelerator": _accelerator_name(),
"num_accelerators": _num_accelerators_per_dp_rank(
tokenizer_manager.server_args.tp_size,
configured_pp_size(),
get_parallel().dp_size,
get_parallel().enable_dp_attention,
get_parallel().config.pp_size,
get_parallel().config.dp_size,
get_parallel().config.enable_dp_attention,
),
"loads": loads,
}
python/sglang/srt/eplb/expert_location.py
+9 -7
View File
@@ -26,7 +26,6 @@ import torch.distributed
import torch.nn.functional as F
from sglang.srt.runtime_context import (
configured_tp_size,
get_device,
get_exec,
)
@@ -202,7 +201,7 @@ class ExpertLocationMetadata:
model_config_for_expert_location = common["model_config_for_expert_location"]
num_physical_experts = common["num_physical_experts"]
num_groups = model_config_for_expert_location.num_groups
num_nodes = 1 if use_flat_topology else get_parallel().nnodes
num_nodes = 1 if use_flat_topology else get_parallel().config.nnodes
from sglang.srt.eplb import eplb_algorithms
@@ -245,14 +244,15 @@ class ExpertLocationMetadata:
+ get_exec().moe.ep_num_redundant_experts
)
# elastic-EP scale-up rewrites ep_size on the published config
ep_size = get_parallel().ep_size
ep_size = get_parallel().config.ep_size
num_physical_experts = base_num_physical_experts
initial_ep_size = get_parallel().elastic_ep_initial_size
initial_ep_size = get_parallel().config.elastic_ep_initial_size
if initial_ep_size is not None:
if get_exec().moe.ep_join_mode == "scale":
ep_size = max(
ep_size,
get_parallel().ep_join_rank_offset + configured_tp_size(),
get_parallel().config.ep_join_rank_offset
+ get_parallel().config.tp_size,
)
num_physical_experts, num_local_physical_experts = (
_compute_elastic_expert_layout(
@@ -580,7 +580,7 @@ def _compute_logical_to_all_physical_map(
num_local_gpu_physical_experts = num_physical_experts // ep_size
prefer_same_node = _prefer_same_node_experts()
num_gpus_per_node = (
get_parallel().ep_size // get_parallel().nnodes
get_parallel().config.ep_size // get_parallel().config.nnodes
if prefer_same_node
else None
)
@@ -644,7 +644,9 @@ def compute_logical_to_rank_dispatch_physical_map(
num_local_gpu_physical_experts = num_physical_experts // ep_size
prefer_same_node = _prefer_same_node_experts()
num_gpus_per_node = (
get_parallel().ep_size // get_parallel().nnodes if prefer_same_node else None
get_parallel().config.ep_size // get_parallel().config.nnodes
if prefer_same_node
else None
)
num_local_node_physical_experts = (
num_local_gpu_physical_experts * num_gpus_per_node
python/sglang/srt/layers/attention/dsa/dsa_indexer.py
+1 -2
View File
@@ -40,7 +40,6 @@ from sglang.srt.model_executor.runner_backend_utils.tc_piecewise_cuda_graph impo
is_in_tc_piecewise_cuda_graph,
)
from sglang.srt.runtime_context import (
configured_pp_size,
get_device,
get_exec,
get_parallel,
@@ -249,7 +248,7 @@ class Indexer(DSANPUIndexerMixin, BaseFusedOp):
if _is_cuda:
self.sm_count = deep_gemm.get_num_sms()
self.half_device_sm_count = ceil_align(self.sm_count // 2, 8)
pp_size = configured_pp_size()
pp_size = get_parallel().config.pp_size
self.logits_with_pp_recv = pp_size > 1 and not get_pp_group().is_last_rank
else:
self.logits_with_pp_recv = False
python/sglang/srt/layers/attention/dsa/utils.py
+3 -3
View File
@@ -106,7 +106,7 @@ def should_use_dsa_fused_topk(seed_dsa_topk_from_draft_extend: bool) -> bool:
def is_dsa_enable_prefill_cp():
if not envs.SGLANG_ENABLE_CP_V2.get():
return get_parallel().enable_dsa_prefill_context_parallel
return get_parallel().config.enable_dsa_prefill_context_parallel
# Derive from the runtime CP topology + model arch rather than the legacy
# flag under CP-v2: DSA prefill CP is active when the CP group is on for a
@@ -122,14 +122,14 @@ def is_dsa_enable_prefill_cp():
def is_dsa_prefill_cp_in_seq_split():
return (
is_dsa_enable_prefill_cp()
and get_parallel().dsa_prefill_cp_mode == "in-seq-split"
and get_parallel().config.dsa_prefill_cp_mode == "in-seq-split"
)
def is_dsa_prefill_cp_round_robin_split():
return (
is_dsa_enable_prefill_cp()
and get_parallel().dsa_prefill_cp_mode == "round-robin-split"
and get_parallel().config.dsa_prefill_cp_mode == "round-robin-split"
)
python/sglang/srt/layers/attention/flashattention_backend.py
+4 -1
View File
@@ -57,7 +57,10 @@ from sglang.kernels.ops.attention.flash_attention import (
def _should_disable_scheduler_metadata_precompute() -> bool:
return bool(get_parallel().enable_prefill_cp or get_parallel().enable_dp_attention)
return bool(
get_parallel().config.enable_prefill_cp
or get_parallel().config.enable_dp_attention
)
@dataclass
python/sglang/srt/layers/communicator.py
+4 -4
View File
@@ -269,7 +269,7 @@ class AttnTpContext:
def init_context(self, q_lora_rank, is_dsa):
self.is_dsa = is_dsa
self.allow_input_scattered = (
get_parallel().enable_attn_tp_input_scattered
get_parallel().config.enable_attn_tp_input_scattered
and (_is_cuda or _is_npu)
and q_lora_rank is not None
and not is_dsa
@@ -280,7 +280,7 @@ class AttnTpContext:
and not check_cuda_graph_backend(Phase.PREFILL, Backend.TC_PIECEWISE)
and get_spec().speculative_algorithm != "EAGLE3"
)
if get_parallel().enable_attn_tp_input_scattered:
if get_parallel().config.enable_attn_tp_input_scattered:
if not self.allow_input_scattered:
logging.info(
"attn_tp_input_scattered is not enabled while other conditions are not met"
@@ -438,11 +438,11 @@ class LayerScatterModes:
def enable_moe_dense_fully_dp():
return get_parallel().moe_dense_tp_size == 1
return get_parallel().config.moe_dense_tp_size == 1
def enable_dwdp():
return get_parallel().dwdp_size > 1
return get_parallel().config.dwdp_size > 1
class LayerCommunicator:
python/sglang/srt/layers/cp/base.py
+1 -1
View File
@@ -283,7 +283,7 @@ def get_cp_strategy() -> Optional[ContextParallelStrategy]:
server_args = get_server_args()
except ValueError:
return None
if server_args is not None and get_parallel().enable_prefill_cp:
if server_args is not None and get_parallel().config.enable_prefill_cp:
init_cp_strategy(server_args)
return _STRATEGY
python/sglang/srt/layers/cp/cp_decode_attn_tp.py
+1 -1
View File
@@ -51,7 +51,7 @@ class CpDecodeAttnTpContext:
"""Slices replicated attention weights across CP ranks during decode."""
def __init__(self):
enable_attn_tp = get_parallel().enable_cp_decode_attn_tp
enable_attn_tp = get_parallel().config.enable_cp_decode_attn_tp
if enable_attn_tp and get_parallel().attn_cp_size > 1:
self.decode_tp_rank = get_parallel().attn_cp_rank
python/sglang/srt/layers/dp_attention.py
+9 -8
View File
@@ -29,8 +29,6 @@ from sglang.srt.distributed.device_communicators.pynccl_allocator import (
use_symmetric_memory,
)
from sglang.srt.runtime_context import (
configured_attn_cp_size,
configured_moe_dp_size,
get_device,
get_exec,
get_flags,
@@ -349,9 +347,9 @@ def initialize_dp_attention(
dp.max_len_with_idle = (
getattr(model_config.hf_config, "hybrid_override_pattern", None) is not None
)
enable_dp_attention = get_parallel().enable_dp_attention
dp_size = get_parallel().dp_size
attn_cp_size = configured_attn_cp_size()
enable_dp_attention = get_parallel().config.enable_dp_attention
dp_size = get_parallel().config.dp_size
attn_cp_size = get_parallel().config.attn_cp_size
dp.enabled = enable_dp_attention
@@ -363,8 +361,11 @@ def initialize_dp_attention(
)
_ATTN_DP_SIZE = dp_size if enable_dp_attention else 1
if get_exec().moe.elastic_ep_backend is not None and get_parallel().max_ep_size:
_ATTN_DP_RANK = tp_rank + get_parallel().ep_join_rank_offset
if (
get_exec().moe.elastic_ep_backend is not None
and get_parallel().config.max_ep_size
):
_ATTN_DP_RANK = tp_rank + get_parallel().config.ep_join_rank_offset
if server_args.is_ep_scale_joiner:
dp.joiner_skip_all_gather = True
@@ -1033,7 +1034,7 @@ def is_enable_moe_cp_allgather() -> bool:
(``parallel_state.py``), so the live sizes are equal and the comparison would
always be false.
"""
return configured_attn_cp_size() > configured_moe_dp_size()
return get_parallel().config.attn_cp_size > get_parallel().config.moe_dp_size
def moe_cp_all_gather_into_tensor(output: torch.Tensor, input: torch.Tensor):
python/sglang/srt/layers/logits_processor.py
+4 -2
View File
@@ -296,8 +296,10 @@ class LogitsProcessor(nn.Module):
self.config = config
self.vocab_size = config.vocab_size
self.logit_scale = logit_scale
self.use_attn_tp_group = get_parallel().enable_dp_lm_head
self.use_tp_lm_head_all_to_all = get_parallel().enable_tp_lm_head_all_to_all
self.use_attn_tp_group = get_parallel().config.enable_dp_lm_head
self.use_tp_lm_head_all_to_all = (
get_parallel().config.enable_tp_lm_head_all_to_all
)
self.use_fp32_lm_head = get_exec().features.enable_fp32_lm_head
if self.use_attn_tp_group:
self.attn_tp_size = get_parallel().attn_tp_size
python/sglang/srt/layers/moe/fused_moe_triton/layer.py
+2 -2
View File
@@ -293,10 +293,10 @@ class FusedMoE(torch.nn.Module):
self._num_global_routed = num_experts - num_shared_slots
if get_exec().moe.ep_join_mode == "scale":
storage_ep_size = get_parallel().elastic_ep_initial_size
storage_ep_size = get_parallel().config.elastic_ep_initial_size
assert storage_ep_size is not None
self._expert_storage_rank = (
get_parallel().ep_join_rank_offset + self.moe_ep_rank
get_parallel().config.ep_join_rank_offset + self.moe_ep_rank
)
else:
storage_ep_size = self.moe_ep_size
python/sglang/srt/layers/moe/moe_runner/flashinfer_cutedsl.py
+1 -1
View File
@@ -337,7 +337,7 @@ def ensure_cutedsl_wrapper(layer: torch.nn.Module) -> None:
else:
# Standard allgather path: the MoE sees up to dp_size local forwards
# gathered together, so scale the per-rank forward bound by dp_size.
max_num_tokens = get_parallel().dp_size * cutedsl_moe_max_num_tokens()
max_num_tokens = get_parallel().config.dp_size * cutedsl_moe_max_num_tokens()
top_k = layer.top_k if layer.top_k is not None else layer.moe_runner_config.top_k
# inference_mode(False) ensures the wrapper's pre-allocated CUDA-graph
# buffers are normal tensors. This call typically happens inside
python/sglang/srt/layers/moe/token_dispatcher/nixl.py
+2 -2
View File
@@ -135,7 +135,7 @@ class NixlEPBuffer:
offset = ElasticEPStateManager.get_ep_join_rank_offset()
global_rank = rank + offset
max_ep_size = get_parallel().max_ep_size or world_size
max_ep_size = get_parallel().config.max_ep_size or world_size
nixl_max_ranks = max_ep_size
num_rdma_bytes = 0
@@ -233,7 +233,7 @@ class _NixlEPDispatcherImplBase:
)
self._active_world_size = dist.get_world_size(group)
_max_ep = get_parallel().max_ep_size or self._active_world_size
_max_ep = get_parallel().config.max_ep_size or self._active_world_size
self._mask_buffer = (
torch.zeros(_max_ep, dtype=torch.int32, device="cuda")
if self.active_ranks is not None
python/sglang/srt/layers/moe/token_dispatcher/pplx.py
+1 -1
View File
@@ -155,7 +155,7 @@ class PplxAllToAllManager:
# pplx forces ep_size == world_size
# with pp_size == 1 (enforced in _ensure_nvshmem), so the EP group spans
# a single node iff the whole job runs on one node.
is_internode = get_parallel().nnodes > 1
is_internode = get_parallel().config.nnodes > 1
if is_internode:
cls._all_to_all = AllToAll.internode(
python/sglang/srt/layers/moe/utils.py
+1 -1
View File
@@ -607,7 +607,7 @@ def should_skip_post_experts_all_reduce(*, is_tp_path: bool) -> bool:
"""
if should_skip_mlp_all_reduce():
return True
if get_parallel().dwdp_size > 1:
if get_parallel().config.dwdp_size > 1:
return True
if should_use_dp_reduce_scatterv():
return True
python/sglang/srt/layers/utils/cp_utils.py
+3 -3
View File
@@ -63,18 +63,18 @@ class ContextParallelMetadata:
def is_prefill_context_parallel_enabled():
return get_parallel().enable_prefill_context_parallel
return get_parallel().config.enable_prefill_context_parallel
def is_prefill_cp_in_seq_split():
return (
is_prefill_context_parallel_enabled()
and get_parallel().prefill_cp_mode == "in-seq-split"
and get_parallel().config.prefill_cp_mode == "in-seq-split"
)
def is_mla_prefill_cp_enabled() -> bool:
return get_parallel().enable_prefill_context_parallel and uses_mla_backend()
return get_parallel().config.enable_prefill_context_parallel and uses_mla_backend()
def mla_use_prefill_cp(forward_batch, mla_enable_prefill_cp=None):
python/sglang/srt/lora/lora_manager.py
+1 -1
View File
@@ -99,7 +99,7 @@ class LoRAManager:
self.pending_lora_load_events = {}
self.eviction_policy = server_args.lora_eviction_policy
self.enable_dp_attention: bool = get_parallel().enable_dp_attention
self.enable_dp_attention: bool = get_parallel().config.enable_dp_attention
self._experts_shared_outer_override: Optional[bool] = (
server_args.experts_shared_outer_loras
)
python/sglang/srt/managers/data_parallel_controller.py
+34 -27
View File
@@ -50,9 +50,6 @@ from sglang.srt.observability.req_time_stats import DPControllerReqTimeStats
from sglang.srt.observability.startup_time import aggregate_scheduler_startup_times
from sglang.srt.observability.trace import process_tracing_init, trace_set_thread_info
from sglang.srt.runtime_context import (
configured_attn_cp_size,
configured_moe_dp_size,
configured_pp_size,
get_device,
get_disagg,
get_exec,
@@ -151,12 +148,12 @@ class DataParallelController:
self.server_args = server_args
self.port_args = port_args
self.load_balance_method = LoadBalanceMethod.from_str(
get_parallel().load_balance_method
get_parallel().config.load_balance_method
)
self.run_scheduler_process_func = run_scheduler_process_func
# Init inter-process communication
self.context = zmq.Context(1 + get_parallel().dp_size)
self.context = zmq.Context(1 + get_parallel().config.dp_size)
if server_args.node_rank == 0:
self.recv_from_tokenizer = get_zmq_socket(
self.context, zmq.PULL, port_args.scheduler_input_ipc_name, False
@@ -176,8 +173,8 @@ class DataParallelController:
LoadBalanceMethod.TOTAL_TOKENS,
)
self.launch_dp_size: int = get_parallel().dp_size
self.max_dp_size: int = server_args.max_ep_size or get_parallel().dp_size
self.launch_dp_size: int = get_parallel().config.dp_size
self.max_dp_size: int = server_args.max_ep_size or get_parallel().config.dp_size
assert self.max_dp_size >= self.launch_dp_size, (
f"--max-ep-size ({self.max_dp_size}) must be >= "
f"--dp ({self.launch_dp_size})."
@@ -187,7 +184,7 @@ class DataParallelController:
self.max_dp_size - self.launch_dp_size
)
self.dp_budget = DPBudget(get_parallel().dp_size)
self.dp_budget = DPBudget(get_parallel().config.dp_size)
self.load_snapshot_reader = create_load_snapshot_reader(
port_args,
caller="DataParallelController",
@@ -204,14 +201,16 @@ class DataParallelController:
self._active_workers: List[int] = list(range(self.launch_dp_size))
self._active_count_cache: int = self.launch_dp_size
if get_parallel().enable_dp_attention:
if get_parallel().config.enable_dp_attention:
self.launch_dp_attention_schedulers(server_args, port_args)
# When local control broadcast is enabled, send control messages to
# every DP group leader (attn_tp_rank=0) so each leader broadcasts
# within its own attn_tp_group instead of the full tp_group.
# Otherwise fall back to the original behaviour: send to only the
# first leader, which then broadcasts over the full tp_group.
local_ctrl = get_parallel().enable_dp_attention_local_control_broadcast
local_ctrl = (
get_parallel().config.enable_dp_attention_local_control_broadcast
)
self.control_message_step = 1 if local_ctrl else server_args.tp_size
else:
self.launch_dp_schedulers(server_args, port_args)
@@ -375,7 +374,7 @@ class DataParallelController:
threads = []
sockets = []
ready_events = []
for dp_rank in range(get_parallel().dp_size):
for dp_rank in range(get_parallel().config.dp_size):
tmp_port_args = PortArgs.init_new(server_args)
tmp_port_args.tokenizer_ipc_name = port_args.tokenizer_ipc_name
tmp_port_args.detokenizer_ipc_name = port_args.detokenizer_ipc_name
@@ -395,7 +394,9 @@ class DataParallelController:
)
threads.append(thread)
base_gpu_id += (
server_args.tp_size * configured_pp_size() * server_args.gpu_id_step
server_args.tp_size
* get_parallel().config.pp_size
* server_args.gpu_id_step
)
if server_args.node_rank == 0:
@@ -577,7 +578,7 @@ class DataParallelController:
bind_count = (
self.max_dp_size
if server_args.elastic_ep_backend is not None
else get_parallel().dp_size
else get_parallel().config.dp_size
)
for slot in range(bind_count):
worker_port, worker_socket = get_zmq_socket_on_host(
@@ -607,7 +608,7 @@ class DataParallelController:
dp_rank: Optional[int],
worker_ports: Optional[List[int]] = None,
):
if not get_parallel().enable_dp_attention:
if not get_parallel().config.enable_dp_attention:
logger.info(f"Launch DP{dp_rank} starting at GPU #{base_gpu_id}.")
memory_saver_adapter = TorchMemorySaverAdapter.create(
@@ -616,8 +617,8 @@ class DataParallelController:
scheduler_pipe_readers = []
pp_size_per_node = max(configured_pp_size() // server_args.nnodes, 1)
nnodes_per_pp_rank = max(server_args.nnodes // configured_pp_size(), 1)
pp_size_per_node = max(get_parallel().config.pp_size // server_args.nnodes, 1)
nnodes_per_pp_rank = max(server_args.nnodes // get_parallel().config.pp_size, 1)
pp_rank_range = range(
pp_size_per_node * (server_args.node_rank // nnodes_per_pp_rank),
pp_size_per_node * (server_args.node_rank // nnodes_per_pp_rank + 1),
@@ -641,14 +642,14 @@ class DataParallelController:
for tp_rank in tp_rank_range:
rank_port_args = port_args
if get_parallel().enable_dp_attention:
if get_parallel().config.enable_dp_attention:
# dp attention has different sharding logic
_, _, dp_rank, _ = compute_dp_attention_world_info(
get_parallel().enable_dp_attention,
get_parallel().config.enable_dp_attention,
tp_rank,
server_args.tp_size,
get_parallel().dp_size,
configured_attn_cp_size(),
get_parallel().config.dp_size,
get_parallel().config.attn_cp_size,
)
# compute zmq ports for this dp rank
rank_port_args = PortArgs.init_new(
@@ -677,26 +678,32 @@ class DataParallelController:
+ (tp_rank % tp_size_per_node) * server_args.gpu_id_step
)
attn_dp_size = (
get_parallel().dp_size if get_parallel().enable_dp_attention else 1
get_parallel().config.dp_size
if get_parallel().config.enable_dp_attention
else 1
)
# Parallelism hierarchy (outermost to innermost):
# - Attention: Global(TP) -> DP -> ATTN_CP -> ATTN_TP (innermost)
# - MoE: Global(TP) -> MOE_DP -> EP -> MOE_TP (innermost)
attn_tp_size = (
server_args.tp_size // attn_dp_size // configured_attn_cp_size()
server_args.tp_size
// attn_dp_size
// get_parallel().config.attn_cp_size
)
attn_cp_rank = (tp_rank // attn_tp_size) % configured_attn_cp_size()
attn_cp_rank = (
tp_rank // attn_tp_size
) % get_parallel().config.attn_cp_size
moe_dp_rank = tp_rank // (
server_args.tp_size // configured_moe_dp_size()
server_args.tp_size // get_parallel().config.moe_dp_size
)
moe_ep_rank = (
tp_rank
% (server_args.tp_size // configured_moe_dp_size())
% (server_args.tp_size // get_parallel().config.moe_dp_size)
// (
server_args.tp_size
// configured_moe_dp_size()
// get_parallel().ep_size
// get_parallel().config.moe_dp_size
// get_parallel().config.ep_size
)
)
python/sglang/srt/managers/disagg_service.py
+2 -1
View File
@@ -49,7 +49,8 @@ def maybe_create_ascend_config_store(
which the rust registry ports verbatim), leaving this store as the only
``start_disagg_service`` duty left to perform."""
if not (
get_parallel().node_rank == 0 and transfer_backend == TransferBackend.ASCEND
get_parallel().config.node_rank == 0
and transfer_backend == TransferBackend.ASCEND
):
return
try:
python/sglang/srt/managers/load_snapshot.py
+7 -7
View File
@@ -71,7 +71,7 @@ def should_use_zmq() -> bool:
``SGLANG_LOAD_SNAPSHOT_USE_ZMQ`` forces zmq mode for testing.
"""
return (
get_parallel().enable_dp_attention and get_parallel().nnodes > 1
get_parallel().config.enable_dp_attention and get_parallel().config.nnodes > 1
) or envs.SGLANG_LOAD_SNAPSHOT_USE_ZMQ.get()
@@ -116,15 +116,15 @@ def zmq_reader_owner(caller: str) -> bool:
"""
if not should_use_zmq():
return False
if get_parallel().node_rank != 0:
if get_parallel().config.node_rank != 0:
return False
if caller == "DataParallelController":
return (
get_parallel().dp_size > 1
and get_parallel().load_balance_method.lower() in _LOAD_AWARE_METHODS
get_parallel().config.dp_size > 1
and get_parallel().config.load_balance_method.lower() in _LOAD_AWARE_METHODS
)
if get_parallel().dp_size > 1 and (
get_parallel().load_balance_method.lower() in _LOAD_AWARE_METHODS
if get_parallel().config.dp_size > 1 and (
get_parallel().config.load_balance_method.lower() in _LOAD_AWARE_METHODS
):
return False
return caller == _tokenizer_load_snapshot_owner_caller()
@@ -658,7 +658,7 @@ def create_load_snapshot_reader(port_args, caller: str):
``"MultiTokenizerRouter"`` -- determines who binds the zmq PULL
socket when zmq mode is active.
"""
dp_size = get_parallel().dp_size
dp_size = get_parallel().config.dp_size
if zmq_reader_owner(caller):
return ZmqShmLoadSnapshotReader(
_zmq_addr_for(port_args), shm_path_for(port_args.instance_id), dp_size
python/sglang/srt/managers/prefill_delayer.py
+1 -1
View File
@@ -108,7 +108,7 @@ class PrefillDelayer:
f"queue_trigger_enabled={self._queue_trigger_enabled}"
)
self.dp_size = dp_size
self.enable_dp_attention = get_parallel().enable_dp_attention
self.enable_dp_attention = get_parallel().config.enable_dp_attention
dp_size_dim = dp_size if self.enable_dp_attention else 1
# Mirror scheduler_dp_attn_mixin's NCCL all-gather path: when the
python/sglang/srt/managers/scheduler.py
+29 -31
View File
@@ -29,11 +29,6 @@ from typing import TYPE_CHECKING, Any, Deque, Dict, List, Optional, Set, Tuple,
from sglang.srt.runtime_context import (
attention_backends,
configured_attn_cp_size,
configured_dcp_size,
configured_moe_dp_size,
configured_pp_size,
configured_tp_size,
get_device,
get_disagg,
get_exec,
@@ -463,38 +458,38 @@ class Scheduler(
self.max_recv_per_poll = envs.SGLANG_SCHEDULER_MAX_RECV_PER_POLL.get()
self.max_new_tokens_limit = envs.SGLANG_MAX_NEW_TOKENS_LIMIT.get()
self.enable_hisparse = get_memory().enable_hisparse
self.enable_dp_attention = get_parallel().enable_dp_attention
self.enable_dp_attention = get_parallel().config.enable_dp_attention
self.enable_unified_memory = get_memory().enable_unified_memory
# Distributed rank info
attn_tp_rank, attn_tp_size, attn_dp_rank, attn_dp_size = (
compute_dp_attention_world_info(
get_parallel().enable_dp_attention,
get_parallel().config.enable_dp_attention,
tp_rank,
configured_tp_size(),
get_parallel().dp_size,
configured_attn_cp_size(),
get_parallel().config.tp_size,
get_parallel().config.dp_size,
get_parallel().config.attn_cp_size,
)
)
self.ps = ParallelState(
tp_rank=tp_rank,
tp_size=configured_tp_size(),
tp_size=get_parallel().config.tp_size,
pp_rank=pp_rank,
pp_size=configured_pp_size(),
pp_size=get_parallel().config.pp_size,
dp_rank=dp_rank,
dp_size=get_parallel().dp_size,
dp_size=get_parallel().config.dp_size,
attn_tp_rank=attn_tp_rank,
attn_tp_size=attn_tp_size,
attn_cp_rank=attn_cp_rank,
attn_cp_size=configured_attn_cp_size(),
attn_dcp_rank=tp_rank % configured_dcp_size(),
attn_dcp_size=configured_dcp_size(),
attn_cp_size=get_parallel().config.attn_cp_size,
attn_dcp_rank=tp_rank % get_parallel().config.dcp_size,
attn_dcp_size=get_parallel().config.dcp_size,
attn_dp_rank=attn_dp_rank,
attn_dp_size=attn_dp_size,
moe_ep_rank=moe_ep_rank,
moe_ep_size=get_parallel().ep_size,
moe_ep_size=get_parallel().config.ep_size,
moe_dp_rank=moe_dp_rank,
moe_dp_size=configured_moe_dp_size(),
moe_dp_size=get_parallel().config.moe_dp_size,
gpu_id=gpu_id,
)
@@ -1069,7 +1064,7 @@ class Scheduler(
self.min_free_slots_delayer = MinFreeSlotsDelayer(
min_free_slots=min_free_slots
)
if not get_parallel().pp_max_micro_batch_size:
if not get_parallel().config.pp_max_micro_batch_size:
get_context().override(
"scheduler.pp_max_micro_batch_size_default",
pp_max_micro_batch_size=max(
@@ -1414,7 +1409,7 @@ class Scheduler(
gloo_group=self.attn_tp_cpu_group,
tp_rank=self.ps.tp_rank,
tp_size=self.ps.tp_size,
dp_size=get_parallel().dp_size,
dp_size=get_parallel().config.dp_size,
gpu_id=self.ps.gpu_id,
bootstrap_port=get_disagg().disaggregation_bootstrap_port,
max_total_num_tokens=self.max_total_num_tokens,
@@ -3233,7 +3228,7 @@ class Scheduler(
return NextBatchPlan(batch_to_run=ret, running_batch=running_batch)
def get_num_allocatable_reqs(self, running_bs):
res = get_parallel().pp_max_micro_batch_size - running_bs
res = get_parallel().config.pp_max_micro_batch_size - running_bs
res = min(res, self.req_to_token_pool.available_size())
return res
@@ -4890,7 +4885,7 @@ class Scheduler(
old_ep_size = ElasticEPStateManager.get_effective_ep_size()
new_ep_size = recv_req.new_ep_size
max_ep_size = get_parallel().max_ep_size or old_ep_size
max_ep_size = get_parallel().config.max_ep_size or old_ep_size
logger.debug(
"[Elastic EP][scale] request received: new_ep_size=%d "
@@ -5095,7 +5090,7 @@ def dispatch_event_loop(scheduler: Scheduler):
if disaggregation_mode == DisaggregationMode.NULL:
if scheduler.enable_pdmux:
scheduler.event_loop_pdmux()
elif configured_pp_size() > 1:
elif get_parallel().config.pp_size > 1:
scheduler.event_loop_pp()
elif scheduler.enable_overlap_mlx:
scheduler.event_loop_overlap_mlx()
@@ -5104,14 +5099,14 @@ def dispatch_event_loop(scheduler: Scheduler):
else:
scheduler.event_loop_normal()
elif disaggregation_mode == DisaggregationMode.PREFILL:
if configured_pp_size() > 1:
if get_parallel().config.pp_size > 1:
scheduler.event_loop_pp_disagg_prefill()
elif scheduler.enable_overlap:
scheduler.event_loop_overlap_disagg_prefill()
else:
scheduler.event_loop_normal_disagg_prefill()
elif disaggregation_mode == DisaggregationMode.DECODE:
if configured_pp_size() > 1:
if get_parallel().config.pp_size > 1:
scheduler.event_loop_pp_disagg_decode()
elif scheduler.enable_overlap:
scheduler.event_loop_overlap_disagg_decode()
@@ -5152,15 +5147,15 @@ def configure_scheduler_process(
prefix = ""
if shown_dp is not None:
prefix += f" DP{shown_dp}"
if configured_pp_size() > 1:
if get_parallel().config.pp_size > 1:
prefix += f" PP{pp_rank}"
if configured_attn_cp_size() > 1:
if get_parallel().config.attn_cp_size > 1:
prefix += f" ATTN_CP{attn_cp_rank}"
if configured_moe_dp_size() > 1:
if get_parallel().config.moe_dp_size > 1:
prefix += f" MOE_DP{moe_dp_rank}"
if configured_tp_size() > 1:
if get_parallel().config.tp_size > 1:
prefix += f" TP{shown_tp}"
if get_parallel().ep_size > 1:
if get_parallel().config.ep_size > 1:
prefix += f" EP{shown_moe_ep}"
# Config the process
@@ -5174,7 +5169,10 @@ def configure_scheduler_process(
# Set cpu affinity to this gpu process
if envs.SGLANG_SET_CPU_AFFINITY.get():
set_gpu_proc_affinity(
configured_pp_size(), configured_tp_size(), get_parallel().nnodes, gpu_id
get_parallel().config.pp_size,
get_parallel().config.tp_size,
get_parallel().config.nnodes,
gpu_id,
)
if not envs.SGLANG_NUMA_BIND_V2.get():
numa_node = get_numa_node_if_available(server_args, gpu_id)
python/sglang/srt/managers/scheduler_components/dp_attn.py
+2 -2
View File
@@ -411,7 +411,7 @@ class SchedulerDPAttnAdapter:
return prepare_mlp_sync_batch_raw(
local_batch,
model_runner=self.model_runner,
dp_size=get_parallel().dp_size,
dp_size=get_parallel().config.dp_size,
attn_tp_size=self.ps.attn_tp_size,
attn_cp_size=self.ps.attn_cp_size,
tp_group=self.tp_group,
@@ -420,7 +420,7 @@ class SchedulerDPAttnAdapter:
require_mlp_tp_gather=require_mlp_tp_gather(self.server_args),
disable_overlap_schedule=get_schedule().disable_overlap_schedule,
offload_tags=self.offload_tags,
dwdp=get_parallel().dwdp_size > 1,
dwdp=get_parallel().config.dwdp_size > 1,
)
def maybe_prepare_mlp_sync_batch(
python/sglang/srt/managers/scheduler_components/metrics_reporter.py
+2 -2
View File
@@ -23,10 +23,10 @@ from sglang.srt.observability.metrics_collector import (
compute_routing_key_stats,
)
from sglang.srt.runtime_context import (
configured_pp_size,
get_context,
get_disagg,
get_observability,
get_parallel,
get_spec,
)
from sglang.srt.utils.device_timer import DeviceTimer
@@ -1117,7 +1117,7 @@ class SchedulerMetricsReporter:
active_lora_ids = set()
# For PP mode, check all running micro batches
if configured_pp_size() > 1:
if get_parallel().config.pp_size > 1:
for batch in self.scheduler.running_mbs:
if batch and hasattr(batch, "reqs"):
for req in batch.reqs:
python/sglang/srt/managers/scheduler_components/recv_skipper.py
+1 -1
View File
@@ -34,7 +34,7 @@ class SchedulerRecvSkipper:
return ForwardMode.DECODE
def __init__(self):
self._use_synced_mode = get_parallel().enable_dp_attention
self._use_synced_mode = get_parallel().config.enable_dp_attention
self._counter = 0
self._threshold = get_schedule().scheduler_recv_interval
# All can be tuned if needed
python/sglang/srt/managers/scheduler_components/request_receiver.py
+3 -3
View File
@@ -151,7 +151,7 @@ class SchedulerRequestReceiver:
return recv_reqs
def _broadcast_reqs_across_ranks(self, recv_reqs: Optional[List]) -> List:
if get_parallel().enable_dp_attention:
if get_parallel().config.enable_dp_attention:
if self.ps.attn_tp_rank == 0 and self.ps.attn_cp_rank == 0:
work_reqs, control_reqs = self._split_work_and_control_reqs(recv_reqs)
else:
@@ -180,7 +180,7 @@ class SchedulerRequestReceiver:
# instead of the full tp_group. This avoids an expensive
# all-ranks gloo sync.
_local_ctrl = (
get_parallel().enable_dp_attention_local_control_broadcast
get_parallel().config.enable_dp_attention_local_control_broadcast
or is_ep_scale_joiner()
)
if _local_ctrl:
@@ -258,7 +258,7 @@ class SchedulerRequestReceiver:
# peer ranks may still be unpickling ShmPointerMMData
# (-> shm_open). Synchronize the same CPU groups that carried
# SHM-backed work requests before materialize() unlinks them.
if get_parallel().enable_dp_attention:
if get_parallel().config.enable_dp_attention:
if self.ps.attn_tp_size > 1:
barrier(group=self.attn_tp_cpu_group)
if self.ps.attn_cp_size > 1:
python/sglang/srt/managers/scheduler_pp_mixin.py
+10 -8
View File
@@ -128,7 +128,7 @@ class SchedulerPPMixin:
next_pp_outputs = None
next_batch_result = None
d2h_event = None
if get_parallel().pp_async_batch_depth > 0:
if get_parallel().config.pp_async_batch_depth > 0:
next_pp_outputs, next_batch_result, d2h_event = (
self._pp_commit_send_output_work_and_preprocess_output_tensors(
next_first_rank_mb_id,
@@ -144,7 +144,7 @@ class SchedulerPPMixin:
self.mb_metadata,
self.last_rank_comm_queue,
)
if get_parallel().pp_async_batch_depth == 0:
if get_parallel().config.pp_async_batch_depth == 0:
next_pp_outputs, next_batch_result, d2h_event = (
self._pp_commit_send_output_work_and_preprocess_output_tensors(
next_first_rank_mb_id,
@@ -274,7 +274,7 @@ class SchedulerPPMixin:
server_is_idle = False
pp_proxy_tensors = self._pp_recv_proxy_tensors()
if get_parallel().pp_async_batch_depth > 0:
if get_parallel().config.pp_async_batch_depth > 0:
next_pp_outputs, next_batch_result, d2h_event = (
self._pp_commit_send_output_work_and_preprocess_output_tensors(
next_first_rank_mb_id,
@@ -292,7 +292,7 @@ class SchedulerPPMixin:
self.mb_metadata,
self.last_rank_comm_queue,
)
if get_parallel().pp_async_batch_depth == 0:
if get_parallel().config.pp_async_batch_depth == 0:
next_pp_outputs, next_batch_result, d2h_event = (
self._pp_commit_send_output_work_and_preprocess_output_tensors(
next_first_rank_mb_id,
@@ -435,7 +435,7 @@ class SchedulerPPMixin:
pp_proxy_tensors = self._pp_recv_proxy_tensors()
# early send output if possible
if get_parallel().pp_async_batch_depth > 0:
if get_parallel().config.pp_async_batch_depth > 0:
next_pp_outputs, next_batch_result, d2h_event = (
self._pp_commit_send_output_work_and_preprocess_output_tensors(
next_first_rank_mb_id,
@@ -453,7 +453,7 @@ class SchedulerPPMixin:
self.last_rank_comm_queue,
)
if get_parallel().pp_async_batch_depth == 0:
if get_parallel().config.pp_async_batch_depth == 0:
next_pp_outputs, next_batch_result, d2h_event = (
self._pp_commit_send_output_work_and_preprocess_output_tensors(
next_first_rank_mb_id,
@@ -564,10 +564,12 @@ class SchedulerPPMixin:
self.on_idle()
def init_pp_loop_state(self: Scheduler):
self.pp_loop_size: int = self.ps.pp_size + get_parallel().pp_async_batch_depth
self.pp_loop_size: int = (
self.ps.pp_size + get_parallel().config.pp_async_batch_depth
)
# In CP mode, attention weights are duplicated, eliminating the need for the attention TP all-gather operation.
self.require_attn_tp_allgather = (
not get_parallel().enable_dsa_prefill_context_parallel
not get_parallel().config.enable_dsa_prefill_context_parallel
)
self.mbs = [None] * self.pp_loop_size
self.last_mbs = [None] * self.pp_loop_size
python/sglang/srt/managers/tokenizer_control_mixin.py
+22 -14
View File
@@ -165,7 +165,7 @@ class TokenizerControlMixin:
mode = spec[2] if len(spec) > 2 else "queueing"
comm = FanOutCommunicator(
self._dispatch_to_scheduler,
get_parallel().dp_size,
get_parallel().config.dp_size,
mode,
)
setattr(self, f"{name}_communicator", comm)
@@ -174,8 +174,8 @@ class TokenizerControlMixin:
def update_control_communicator_fan_out(self: TokenizerManager, worker_count: int):
primary_group_control = (
get_parallel().enable_dp_attention
and not get_parallel().enable_dp_attention_local_control_broadcast
get_parallel().config.enable_dp_attention
and not get_parallel().config.enable_dp_attention_local_control_broadcast
)
if primary_group_control:
control_fan_out = (
@@ -428,7 +428,8 @@ class TokenizerControlMixin:
) -> Tuple[bool, str]:
self.auto_create_handle_loop()
assert (
get_parallel().dp_size == 1 or get_parallel().enable_dp_attention
get_parallel().config.dp_size == 1
or get_parallel().config.enable_dp_attention
), "dp_size must be 1 or dp attention must be enabled for update weights from distributed"
results = await self.init_weights_update_group_communicator(obj)
@@ -441,7 +442,8 @@ class TokenizerControlMixin:
) -> Tuple[bool, str]:
self.auto_create_handle_loop()
assert (
get_parallel().dp_size == 1 or get_parallel().enable_dp_attention
get_parallel().config.dp_size == 1
or get_parallel().config.enable_dp_attention
), "dp_size must be 1 or dp attention must be enabled for destroy parameter update group"
results = await self.destroy_weights_update_group_communicator(obj)
@@ -454,7 +456,8 @@ class TokenizerControlMixin:
) -> Tuple[bool, str]:
self.auto_create_handle_loop()
assert (
get_parallel().dp_size == 1 or get_parallel().enable_dp_attention
get_parallel().config.dp_size == 1
or get_parallel().config.enable_dp_attention
), "dp_size must be 1 or dp attention must be enabled for update weights from distributed"
if obj.abort_all_requests:
@@ -487,7 +490,7 @@ class TokenizerControlMixin:
self.auto_create_handle_loop()
# TODO: support DP
assert (
get_parallel().dp_size == 1
get_parallel().config.dp_size == 1
), "dp_size must be 1 for init_weights_send_group_for_remote_instance"
result = (
await self.init_weights_send_group_for_remote_instance_communicator(obj)
@@ -502,7 +505,7 @@ class TokenizerControlMixin:
self.auto_create_handle_loop()
# TODO: support DP
assert (
get_parallel().dp_size == 1
get_parallel().config.dp_size == 1
), "dp_size must be 1 for send_weights_to_remote_instance"
result = (await self.send_weights_to_remote_instance_communicator(obj))[0]
return result.success, result.message
@@ -514,7 +517,8 @@ class TokenizerControlMixin:
) -> Tuple[bool, str]:
self.auto_create_handle_loop()
assert (
get_parallel().dp_size == 1 or get_parallel().enable_dp_attention
get_parallel().config.dp_size == 1
or get_parallel().config.enable_dp_attention
), "dp_size must be 1 or dp attention must be enabled for update weights from tensor"
if obj.abort_all_requests:
@@ -552,7 +556,8 @@ class TokenizerControlMixin:
try:
# For now, we only support single data parallel instance
assert (
get_parallel().dp_size == 1 or get_parallel().enable_dp_attention
get_parallel().config.dp_size == 1
or get_parallel().config.enable_dp_attention
), "dp_size must be 1 or dp attention must be enabled for update weights from IPC"
logger.info("Starting IPC weight update")
@@ -615,7 +620,8 @@ class TokenizerControlMixin:
)
assert (
get_parallel().dp_size == 1 or get_parallel().enable_dp_attention
get_parallel().config.dp_size == 1
or get_parallel().config.enable_dp_attention
), "dp_size must be 1 or dp attention must be enabled for dynamic lora loading"
logger.info(
"Start load Lora adapter. Lora name=%s, path=%s",
@@ -693,7 +699,8 @@ class TokenizerControlMixin:
)
assert (
get_parallel().dp_size == 1 or get_parallel().enable_dp_attention
get_parallel().config.dp_size == 1
or get_parallel().config.enable_dp_attention
), "dp_size must be 1 or dp attention must be enabled for dynamic lora loading"
logger.info(
"Start load Lora adapter from tensors. Lora name=%s",
@@ -773,7 +780,8 @@ class TokenizerControlMixin:
), "lora_name must be provided to unload LoRA adapter"
assert (
get_parallel().dp_size == 1 or get_parallel().enable_dp_attention
get_parallel().config.dp_size == 1
or get_parallel().config.enable_dp_attention
), "dp_size must be 1 or dp attention must be enabled for dynamic lora loading"
logger.info(
"Start unload Lora adapter. Lora name=%s",
@@ -793,7 +801,7 @@ class TokenizerControlMixin:
self.auto_create_handle_loop()
results = await self.get_weights_by_name_communicator(obj)
all_parameters = [r.parameter for r in results]
if get_parallel().dp_size == 1:
if get_parallel().config.dp_size == 1:
return all_parameters[0]
else:
return all_parameters
python/sglang/srt/managers/tokenizer_manager.py
+2 -2
View File
@@ -411,7 +411,7 @@ class TokenizerManager(TokenizerControlMixin, TokenizerManagerScoreMixin):
self.server_args = server_args
ensure_published(server_args, role="tokenizer")
self.startup_time: Optional[Dict[str, Any]] = None
self.elastic_worker_count = get_parallel().dp_size
self.elastic_worker_count = get_parallel().config.dp_size
self.elastic_pending_ep_size = None
self.elastic_scale_phase = "idle"
self.elastic_last_error = None
@@ -1548,7 +1548,7 @@ class TokenizerManager(TokenizerControlMixin, TokenizerManagerScoreMixin):
return batch_size > 0 and (
get_serving().enable_tokenizer_batch_encode
or (
(not get_parallel().enable_dp_attention)
(not get_parallel().config.enable_dp_attention)
and (not self._batch_has_text(batch_size, requests))
)
)
python/sglang/srt/mem_cache/kv_cache_builder.py
+3 -1
View File
@@ -275,7 +275,9 @@ def build_kv_cache(
),
is_eagle=spec_algorithm.is_eagle(),
tp_cache_group=(
attn_tp_cpu_group if get_parallel().enable_dp_attention else tp_cpu_group
attn_tp_cpu_group
if get_parallel().config.enable_dp_attention
else tp_cpu_group
),
attn_cp_cache_group=attn_cp_cpu_group,
attn_tp_cache_group=attn_tp_cpu_group,
python/sglang/srt/mem_cache/kv_cache_configurator.py
+1 -2
View File
@@ -65,7 +65,6 @@ from sglang.srt.mem_cache.memory_pool import (
from sglang.srt.mem_cache.swa_memory_pool import SWAKVPool
from sglang.srt.platforms import current_platform
from sglang.srt.runtime_context import (
configured_pp_size,
get_context,
get_disagg,
get_exec,
@@ -1922,7 +1921,7 @@ class KVCacheConfigurator:
token_capacity = min(token_capacity, user_limit)
# Sync across PP ranks (each may have different layer counts)
if configured_pp_size() > 1:
if get_parallel().config.pp_size > 1:
tensor = torch.tensor(token_capacity, dtype=torch.int64)
torch.distributed.all_reduce(
tensor,
python/sglang/srt/mem_cache/pool_host/base.py
+1 -1
View File
@@ -45,7 +45,7 @@ def ranks_per_host() -> int:
return 1
if world_group.world_size == 1:
return 1
return max(world_group.world_size // get_parallel().nnodes, 1)
return max(world_group.world_size // get_parallel().config.nnodes, 1)
def host_memory_budget_bytes() -> int:
python/sglang/srt/model_executor/cpu_graph_runner.py
+2 -3
View File
@@ -40,7 +40,6 @@ from sglang.srt.model_executor.forward_batch_info import (
from sglang.srt.model_executor.forward_context import ForwardContext, forward_context
from sglang.srt.model_executor.runner_utils.capture_mode import model_capture_mode
from sglang.srt.runtime_context import (
configured_pp_size,
get_exec,
get_flags,
get_lora,
@@ -611,8 +610,8 @@ class CPUGraphRunner:
model_runner.server_args.enable_profile_cuda_graph
)
self.tp_size = model_runner.server_args.tp_size
self.dp_size = get_parallel().dp_size
self.pp_size = configured_pp_size()
self.dp_size = get_parallel().config.dp_size
self.pp_size = get_parallel().config.pp_size
self.capture_forward_mode = ForwardMode.DECODE
self.capture_hidden_mode = self.return_hidden_states_mode
python/sglang/srt/model_executor/model_runner.py
+13 -11
View File
@@ -489,17 +489,19 @@ class ModelRunner:
if not (get_exec().moe.elastic_ep_backend is not None and is_ep_scale_joiner()):
return
join_effective_ep_size = get_parallel().ep_join_rank_offset + self.ps.tp_size
join_effective_ep_size = (
get_parallel().config.ep_join_rank_offset + self.ps.tp_size
)
dist.barrier(group=self.tp_group.cpu_group)
if self.ps.tp_rank == 0:
register_scale_cohort(
get_parallel().ep_join_rank_offset,
get_parallel().config.ep_join_rank_offset,
join_effective_ep_size,
)
join_scale_process_group()
get_context().override("elastic_ep.scale_join", ep_size=join_effective_ep_size)
global_ep_rank = self.ps.tp_rank + get_parallel().ep_join_rank_offset
global_ep_rank = self.ps.tp_rank + get_parallel().config.ep_join_rank_offset
broadcast_global_expert_location_metadata(
model_config=self.model_config,
moe_ep_rank=global_ep_rank,
@@ -699,7 +701,7 @@ class ModelRunner:
if self.is_draft_worker:
return
expert_rank = self.ps.moe_ep_rank + (
get_parallel().ep_join_rank_offset if is_ep_scale_joiner() else 0
get_parallel().config.ep_join_rank_offset if is_ep_scale_joiner() else 0
)
set_global_expert_location_metadata(
compute_initial_expert_location_metadata(
@@ -897,7 +899,7 @@ class ModelRunner:
device=self.device,
tp_group=(
self.attention_tp_group.cpu_group
if get_parallel().enable_dp_attention
if get_parallel().config.enable_dp_attention
else self.tp_group.cpu_group
),
host_to_device_ratio=hisparse_cfg.host_to_device_ratio,
@@ -933,7 +935,7 @@ class ModelRunner:
def post_capture_elastic_ep_recover(self):
join_process_groups()
global_ep_rank = self.ps.tp_rank + get_parallel().ep_join_rank_offset
global_ep_rank = self.ps.tp_rank + get_parallel().config.ep_join_rank_offset
broadcast_global_expert_location_metadata(
model_config=self.model_config,
moe_ep_rank=global_ep_rank,
@@ -973,7 +975,7 @@ class ModelRunner:
self.decode_attn_backend = backends.decode_attn_backend
self.decode_attn_backend_group = backends.decode_attn_backend_group
if get_parallel().dcp_enabled and get_parallel().dcp_replicate_q_proj:
if get_parallel().dcp_enabled and get_parallel().config.dcp_replicate_q_proj:
self._prepare_replicated_q_proj()
def _prepare_replicated_q_proj(self) -> None:
@@ -1257,7 +1259,7 @@ class ModelRunner:
def maybe_init_dwdp(self):
if self.is_draft_worker:
return
if get_parallel().dwdp_size <= 1:
if get_parallel().config.dwdp_size <= 1:
return
from sglang.srt.layers.moe.dwdp import DwdpManager
@@ -1431,7 +1433,7 @@ class ModelRunner:
# rather than spawning additional processes, so dp_size must not be
# multiplied into the process count here (unlike regular DP, where
# dp_size * tp_size * pp_size is the true worker count).
dp_size = 1 if get_parallel().enable_dp_attention else self.ps.dp_size
dp_size = 1 if get_parallel().config.enable_dp_attention else self.ps.dp_size
self.local_omp_cpuid = numa_utils.init_threads_binding(
numa_index=self.gpu_id,
world_size=dp_size * self.ps.tp_size * self.ps.pp_size,
@@ -1916,7 +1918,7 @@ class ModelRunner:
if added <= 0:
return
initial_ep_size = get_parallel().elastic_ep_initial_size
initial_ep_size = get_parallel().config.elastic_ep_initial_size
assert initial_ep_size is not None
get_context().override("elastic_ep.scale", ep_size=effective_size)
@@ -1935,7 +1937,7 @@ class ModelRunner:
set_global_expert_location_metadata(new_metadata, allow_overwrite=True)
def _elastic_global_rank(self) -> int:
return self.ps.tp_rank + get_parallel().ep_join_rank_offset
return self.ps.tp_rank + get_parallel().config.ep_join_rank_offset
def _rearm_eplb_after_elastic_scale(self) -> None:
if self.eplb_manager is None:
python/sglang/srt/model_executor/model_runner_components/remote_instance_weight_transporter.py
+4 -2
View File
@@ -78,11 +78,13 @@ class RemoteInstanceWeightTransporter:
"""
import requests as http_requests
if get_parallel().dist_init_addr:
if get_parallel().config.dist_init_addr:
# Multi-node: bootstrap server is on the head node (node_rank==0).
# Derive host from dist_init_addr (shared across all nodes).
bootstrap_host = (
NetworkAddress.parse(get_parallel().dist_init_addr).resolved().host
NetworkAddress.parse(get_parallel().config.dist_init_addr)
.resolved()
.host
)
else:
bootstrap_host = "127.0.0.1"
python/sglang/srt/model_executor/model_runner_components/spec_aux_hidden_state.py
+2 -2
View File
@@ -7,8 +7,8 @@ import msgspec
from sglang.srt.configs.model_config import ModelConfig
from sglang.srt.runtime_context import (
configured_tp_size,
get_model,
get_parallel,
get_spec,
)
@@ -242,7 +242,7 @@ def _resolve_dflash_draft_cell_size(
draft_model_config=draft_model_config,
draft_num_layers=draft_num_layers,
draft_kv_cache_dtype=draft_kv_cache_dtype,
tp_size=configured_tp_size(),
tp_size=get_parallel().config.tp_size,
)
except Exception as e: # noqa: BLE001
logger.warning(
python/sglang/srt/model_executor/model_runner_components/startup_weight_load.py
+6 -10
View File
@@ -21,10 +21,6 @@ from sglang.srt.model_loader.weight_utils import (
)
from sglang.srt.platforms import current_platform
from sglang.srt.runtime_context import (
configured_attn_cp_size,
configured_dcp_size,
configured_pp_size,
configured_tp_size,
get_device,
get_exec,
get_lora,
@@ -120,12 +116,12 @@ class StartupWeightLoadOptions:
prefill_cuda_graph_backend=cuda_graph_config.prefill.backend,
is_draft_worker=is_draft_worker,
speculative_algorithm=get_spec().speculative_algorithm,
tp_size=configured_tp_size(),
attn_cp_size=configured_attn_cp_size(),
dcp_size=configured_dcp_size(),
pp_size=configured_pp_size(),
dp_size=get_parallel().dp_size,
ep_size=get_parallel().ep_size,
tp_size=get_parallel().config.tp_size,
attn_cp_size=get_parallel().config.attn_cp_size,
dcp_size=get_parallel().config.dcp_size,
pp_size=get_parallel().config.pp_size,
dp_size=get_parallel().config.dp_size,
ep_size=get_parallel().config.ep_size,
cpu_offload_gb=get_exec().offload.cpu_offload_gb,
offload_group_size=get_exec().offload.offload_group_size,
enable_memory_saver=get_exec().features.enable_memory_saver,
python/sglang/srt/model_executor/runner/base_runner.py
+8 -9
View File
@@ -47,7 +47,6 @@ from sglang.srt.model_executor.runner.flashinfer_autotune import (
should_run_flashinfer_autotune,
)
from sglang.srt.runtime_context import (
configured_pp_size,
get_disagg,
get_exec,
get_flags,
@@ -219,8 +218,8 @@ class BaseRunner(ABC):
self.device_module = torch.get_device_module(self.device)
self.tp_size = model_runner.server_args.tp_size
# elastic-EP scale-up rewrites dp_size on the published config
self.dp_size = get_parallel().dp_size
self.pp_size = configured_pp_size()
self.dp_size = get_parallel().config.dp_size
self.pp_size = get_parallel().config.pp_size
self.enable_pdmux = model_runner.server_args.enable_pdmux
self.return_hidden_states_mode = (
CaptureHiddenMode.NULL
@@ -290,7 +289,7 @@ class BaseRunner(ABC):
"""
if (
not get_parallel().dcp_enabled
or get_parallel().dcp_comm_backend != "fi_a2a"
or get_parallel().config.dcp_comm_backend != "fi_a2a"
):
return
@@ -349,8 +348,8 @@ class BaseRunner(ABC):
hidden_size=mr.model_config.hidden_size,
vocab_size=mr.model_config.vocab_size,
dtype=mr.model_config.dtype,
dp_size=get_parallel().dp_size,
pp_size=configured_pp_size(),
dp_size=get_parallel().config.dp_size,
pp_size=get_parallel().config.pp_size,
is_encoder_decoder=mr.model_config.is_encoder_decoder,
require_mlp_tp_gather=require_mlp_tp_gather(mr.server_args),
seq_len_fill_value=mr.attn_backend.get_cuda_graph_seq_len_fill_value(),
@@ -522,7 +521,7 @@ class BaseRunner(ABC):
extend_prefix_lens = None
extend_start_loc = None
if configured_pp_size() > 1:
if get_parallel().config.pp_size > 1:
# PP0 already cp-split hidden_states before send.
pp_hidden_tokens = num_tokens
if (
@@ -542,7 +541,7 @@ class BaseRunner(ABC):
assert require_mlp_tp_gather_ or require_attn_tp_gather_
if require_mlp_tp_gather_:
global_num_tokens_cpu = [num_tokens] * get_parallel().dp_size
global_num_tokens_cpu = [num_tokens] * get_parallel().config.dp_size
elif require_attn_tp_gather_:
global_num_tokens_cpu = [num_tokens]
else:
@@ -646,7 +645,7 @@ class BaseRunner(ABC):
kwargs = {}
if (
configured_pp_size() > 1
get_parallel().config.pp_size > 1
and "pp_proxy_tensors" in inspect.signature(mr.model.forward).parameters
):
kwargs["pp_proxy_tensors"] = PPProxyTensors(
python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py
+1 -1
View File
@@ -238,7 +238,7 @@ class DecodeCudaGraphRunner(BaseCudaGraphRunner):
self.require_mlp_tp_gather or self.require_attn_tp_gather
)
self.require_mlp_sync = (
get_parallel().enable_dp_attention or self.require_gathered_buffer
get_parallel().config.enable_dp_attention or self.require_gathered_buffer
)
self.enable_two_batch_overlap = (
model_runner.server_args.enable_two_batch_overlap
python/sglang/srt/model_executor/runner/eager_runner.py
+1 -1
View File
@@ -141,7 +141,7 @@ class EagerRunner(BaseRunner):
encoder_lens_dtype=(
torch.int64 if torch.device(mr.device).type == "cpu" else torch.int32
),
dp_size=get_parallel().dp_size,
dp_size=get_parallel().config.dp_size,
)
# Eager has no capture step, so warm up here (run-once via mr._kernel_warmed_up).
self.warmup()
python/sglang/srt/model_executor/runner/prefill_cuda_graph_runner.py
+1 -1
View File
@@ -348,7 +348,7 @@ class PrefillCudaGraphRunner(BaseCudaGraphRunner):
self.moe_fusions = self.model_runner.moe_fusions
self.dsa_indexers = getattr(self.model_runner, "dsa_indexers", None)
self.dp_size = get_parallel().dp_size
self.dp_size = get_parallel().config.dp_size
self.require_mlp_tp_gather = require_mlp_tp_gather(model_runner.server_args)
self.require_attn_tp_gather = require_attn_tp_gather(model_runner.server_args)
python/sglang/srt/model_loader/loader.py
+3 -4
View File
@@ -49,7 +49,6 @@ from sglang.srt.model_loader.remote_instance_weight_loader_utils import (
register_memory_region,
)
from sglang.srt.runtime_context import (
configured_moe_dp_size,
get_exec,
get_model,
get_parallel,
@@ -1891,9 +1890,9 @@ class PreshardedModelLoader(DefaultModelLoader):
"dp": _safe(lambda: parallel.moe_dp_size),
"ep": _safe(lambda: parallel.moe_ep_size),
"pp": _safe(lambda: parallel.pp_size),
"moe_dense_tp_size": parallel.moe_dense_tp_size,
"moe_dp_size": configured_moe_dp_size(),
"enable_dp_lm_head": parallel.enable_dp_lm_head,
"moe_dense_tp_size": parallel.config.moe_dense_tp_size,
"moe_dp_size": get_parallel().config.moe_dp_size,
"enable_dp_lm_head": parallel.config.enable_dp_lm_head,
"enable_fp32_lm_head": get_exec().features.enable_fp32_lm_head,
"quantization": model_config.quantization,
"model_dtype": str(model_config.dtype),
python/sglang/srt/models/apertus.py
+1 -1
View File
@@ -442,7 +442,7 @@ class ApertusForCausalLM(nn.Module):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
self.pooler = Pooler(pooling_type=PoolingType.LAST, normalize=True)
python/sglang/srt/models/arcee.py
+1 -1
View File
@@ -405,7 +405,7 @@ class ArceeForCausalLM(nn.Module):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
self.pooler = Pooler(pooling_type=PoolingType.LAST, normalize=True)
python/sglang/srt/models/bailing_moe.py
+1 -1
View File
@@ -817,7 +817,7 @@ class BailingMoEForCausalLM(nn.Module):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/bailing_moe_linear.py
+1 -1
View File
@@ -1084,7 +1084,7 @@ class BailingMoELinearForCausalLM(nn.Module):
config.hidden_size,
params_dtype=torch.float32,
quant_config=quant_config,
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
)
self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/bailing_moe_nextn.py
+1 -1
View File
@@ -208,7 +208,7 @@ class BailingMoeForCausalLMNextN(nn.Module):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("model.shared_head.head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
if hasattr(self.config, "model_type") and config.model_type == "bailing_hybrid":
python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py
+2 -2
View File
@@ -299,7 +299,7 @@ class DeepseekMLAForwardMixin:
# --dcp-replicate-q-proj: project full-head Q locally from pre-gathered
# weights and skip the per-layer Q all-gather (bf16 decode absorb only).
q_replicate_active = (
get_parallel().dcp_replicate_q_proj
get_parallel().config.dcp_replicate_q_proj
and is_dcp_mla_decode_phase(forward_batch)
and not self.use_deep_gemm_bmm
and self.w_kc_qrep is not None
@@ -778,7 +778,7 @@ class DeepseekMLAForwardMixin:
attn_output, self.num_local_heads
)
else:
dcp_comm_backend = get_parallel().dcp_comm_backend
dcp_comm_backend = get_parallel().config.dcp_comm_backend
is_lse_base_on_e = is_mla_dcp_lse_base_on_e(
self.current_attention_backend
)
python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla_rocm.py
+2 -2
View File
@@ -331,7 +331,7 @@ class DeepseekMLARocmForwardMixin:
from sglang.srt.model_executor.runner import get_is_capture_mode
q_replicate_active = (
get_parallel().dcp_replicate_q_proj
get_parallel().config.dcp_replicate_q_proj
and is_dcp_mla_decode_phase(forward_batch)
and not self.use_deep_gemm_bmm
and self.w_kc_qrep is not None
@@ -778,7 +778,7 @@ class DeepseekMLARocmForwardMixin:
attn_output, self.num_local_heads
)
else:
dcp_comm_backend = get_parallel().dcp_comm_backend
dcp_comm_backend = get_parallel().config.dcp_comm_backend
is_lse_base_on_e = is_mla_dcp_lse_base_on_e(
self.current_attention_backend
)
python/sglang/srt/models/deepseek_nextn.py
+1 -1
View File
@@ -366,7 +366,7 @@ class DeepseekV3ForCausalLMNextN(DeepseekV3ForCausalLM):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("model.shared_head.head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/deepseek_v2.py
+1 -1
View File
@@ -3003,7 +3003,7 @@ class DeepseekV2ForCausalLM(nn.Module, DeepseekV2WeightLoaderMixin):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
else:
# ranks other than the last rank will have a placeholder layer
python/sglang/srt/models/deepseek_v4.py
+1 -1
View File
@@ -3230,7 +3230,7 @@ class DeepseekV4ForCausalLM(nn.Module):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
else:
self.lm_head = PPMissingLayer()
python/sglang/srt/models/deepseek_v4_dspark.py
+1 -1
View File
@@ -756,7 +756,7 @@ class DeepseekV4ForCausalLMDSpark(nn.Module):
config.vocab_size,
config.hidden_size,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
else:
self.embed_tokens: Optional[nn.Module] = None
python/sglang/srt/models/deepseek_v4_nextn.py
+1 -1
View File
@@ -250,7 +250,7 @@ class DeepseekV4ForCausalLMNextN(DeepseekV4ForCausalLM):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("model.shared_head.head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/dots3_common/modeling.py
+1 -1
View File
@@ -1876,7 +1876,7 @@ class Dots3LanguageModelForCausalLM(nn.Module):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/dots3_common/nextn.py
+1 -1
View File
@@ -167,7 +167,7 @@ class Dots3NoteForCausalLMNextN(Dots3LanguageModelForCausalLM):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("model.shared_head.head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
self._mtp_loaded_embed = False
python/sglang/srt/models/exaone4.py
+1 -1
View File
@@ -439,7 +439,7 @@ class Exaone4ForCausalLM(nn.Module):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/exaone_moe.py
+1 -1
View File
@@ -643,7 +643,7 @@ class ExaoneMoEForCausalLM(nn.Module):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
# For EAGLE3 support
python/sglang/srt/models/exaone_moe_mtp.py
+1 -1
View File
@@ -63,7 +63,7 @@ class ExaoneMoEForCausalLMMTP(ExaoneMoEForCausalLM):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/falcon_h1.py
+1 -1
View File
@@ -472,7 +472,7 @@ class FalconH1ForCausalLM(nn.Module):
quant_config=quant_config,
org_num_embeddings=config.vocab_size,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.lm_head = self.lm_head.float()
self.lm_head_multiplier = config.lm_head_multiplier
python/sglang/srt/models/glm4_moe.py
+1 -1
View File
@@ -1163,7 +1163,7 @@ class Glm4MoeForCausalLM(nn.Module):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/glm4_moe_lite.py
+1 -1
View File
@@ -905,7 +905,7 @@ class Glm4MoeLiteForCausalLM(nn.Module, DeepseekV2WeightLoaderMixin):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/glm4_moe_lite_nextn.py
+1 -1
View File
@@ -159,7 +159,7 @@ class Glm4MoeLiteForCausalLMNextN(Glm4MoeLiteForCausalLM):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("model.shared_head.head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/glm4_moe_nextn.py
+1 -1
View File
@@ -144,7 +144,7 @@ class Glm4MoeForCausalLMNextN(Glm4MoeForCausalLM):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("model.shared_head.head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/glm4v_moe.py
+1 -1
View File
@@ -70,7 +70,7 @@ class Glm4vMoeForConditionalGeneration(Glm4vForConditionalGeneration):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
else:
# ranks other than the last rank will have a placeholder layer
python/sglang/srt/models/glm_ocr_nextn.py
+1 -1
View File
@@ -135,7 +135,7 @@ class GlmOcrForConditionalGenerationNextN(GlmOcrForConditionalGeneration):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("model.shared_head.head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/gpt_oss.py
+2 -2
View File
@@ -258,7 +258,7 @@ class GptOssSparseMoeBlock(nn.Module):
hidden_states: torch.Tensor,
forward_batch: Optional[ForwardBatch] = None,
) -> torch.Tensor:
if get_parallel().dwdp_size > 1:
if get_parallel().config.dwdp_size > 1:
return self.forward_dwdp(hidden_states)
if not get_moe_a2a_backend().is_deepep():
@@ -786,7 +786,7 @@ class GptOssForCausalLM(nn.Module):
config.hidden_size,
# quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
self.capture_aux_hidden_states = False
python/sglang/srt/models/kimi_k25.py
+2 -2
View File
@@ -46,9 +46,9 @@ from sglang.srt.multimodal.mm_utils import (
run_dp_sharded_mrope_vision_model,
)
from sglang.srt.runtime_context import (
configured_tp_size,
get_exec,
get_mm,
get_parallel,
)
from sglang.srt.utils import add_prefix, is_cuda, is_npu
@@ -733,7 +733,7 @@ class KimiK25ForConditionalGeneration(nn.Module):
# Match the configured TP consumer count captured when the
# tokenizer creates MmItemMemoryPool. A live attention subgroup
# size could leave acknowledgements missing and strand the lease.
ipc_consumer_count = max(configured_tp_size(), 1)
ipc_consumer_count = max(get_parallel().config.tp_size, 1)
device_index = device.index
if device.type == "cuda" and device_index is None:
device_index = torch.cuda.current_device()
python/sglang/srt/models/kimi_k3.py
+5 -6
View File
@@ -119,7 +119,6 @@ from sglang.srt.multimodal.kimi_k3_image_processing import (
)
from sglang.srt.multimodal.mm_utils import materialize_multimodal_features
from sglang.srt.runtime_context import (
configured_tp_size,
get_exec,
get_parallel,
get_server_args,
@@ -298,7 +297,7 @@ class KimiK3MLP(nn.Module):
# but allow the NPU launcher to retain the proven attention-TP layout
# without a device-type branch in shared model code.
self._dense_attn_tp = (
get_parallel().enable_dense_mlp_attn_tp
get_parallel().config.enable_dense_mlp_attn_tp
and is_dp_attention_enabled()
and tp_rank is None
and tp_size is None
@@ -556,13 +555,13 @@ class KimiK3MoE(nn.Module):
# a TP-sharded partial sum could never be reduced across ranks that
# hold different tokens.
self._shared_experts_tp1 = (
self._ep_a2a and not get_parallel().enable_shared_experts_attn_tp
self._ep_a2a and not get_parallel().config.enable_shared_experts_attn_tp
)
# NPU compatibility mode keeps DeepEP's DP-local token dispatch but
# uses the original TP-sharded shared MLP. Gather only that branch's
# inputs, then reduce-scatter its output back to the DP-local rows.
self._shared_experts_attn_tp_comm = (
get_parallel().enable_shared_experts_attn_tp
get_parallel().config.enable_shared_experts_attn_tp
and self._ep_a2a
and self._dp_attention
and get_parallel().attn_tp_size > 1
@@ -2872,7 +2871,7 @@ class KimiK3LinearForCausalLM(nn.Module):
config.hidden_size,
quant_config=quant_config,
prefix=maybe_prefix(prefix, "lm_head"),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
else:
self.lm_head = PPMissingLayer()
@@ -3384,7 +3383,7 @@ class KimiK3ForConditionalGeneration(nn.Module):
# Match the configured TP consumer count captured when the
# tokenizer creates MmItemMemoryPool. A live attention subgroup
# size could leave acknowledgements missing and strand the lease.
ipc_consumer_count = max(configured_tp_size(), 1)
ipc_consumer_count = max(get_parallel().config.tp_size, 1)
device_index = device.index
if device.type == "cuda" and device_index is None:
device_index = torch.cuda.current_device()
python/sglang/srt/models/laguna.py
+1 -1
View File
@@ -667,7 +667,7 @@ class LagunaForCausalLM(nn.Module):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
else:
self.lm_head = PPMissingLayer()
python/sglang/srt/models/llada2.py
+1 -1
View File
@@ -824,7 +824,7 @@ class LLaDA2MoeModelLM(nn.Module):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config, return_full_logits=True)
python/sglang/srt/models/llama.py
+1 -1
View File
@@ -536,7 +536,7 @@ class LlamaForCausalLM(nn.Module):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
self.pooler = Pooler(pooling_type=PoolingType.LAST, normalize=True)
python/sglang/srt/models/longcat_flash.py
+1 -1
View File
@@ -721,7 +721,7 @@ class LongcatFlashForCausalLM(nn.Module):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
self.capture_aux_hidden_states = False
python/sglang/srt/models/mellum.py
+1 -1
View File
@@ -520,7 +520,7 @@ class MellumForCausalLM(Qwen3MoeForCausalLM):
cfg.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(cfg)
self.capture_aux_hidden_states = False
python/sglang/srt/models/mimo_v2.py
+1 -1
View File
@@ -1187,7 +1187,7 @@ class MiMoV2ForCausalLM(nn.Module, AudioEncoderMixin):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
else:
self.lm_head = PPMissingLayer()
python/sglang/srt/models/mimo_v2_nextn.py
+1 -1
View File
@@ -259,7 +259,7 @@ class MiMoV2MTP(MiMoV2ForCausalLM):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/minimax_m3.py
+1 -1
View File
@@ -1574,7 +1574,7 @@ class MiniMaxM3SparseForCausalLM(nn.Module):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/minimax_m3_vl.py
+1 -1
View File
@@ -123,7 +123,7 @@ class MiniMaxM3SparseForConditionalGeneration(nn.Module):
text_config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("language_model.lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
else:
self.lm_head = PPMissingLayer()
python/sglang/srt/models/nemotron_h.py
+1 -1
View File
@@ -964,7 +964,7 @@ class NemotronHForCausalLM(nn.Module):
else lora_config.lora_vocab_padding_size
),
quant_config=quant_config,
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
prefix=add_prefix("lm_head", prefix),
)
else:
python/sglang/srt/models/nemotron_h_mtp.py
+1 -1
View File
@@ -339,7 +339,7 @@ class NemotronHForCausalLMMTP(NemotronHForCausalLM):
self.config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/qwen2_moe.py
+1 -1
View File
@@ -1119,7 +1119,7 @@ class Qwen2MoeForCausalLM(nn.Module):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
# For EAGLE3 support
python/sglang/srt/models/qwen3.py
+1 -1
View File
@@ -492,7 +492,7 @@ class Qwen3ForCausalLM(nn.Module):
config.vocab_size,
config.hidden_size,
quant_config=quant_config,
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
prefix=add_prefix("lm_head", prefix),
)
else:
python/sglang/srt/models/qwen3_5_text.py
+1 -1
View File
@@ -80,7 +80,7 @@ class Qwen3_5ForCausalLM(nn.Module):
quant_config=quant_config,
org_num_embeddings=config.vocab_size,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
else:
self.lm_head = PPMissingLayer()
python/sglang/srt/models/qwen3_moe.py
+1 -1
View File
@@ -961,7 +961,7 @@ class Qwen3MoeForCausalLM(nn.Module):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
self.capture_aux_hidden_states = False
python/sglang/srt/models/qwen3_moe_mtp.py
+1 -1
View File
@@ -63,7 +63,7 @@ class Qwen3MoeForCausalLMMTP(Qwen3MoeForCausalLM):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
python/sglang/srt/models/qwen3_next.py
+1 -1
View File
@@ -1027,7 +1027,7 @@ class Qwen3NextForCausalLM(nn.Module):
quant_config=quant_config,
org_num_embeddings=config.vocab_size,
prefix=add_prefix("lm_head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
# For EAGLE3 support
python/sglang/srt/models/qwen3_next_mtp.py
+1 -1
View File
@@ -80,7 +80,7 @@ class Qwen3NextForCausalLMMTP(Qwen3NextForCausalLM):
config.hidden_size,
quant_config=quant_config,
prefix=add_prefix("model.shared_head.head", prefix),
use_attn_tp_group=get_parallel().enable_dp_lm_head,
use_attn_tp_group=get_parallel().config.enable_dp_lm_head,
)
self.logits_processor = LogitsProcessor(config)
# Mirror Qwen3NextForCausalLM.__init__'s shared-expert fusion setup so

Some files were not shown because too many files have changed in this diff Show More