minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

config: the alias form of the runner-side instance read

Browse Source 
The previous batch counted `self.server_args.X` and called the runner surface
done. It was not: the same read spelled through a local alias --
`server_args = model_runner.server_args` (or `sa = kvc.server_args`, `args = ...`)
followed by `server_args.leaf` -- is the same process-global read wearing a
local name, and the AST census counts **57 of them** across eleven files that
the grep never saw. Census per function, following the alias.

52 were leaves and go to their bag (`spec` 11, `schedule` 9, `memory` 7,
`exec.graph` 5, `exec.moe` 5, `parallel` 4, `disagg` 4, `model` 3,
`exec.mamba` 2, `exec.overlap` 2). Five were not leaves:
three derived members on the eager runner --
`max_speculative_num_draft_tokens` and `enable_mamba_extra_buffer` already had
accessors, and `max_prefill_buffer_tokens` gets one (all its inputs are `schedule`
leaves plus the configured PP size, so it derives from the bags and follows a
post-publish override; `TestDerivedPredicatesAgreeAcrossTiers` pins it against
the member over a 48-case matrix) -- plus `get_attention_backends()`, which the
same commit routes through `attention_backends()`, and a dict that merely shares
the name (`server_args_dict.items`). That dict is the one read left behind.

`build_attention_backends` also stops resolving the pair from the record: it
runs after publish, so it asks `attention_backends()` like every other consumer.
The draft override on the runner still wins first.

`dispatch_event_loop`'s three PP checks read the *configured* PP size, not the
live topology: the MLX runner stub never initializes torch.distributed, so the
live property asserts before the MLX event loop can start (a Codex catch). The
configured leaf answers the same value wherever the live groups exist.

`flashinfer_gdn_prefill_default`'s guard is the one read here that asks what the
*operator* named rather than what the config resolved to, and the bag leaf now
answers exactly that: the per-runner auto-default is stamped on the runner and
deliberately never recorded process-wide, so nothing writes that leaf after
launch and reading it back cannot mistake another runner's default for a flag.

Three test doubles injected a `SimpleNamespace`/`MagicMock` record for exactly
these reads and now publish instead (pool configurator, cache registry, GDN
prefill policy) -- the fixture publishes what the case configures and hands the
published instance to the whole-object contracts that still take one.

The functions this sweep partially converted stop mixing sources (review
catches): the flash-attention constructor's remaining seed reads
(`speculative_eagle_topk`, `speculative_algorithm`, both deterministic gates)
read their bags next to the leaves already converted;
`_should_disable_scheduler_metadata_precompute` reads the parallel config
leaves itself instead of taking the record (its alias binding was the last
use); and the autotune gates (`disable_flashinfer_autotune`, deterministic,
`flashinfer_autotune_skip_ops`) join the moe leaves the same function already
reads from the bags. The pool-configurator fixture drops a parameter nothing
published or read.
This commit is contained in:
Cheng Wan
2026-08-15 00:39:03 -07:00
committed by GitHub
parent 61908870f6
commit f2ab6e306b
18 changed files with 319 additions and 147 deletions
Download Patch File Download Diff File Expand all files Collapse all files
python/sglang/srt/model_executor/runner/eager_runner.py
+16 -9
View File
@@ -49,6 +49,13 @@ from sglang.srt.model_executor.runner_backend_utils.tc_piecewise_cuda_graph impo
enable_tc_piecewise_cuda_graph,
set_tc_piecewise_forward_context,
)
from sglang.srt.runtime_context import (
get_parallel,
get_spec,
mamba_extra_buffer_enabled,
max_prefill_buffer_tokens,
max_speculative_num_draft_tokens,
)
from sglang.srt.utils import is_hip
from sglang.srt.utils.common import (
ceil_align,
@@ -76,14 +83,14 @@ class EagerRunner(BaseRunner):
num_tokens_per_req = 1
if mr.spec_algorithm.is_speculative():
# speculative_adaptive can grow draft tokens at runtime; size to the max.
num_draft_tokens = sa.max_speculative_num_draft_tokens or 1
num_draft_tokens = max_speculative_num_draft_tokens() or 1
if mr.is_draft_worker:
num_tokens_per_req = max(
sa.speculative_eagle_topk or 1,
get_spec().speculative_eagle_topk or 1,
num_draft_tokens,
(
2 * (sa.speculative_num_steps or 0)
if sa.enable_multi_layer_eagle
2 * (get_spec().speculative_num_steps or 0)
if get_spec().enable_multi_layer_eagle
else 0
),
)
@@ -100,14 +107,14 @@ class EagerRunner(BaseRunner):
if (
mr.is_draft_worker
and mr.spec_algorithm.is_frozen_kv_mtp()
and sa.speculative_eagle_topk > 1
and get_spec().speculative_eagle_topk > 1
):
# Frozen-KV MTP expands the draft batch by topk on the bs axis
# (expand_for_topk_draft) before the eager fallback.
max_bs *= sa.speculative_eagle_topk
max_bs *= get_spec().speculative_eagle_topk
# Mirror prepare_mlp_sync_batch padding so the registry holds what load_batch copies.
max_bs = get_eager_max_batch_size(sa, max_bs)
prefill_ceiling = max(mr.max_total_num_tokens, sa.max_prefill_buffer_tokens())
prefill_ceiling = max(mr.max_total_num_tokens, max_prefill_buffer_tokens())
max_num_token = max(prefill_ceiling, max_bs * num_tokens_per_req)
if require_mlp_sync(sa):
from sglang.srt.layers.cp.padding import get_cp_padding_align_size
@@ -123,7 +130,7 @@ class EagerRunner(BaseRunner):
max_num_token=max_num_token,
cache_loc_dtype=torch.int64,
enable_mamba_track=(
sa.enable_mamba_extra_buffer() and mr.spec_algorithm.is_none()
mamba_extra_buffer_enabled() and mr.spec_algorithm.is_none()
),
is_encoder_decoder=is_encoder_decoder,
encoder_len_fill_value=(
@@ -134,7 +141,7 @@ class EagerRunner(BaseRunner):
encoder_lens_dtype=(
torch.int64 if torch.device(mr.device).type == "cpu" else torch.int32
),
dp_size=sa.dp_size,
dp_size=get_parallel().dp_size,
)
# Eager has no capture step, so warm up here (run-once via mr._kernel_warmed_up).
self.warmup()