config: the last runner-side instance reads read the bags
Six reads were left on `self.server_args` outside the per-instance boundary the plan reserves for the tokenizer-manager family, and each had a different reason to be there: - `scheduler.process_input_requests` (`mm_feature_transport`) and `BaseSpecWorker._build_hicache_draft_plan` (`enable_hierarchical_cache`) are plain leaves -> `get_mm()` / `get_memory()`. - `DraftBackendFactory._create_backend` read the split backend through a *runtime-computed name* (`getattr(self.server_args, backend_name)`) and then fell back to the base field by hand -- the census's documented blind spot. The two names it can be handed are exactly the pair `attention_backends()` returns with that fallback already applied, so it reads the pair and indexes it. The draft runner's own stamp still wins when it has one. - `remote_instance_weight_loader_use_transfer_engine` and `pre_capture_activation_reserve_mb` are derived members. Both are computed from published leaves only, so both get a named accessor that derives from the bags (and therefore follows a post-publish override). The first of those two has all its inputs in one bag, so it follows the established shape: one `*_of(cfg)` helper in `arg_groups/overrides.py`, the `ServerArgs` member delegating to it, and the accessor calling it on `get_model()`. `modelexpress_transport_of` splits out the JSON parse both sides need. The second spans four bags plus the configured parallel sizes, so it exists twice like the mamba pair -- and `TestDerivedPredicatesAgreeAcrossTiers` now pins both new pairs equal over their input matrices (92 subtests). `self.server_args.X` outside the tokenizer-manager family: 11 -> 5, and the five that remain are the documented ones (the encode server's own record, the nixl connector's rank arithmetic, `GrammarManager`'s handed instance). The post-capture headroom path calls the same bag-backed `pre_capture_activation_reserve_mb` accessor the configurator uses -- the accessor advertises override-following, and a reserve that reads the record while its sibling reads the bags can disagree after a post-publish override. And the conversions' orphans go with them: `RemoteInstanceWeightTransporter` kept a `server_args` field nothing reads, and `DraftBackendFactory` parked a record it no longer consults -- both drop the parameter, and the four factory call sites stop threading one.
This commit is contained in:
@@ -1420,6 +1420,53 @@ def mamba_extra_buffer_lazy_enabled() -> bool:
|
||||
)
|
||||
|
||||
|
||||
def remote_instance_transfer_engine_enabled(load_format: str | None = None) -> bool:
|
||||
"""Whether remote-instance weight loading runs over the transfer engine.
|
||||
|
||||
Every input is a ``model`` leaf, so this derives from the bags and follows a
|
||||
post-publish override; ``ServerArgs.remote_instance_weight_loader_use_transfer_engine``
|
||||
is the pre-publish equivalent, and both go through the same helper.
|
||||
``load_format`` is the caller's own (a draft runner loading under
|
||||
``--speculative-draft-load-format`` has one the process record does not).
|
||||
"""
|
||||
from sglang.srt.arg_groups.overrides import remote_instance_transfer_engine_of
|
||||
|
||||
return remote_instance_transfer_engine_of(get_model(), load_format)
|
||||
|
||||
|
||||
def pre_capture_activation_reserve_mb(gpu_mem: float | None) -> float:
|
||||
"""The activation working-set reserve held back before cuda-graph capture.
|
||||
|
||||
Derived from published leaves across four bags (``disagg`` / ``schedule`` /
|
||||
``exec.graph`` / ``spec``) plus the configured parallel sizes, so it follows
|
||||
a post-publish override; ``ServerArgs.pre_capture_activation_reserve_mb`` is
|
||||
the pre-publish equivalent and
|
||||
``TestDerivedPredicatesAgreeAcrossTiers`` pins the two equal.
|
||||
"""
|
||||
schedule = get_schedule()
|
||||
if get_disagg().disaggregation_mode == "decode":
|
||||
running_requests = (
|
||||
schedule.max_running_requests
|
||||
or get_exec().graph.cuda_graph_config.decode.max_bs
|
||||
or 1
|
||||
)
|
||||
activation_tokens = max(
|
||||
running_requests * (get_spec().speculative_num_draft_tokens or 1), 2048
|
||||
)
|
||||
elif schedule.chunked_prefill_size > 0:
|
||||
activation_tokens = max(schedule.chunked_prefill_size, 2048)
|
||||
else:
|
||||
activation_tokens = max(schedule.max_prefill_tokens, 2048)
|
||||
reserved_mem = (
|
||||
512
|
||||
+ activation_tokens * 1.5
|
||||
+ _configured_parallel("tp_size") * _configured_parallel("pp_size") / 8 * 1024
|
||||
)
|
||||
if gpu_mem is not None and gpu_mem > 60 * 1024:
|
||||
reserved_mem = max(reserved_mem, 10 * 1024)
|
||||
return reserved_mem
|
||||
|
||||
|
||||
# --- Derived config accessors ------------------------------------------------
|
||||
#
|
||||
# A few values are computed from several config fields plus the HF config, so
|
||||
|
||||
Reference in New Issue
Block a user