[Config] Round 6.3: the record remembers how it was asked for, and is sealed while resolution runs (#38048)
Third of four; stacked on #38047. Two small changes, both about the same thing: the record holds the operator's input, and nothing else should be true of it. ## `/server_info` can answer what was actually typed It reports `resolved_dict()` -- what resolution decided. There was no way to ask the other question, and the two are not derivable from each other: a field nobody set reads the same as one set to the value resolution would have picked anyway. The launcher stores the arguments it parsed and the in-process `Engine` stores the call that built the record. All three readbacks report it beside the resolved values, so both surfaces come back in one request: HTTP `/server_info`, `Engine.get_server_info`, and the gRPC bridge's -- the last one builds from `resolved_dict()` and would otherwise have been the one surface of the three that answers only "what resolution decided". It rides on the record rather than in a field -- it describes how the configuration was asked for, so it is not part of the configuration: no CLI flag, no namespace, not in the bags. Being on the record is what lets a subprocess copy answer the same question the launcher can, and `replace_resolved` carries it because a copy was launched by whatever launched its parent. The crash dump already collected all four surfaces (`server_args`, `config_updates`, `resolved_config`, `launch_command`); this is the one that `/server_info` was missing. ## The record is sealed for the length of resolution The read-only guard armed on `_resolution_finished`, so for the whole run of the pipeline nothing stopped a resolver from assigning a field. Nothing in `srt/` does -- 0 assignments statically, and 0 writes observed across the launch-shape matrix with a watching `__setattr__` -- but that was a convention, and the defect it permits is invisible: a value a resolver wrote onto the record is indistinguishable from a value the operator typed, which is the one distinction the record exists to preserve. It now arms when resolution starts. A resolver that assigns a field fails at boot with a message naming `declare_resolution`, which is where the decision belongs: the stash carries a source and leaves the input intact. `declare_direct_writes` asks for the seal by name through `record_writable`. It hands the record to an out-of-tree platform plugin that sets fields on it; those implementations cannot be converted by editing a resolver here, so the write stays and the diff is captured into the stash afterwards. Naming the exception is the point -- an in-tree resolver reaching for it is doing something it should be declaring. ## Verification Costs nothing: the 211 test-side assignments all happen before `resolve_once`, which a post-resolution write already refused. A full registered-unit sweep (648 files) against the stack's merge-base: 19 failures on both sides, the same 19, none of them config. Driving a deliberate write into a real handler produces the new error, so the seal is tested by more than its own unit test.
This commit is contained in:
@@ -250,7 +250,13 @@ def declare_direct_writes(
|
||||
for field in dataclasses.fields(server_args)
|
||||
}
|
||||
already = len(getattr(server_args, "_resolved_overrides", None) or ())
|
||||
result = resolve(server_args)
|
||||
# The one place the input seal comes off. The plugin writes the record;
|
||||
# the diff below captures what it moved into the stash so the projection
|
||||
# and the bags carry it.
|
||||
from sglang.srt.server_args import record_writable
|
||||
|
||||
with record_writable(server_args):
|
||||
result = resolve(server_args)
|
||||
stash = getattr(server_args, "_resolved_overrides", None)
|
||||
if stash is None:
|
||||
stash = []
|
||||
|
||||
@@ -261,6 +261,14 @@ class Engine(EngineScoreMixin, EngineBase):
|
||||
# Do not print logs by default
|
||||
kwargs["log_level"] = "error"
|
||||
server_args = self.server_args_class(**kwargs)
|
||||
# There was no command line, so the call is what the operator
|
||||
# asked for. `log_level` is filled in above when absent, so it
|
||||
# shows here even when the caller did not pass it.
|
||||
object.__setattr__(
|
||||
server_args,
|
||||
"_launch_command",
|
||||
"Engine(" + ", ".join(f"{k}={v!r}" for k, v in kwargs.items()) + ")",
|
||||
)
|
||||
self.server_args = server_args
|
||||
logger.info(f"server_args={server_args.resolved_dict()}")
|
||||
|
||||
@@ -1362,6 +1370,7 @@ class Engine(EngineScoreMixin, EngineBase):
|
||||
return msgspec_to_builtins(
|
||||
{
|
||||
**self.tokenizer_manager.server_args.resolved_dict(),
|
||||
"launch_command": self.tokenizer_manager.server_args.launch_command,
|
||||
**self._scheduler_init_result.scheduler_infos[0],
|
||||
"startup_time": self.tokenizer_manager.startup_time,
|
||||
"internal_states": internal_states,
|
||||
|
||||
@@ -425,6 +425,11 @@ class RuntimeHandle:
|
||||
|
||||
def get_server_info(self) -> str:
|
||||
result: Dict[str, Any] = self.tokenizer_manager.server_args.resolved_dict()
|
||||
# `resolved_dict` answers with what resolution decided; the launch
|
||||
# command answers with what was asked for, and the two are not
|
||||
# derivable from each other. The HTTP and in-process readbacks both
|
||||
# carry it, so this one does too.
|
||||
result["launch_command"] = self.tokenizer_manager.server_args.launch_command
|
||||
result.update(self.scheduler_info)
|
||||
result["kv_events"] = describe_kv_events_publisher(
|
||||
self.tokenizer_manager.server_args
|
||||
|
||||
@@ -812,8 +812,10 @@ async def get_server_info():
|
||||
async def server_info():
|
||||
"""The startup configuration, plus live scheduler state.
|
||||
|
||||
The values here are the resolution result: what the launcher was given,
|
||||
with every decision resolution made applied over it. Fields the control plane changes
|
||||
Two surfaces, deliberately both: the field values are the resolution
|
||||
result -- what the launcher was given with every decision resolution made
|
||||
applied over it -- and `launch_command` is what was actually asked for,
|
||||
which no amount of reading the resolved values recovers. Fields the control plane changes
|
||||
after publication -- the model a weight update swapped in, its load format,
|
||||
an operator-set weight version -- are reported by `/model_info`, and the
|
||||
HiCache mirror by `GET /hicache/storage-backend`.
|
||||
@@ -828,6 +830,7 @@ async def server_info():
|
||||
return msgspec_to_builtins(
|
||||
{
|
||||
**server_args.resolved_dict(),
|
||||
"launch_command": server_args.launch_command,
|
||||
**_global_state.scheduler_info,
|
||||
"startup_time": _global_state.tokenizer_manager.startup_time,
|
||||
"internal_states": internal_states,
|
||||
|
||||
@@ -40,6 +40,7 @@ import functools
|
||||
import logging
|
||||
import tempfile
|
||||
import uuid
|
||||
from contextlib import contextmanager
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from sglang.kernels.ops.kv_canary.consts import RealKvHashMode
|
||||
@@ -269,6 +270,11 @@ class ServerArgs:
|
||||
)
|
||||
from sglang.srt.arg_groups.pipeline import run_resolution_pipeline
|
||||
|
||||
# Sealed for the duration, not just afterwards: everything below this
|
||||
# line reads the input and declares against it, and the one channel
|
||||
# that still writes the record (`declare_direct_writes`, for
|
||||
# out-of-tree platform plugins) asks for the seal to be lifted by name.
|
||||
self._input_frozen = True
|
||||
try:
|
||||
run_resolution_pipeline(self)
|
||||
except BaseException:
|
||||
@@ -276,11 +282,30 @@ class ServerArgs:
|
||||
# idempotent over their own output.
|
||||
self._resolution_failed = True
|
||||
raise
|
||||
finally:
|
||||
self._input_frozen = False
|
||||
# Set here too, because the dummy/absent-model path returns before the
|
||||
# end of the pipeline that normally sets it: the gate is about whether
|
||||
# the handlers ran, not how far they got.
|
||||
self._resolution_finished = True
|
||||
|
||||
@property
|
||||
def launch_command(self) -> Optional[str]:
|
||||
"""How this record was created, verbatim.
|
||||
|
||||
`resolved_dict` answers with what resolution decided; this answers with
|
||||
what the operator asked for, which is a different question and the one
|
||||
"why is this server configured like this?" usually means. The two are
|
||||
not derivable from each other: a field the operator never set reads the
|
||||
same as one they set to the value resolution would have picked anyway.
|
||||
|
||||
The launcher stores the arguments it parsed; the in-process `Engine`
|
||||
stores the call that built the record, since there was no command line.
|
||||
`None` on a record built directly (a fixture, a subprocess copy that
|
||||
predates this, a config being inspected).
|
||||
"""
|
||||
return getattr(self, "_launch_command", None)
|
||||
|
||||
def resolved_dict(self) -> Dict[str, Any]:
|
||||
"""This configuration as a plain dict of resolved field values.
|
||||
|
||||
@@ -318,6 +343,9 @@ class ServerArgs:
|
||||
copy's deep structure in-process mutates the parent's too.
|
||||
"""
|
||||
replacement = dataclasses.replace(self, **changes)
|
||||
# Provenance, not resolution state: a copy was still launched by
|
||||
# whatever launched its parent, resolved or not.
|
||||
object.__setattr__(replacement, "_launch_command", self.launch_command)
|
||||
if not getattr(self, "_resolution_finished", False):
|
||||
# Not resolved yet: the copy goes through the gate itself.
|
||||
return replacement
|
||||
@@ -669,19 +697,28 @@ class ServerArgs:
|
||||
return cfg.startup_weight_load_mode == "overlap"
|
||||
|
||||
def __setattr__(self, name, value):
|
||||
# Once resolution has finished the record is the READ-ONLY raw input
|
||||
# the config bags were projected from. Resolved config changes go to the bags via
|
||||
# get_context().override(source, ...); a value one runner or worker
|
||||
# owns travels as a constructor argument to it.
|
||||
if getattr(self, "_resolution_finished", False) and (
|
||||
not name.startswith("_") or name in _underscore_field_names()
|
||||
):
|
||||
raise AttributeError(
|
||||
f"server_args.{name} assigned after resolution; server_args is "
|
||||
"read-only -- use get_context().override(source, ...) to change "
|
||||
"resolved config; a value one runner owns travels as a "
|
||||
"constructor argument."
|
||||
)
|
||||
# The record holds the operator's input. It is writable while the
|
||||
# caller is still assembling it and sealed from the moment resolution
|
||||
# starts: a resolver that writes a field would overwrite the very thing
|
||||
# the record exists to remember, and the decision it meant to record
|
||||
# belongs in the stash, where it carries a source and does not destroy
|
||||
# the input it was derived from.
|
||||
if not name.startswith("_") or name in _underscore_field_names():
|
||||
if getattr(self, "_input_frozen", False):
|
||||
raise AttributeError(
|
||||
f"server_args.{name} assigned during resolution; the record "
|
||||
"is the operator's input and resolution does not write it -- "
|
||||
"declare the decision with declare_resolution(server_args, "
|
||||
"source, **fields) so it carries a source and leaves the "
|
||||
"input intact."
|
||||
)
|
||||
if getattr(self, "_resolution_finished", False):
|
||||
raise AttributeError(
|
||||
f"server_args.{name} assigned after resolution; server_args is "
|
||||
"read-only -- use get_context().override(source, ...) to change "
|
||||
"resolved config; a value one runner owns travels as a "
|
||||
"constructor argument."
|
||||
)
|
||||
object.__setattr__(self, name, value)
|
||||
|
||||
def enable_mamba_extra_buffer(self) -> bool:
|
||||
@@ -834,6 +871,27 @@ def get_global_server_args() -> ServerArgs:
|
||||
return get_context().server_args
|
||||
|
||||
|
||||
@contextmanager
|
||||
def record_writable(server_args: Any):
|
||||
"""Lift the input seal for a resolver that genuinely writes the record.
|
||||
|
||||
There is exactly one: `declare_direct_writes`, which hands the record to an
|
||||
out-of-tree platform plugin that sets fields on it. Those implementations
|
||||
live outside this tree and cannot be converted by editing a resolver here,
|
||||
so the write stays and is captured into the stash afterwards. Naming the
|
||||
exception is the point -- an in-tree resolver that reaches for this is
|
||||
doing something it should be declaring instead.
|
||||
"""
|
||||
frozen = getattr(server_args, "_input_frozen", False)
|
||||
if frozen:
|
||||
object.__setattr__(server_args, "_input_frozen", False)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
if frozen:
|
||||
object.__setattr__(server_args, "_input_frozen", True)
|
||||
|
||||
|
||||
def prepare_server_args(argv: List[str]) -> ServerArgs:
|
||||
"""
|
||||
Prepare the server arguments from the command line arguments.
|
||||
@@ -868,7 +926,12 @@ def prepare_server_args(argv: List[str]) -> ServerArgs:
|
||||
force=True,
|
||||
)
|
||||
|
||||
return ServerArgs.from_cli_args(raw_args)
|
||||
server_args = ServerArgs.from_cli_args(raw_args)
|
||||
# Not a field: the record's fields are the configuration, and this is how
|
||||
# the configuration was asked for. It rides along on the record so a
|
||||
# subprocess copy can answer the same question the launcher can.
|
||||
object.__setattr__(server_args, "_launch_command", " ".join(argv))
|
||||
return server_args
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
Reference in New Issue
Block a user