config: keep runtime hicache and weight-version updates off ServerArgs (#33336)
The scheduler's runtime HiCache attach/detach wrote its own ServerArgs so the internal-state readback would show the change; that readback already reports the resolved config, so the writes become get_context().override(...) and the namespace readers see them too. The tokenizer side is per-engine — several Engines can share one process — so its control-plane updates (weight version, model path + load format, HiCache attach/detach) stay with the manager instead of moving to the process-global bags. TokenizerManager gains record_config_updates / config_value / resolved_config_dict, and the readbacks that used to observe the instance write (/server_info, /model_info, the HiCache status endpoint, the gRPC bridge) now overlay those updates onto the startup config. test_server_info's stub grew the real manager instead of a SimpleNamespace, so the overlay it now exercises cannot drift from production. Writer ratchet 26 -> 19.
This commit is contained in:
@@ -912,17 +912,23 @@ class RuntimeContext:
|
||||
"""Serialize the *resolved* config: the pristine ``server_args`` fields
|
||||
with every post-publish ``override`` overlaid.
|
||||
|
||||
Reporting endpoints (``/server_info``, ``get_internal_state``) surface
|
||||
the config the process is *currently* running, not the startup record,
|
||||
so they read this rather than serializing ``server_args`` directly —
|
||||
otherwise runtime updates (weight version, model path, tunables set via
|
||||
``/set_internal_state``) never show up in the readback.
|
||||
``get_internal_state`` reports this, and ``/server_info`` carries it in
|
||||
the ``internal_states`` block, so scheduler-side runtime changes show up
|
||||
in a readback: HiCache attach/detach, the generated forward-pass-metrics
|
||||
endpoint, tunables set via ``/set_internal_state``.
|
||||
|
||||
``base`` defaults to ``dict(vars(server_args))`` (matching the legacy
|
||||
``vars`` dump); pass ``dataclasses.asdict(server_args)`` when nested
|
||||
dataclass fields must be expanded first (``/server_info``). Override
|
||||
leaves are flat ``ServerArgs`` field names, so overlaying them onto the
|
||||
top level of either base is exact.
|
||||
dataclass fields must be expanded first. Override leaves are flat
|
||||
``ServerArgs`` field names, so overlaying them onto the top level of
|
||||
either base is exact.
|
||||
|
||||
This covers the process-global bags only. Per-engine control-plane
|
||||
changes (weight version, model path, the tokenizer's HiCache mirror)
|
||||
live on the tokenizer manager — several ``Engine``s can share one
|
||||
process — and ``TokenizerManager.resolved_config_dict`` overlays those
|
||||
for the top-level ``/server_info`` body. The two are separate logs, not
|
||||
one merged dict.
|
||||
"""
|
||||
d = dict(vars(self.server_args)) if base is None else dict(base)
|
||||
for _source, fields in self._overrides_log:
|
||||
|
||||
Reference in New Issue
Block a user