config: keep runtime hicache and weight-version updates off ServerArgs (#33336)

The scheduler's runtime HiCache attach/detach wrote its own ServerArgs so the
internal-state readback would show the change; that readback already reports
the resolved config, so the writes become get_context().override(...) and the
namespace readers see them too.

The tokenizer side is per-engine — several Engines can share one process — so
its control-plane updates (weight version, model path + load format, HiCache
attach/detach) stay with the manager instead of moving to the process-global
bags. TokenizerManager gains record_config_updates / config_value /
resolved_config_dict, and the readbacks that used to observe the instance write
(/server_info, /model_info, the HiCache status endpoint, the gRPC bridge) now
overlay those updates onto the startup config.

test_server_info's stub grew the real manager instead of a SimpleNamespace, so
the overlay it now exercises cannot drift from production.

Writer ratchet 26 -> 19.
This commit is contained in:
Cheng Wan
2026-08-02 21:23:38 -07:00
committed by GitHub
parent 9bc8848fcf
commit 0b3e8bedd1
16 changed files with 432 additions and 42 deletions
+14 -8
View File
@@ -912,17 +912,23 @@ class RuntimeContext:
"""Serialize the *resolved* config: the pristine ``server_args`` fields
with every post-publish ``override`` overlaid.
Reporting endpoints (``/server_info``, ``get_internal_state``) surface
the config the process is *currently* running, not the startup record,
so they read this rather than serializing ``server_args`` directly —
otherwise runtime updates (weight version, model path, tunables set via
``/set_internal_state``) never show up in the readback.
``get_internal_state`` reports this, and ``/server_info`` carries it in
the ``internal_states`` block, so scheduler-side runtime changes show up
in a readback: HiCache attach/detach, the generated forward-pass-metrics
endpoint, tunables set via ``/set_internal_state``.
``base`` defaults to ``dict(vars(server_args))`` (matching the legacy
``vars`` dump); pass ``dataclasses.asdict(server_args)`` when nested
dataclass fields must be expanded first (``/server_info``). Override
leaves are flat ``ServerArgs`` field names, so overlaying them onto the
top level of either base is exact.
dataclass fields must be expanded first. Override leaves are flat
``ServerArgs`` field names, so overlaying them onto the top level of
either base is exact.
This covers the process-global bags only. Per-engine control-plane
changes (weight version, model path, the tokenizer's HiCache mirror)
live on the tokenizer manager — several ``Engine``s can share one
process — and ``TokenizerManager.resolved_config_dict`` overlays those
for the top-level ``/server_info`` body. The two are separate logs, not
one merged dict.
"""
d = dict(vars(self.server_args)) if base is None else dict(base)
for _source, fields in self._overrides_log: