minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

config: keep runtime hicache and weight-version updates off ServerArgs (#33336)

Browse Source 
The scheduler's runtime HiCache attach/detach wrote its own ServerArgs so the
internal-state readback would show the change; that readback already reports
the resolved config, so the writes become get_context().override(...) and the
namespace readers see them too.

The tokenizer side is per-engine — several Engines can share one process — so
its control-plane updates (weight version, model path + load format, HiCache
attach/detach) stay with the manager instead of moving to the process-global
bags. TokenizerManager gains record_config_updates / config_value /
resolved_config_dict, and the readbacks that used to observe the instance write
(/server_info, /model_info, the HiCache status endpoint, the gRPC bridge) now
overlay those updates onto the startup config.

test_server_info's stub grew the real manager instead of a SimpleNamespace, so
the overlay it now exercises cannot drift from production.

Writer ratchet 26 -> 19.
This commit is contained in:
Cheng Wan
2026-08-02 21:23:38 -07:00
committed by GitHub
parent 9bc8848fcf
commit 0b3e8bedd1
16 changed files with 432 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files
python/sglang/srt/entrypoints/http_server.py
+14 -9
View File
@@ -710,12 +710,13 @@ async def model_info():
"tokenizer_path": _global_state.tokenizer_manager.server_args.tokenizer_path,
"is_generation": _global_state.tokenizer_manager.is_generation,
"preferred_sampling_params": _global_state.tokenizer_manager.server_args.preferred_sampling_params,
"weight_version": _global_state.tokenizer_manager.server_args.weight_version,
"weight_version": _global_state.tokenizer_manager.config_value(
"weight_version"
),
"has_image_understanding": model_config.is_image_understandable_model,
"has_audio_understanding": model_config.is_audio_understandable_model,
"model_type": getattr(model_config.hf_config, "model_type", None),
"architectures": getattr(model_config.hf_config, "architectures", None),
"weight_version": _global_state.tokenizer_manager.server_args.weight_version,
# "hf_config": model_config.hf_config.to_dict(),
}
embedding_model_spec = getattr(model_config, "embedding_model_spec", None)
@@ -761,7 +762,9 @@ async def server_info():
# server_args.model_config is not serializable but should be excluded by asdict.
return msgspec_to_builtins(
{
**dataclasses.asdict(server_args),
**_global_state.tokenizer_manager.resolved_config_dict(
dataclasses.asdict(server_args)
),
**_global_state.scheduler_info,
"internal_states": internal_states,
"version": __version__,
@@ -1091,10 +1094,13 @@ async def hicache_storage_backend_status():
return _admin_api_key_missing_response()
return {
"hicache_storage_backend": _global_state.tokenizer_manager.server_args.hicache_storage_backend,
"hicache_storage_backend_extra_config": _global_state.tokenizer_manager.server_args.hicache_storage_backend_extra_config,
"hicache_storage_prefetch_policy": _global_state.tokenizer_manager.server_args.hicache_storage_prefetch_policy,
"hicache_write_policy": _global_state.tokenizer_manager.server_args.hicache_write_policy,
name: _global_state.tokenizer_manager.config_value(name)
for name in (
"hicache_storage_backend",
"hicache_storage_backend_extra_config",
"hicache_storage_prefetch_policy",
"hicache_write_policy",
)
}
@@ -1385,8 +1391,7 @@ async def update_weight_version(
# Use a simple approach without the complex lock mechanism for now
# since weight_version update is a simple operation that doesn't affect model weights
try:
# Update the weight version in server args (the single source of truth)
_global_state.tokenizer_manager.server_args.override(
_global_state.tokenizer_manager.record_config_updates(
"http.update_weight_version", weight_version=obj.new_version
)