docs: rewrite the runtime-context skill for the namespace-bag config model (#33173)
The skill still described the retired resolve-at-end contract (a single resolved ServerArgs as the source of truth, ServerArgs.override as the mutation entry). Rewrite the config sections for the landed model: - ServerArgs is a pristine published seed; resolved config lives in the namespace bags (get_exec()/get_memory()/.../get_device()), projected at publish(server_args, role=...) per process entry. - Post-publish mutation goes through get_context().override (bag-only); ServerArgs.override is being retired behind the writer ratchet, and rerouting a writer co-flips all its readers in the same commit. - Documented the reads that stay on an instance (per-runner fork fields, per-instance tokenizer/entrypoints boundaries, whole-object passes), get_parallel()'s config/live dual semantics with the live-shadowed sizes rule, preserve_config for nested publishes, and the bag-only declare_load_time_override route. - Testing idioms: publish-seeding (override_server_args + scoped bag overrides) instead of faking accessors or SimpleNamespace stand-ins; per-file test runs to avoid leaked-publish masking. - Guardrails: added the writer ratchet, namespace-coverage lint, and the migration-deferral ratchet.
This commit is contained in:
Cheng Wan
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: sglang-runtime-context
|
||||
description: How SGLang's runtime configuration and process-global state are organized (RuntimeContext tiers, resolve-at-end ServerArgs, override entry point, resource/stream/buffer leases, per-forward flags), the CI guardrails that enforce the design, and the idioms for developing and testing against it. Load this before touching server_args, model overrides, module-level state, or per-forward state in sglang.
|
||||
description: How SGLang's runtime configuration and process-global state are organized (RuntimeContext tiers, publish + namespace config bags, the pristine ServerArgs seed, override entry points, resource/stream/buffer leases, per-forward flags), the CI guardrails that enforce the design, and the idioms for developing and testing against it. Load this before touching server_args, model overrides, module-level state, or per-forward state in sglang.
|
||||
---
|
||||
|
||||
# SGLang runtime-context architecture
|
||||
@@ -10,37 +10,98 @@ One container owns process-static runtime state: `sglang.srt.runtime_context.Run
|
||||
|
||||
| Tier | Accessor | Holds | Lifecycle |
|
||||
|------|----------|-------|-----------|
|
||||
| config | `get_server_args()` | the process-wide **resolved** `ServerArgs` | resolved once in `__post_init__`; audited mutations only |
|
||||
| raw config seed | `get_server_args()` | the published **pristine** `ServerArgs` (resolved-at-startup record; kept for debugging, dumps, per-runner fork copies) | published at process entry; re-publish is **last-publish-wins** (in-process tokenizer build, multi-Engine) and re-projects the bags; read-only |
|
||||
| resolved config | `get_exec()` `get_memory()` `get_schedule()` `get_model()` `get_spec()` `get_serving()` `get_observability()` `get_disagg()` `get_lora()` `get_mm()` `get_device()` | namespace **config bags** — the single source of truth for resolved config; leaves are real attributes (dynamo-traceable) | projected from `server_args` at `publish`; mutated only via `get_context().override` |
|
||||
| runtime flags | `get_flags()` | state that is *not* a pure function of config: `capture` (cuda-graph lifecycle), `moe` (ACTIVE backends, swappable), `dp` (DP-attention runtime flags) | materialized at subsystem init; groups offer `override()` for tests |
|
||||
| resources | `get_resources()`, `get_stream(name)`, `get_buffer(name, factory)` | process-level handles: graph pools, EPLB state, EP dispatcher state, named side streams, workspace buffers | lazy; cleared by `reset_context()` |
|
||||
| per-forward | `get_forward()` | forward-scoped flags (multi-stream switch, MoE output buffer, attn-TP inputs, extend-in-batch) | contextvar-backed; `scoped(**kw)` restores on exit; new threads see defaults |
|
||||
| parallel | `get_parallel()` | read-through wrapper over topology getters (tp/pp/moe/attn sizes, ranks, groups) | stateless; `override()` for tests |
|
||||
| parallel | `get_parallel()` | **dual**: live topology (tp/pp/moe/attn sizes, ranks, groups — `@property`, read-through) *plus* parallel config-bag leaves via `__getattr__` | live: after dist init; config leaves: after publish |
|
||||
|
||||
`reset_context()` (unit-test teardown) drops the published server_args and installs fresh
|
||||
`reset_context()` (unit-test teardown) drops the published config and installs fresh
|
||||
flags/resources/forward tiers.
|
||||
|
||||
## Config: the resolve-at-end contract
|
||||
## Config: publish + namespace bags
|
||||
|
||||
**After `ServerArgs.__post_init__` returns, the fields ARE the resolved configuration.**
|
||||
Model overrides and normalization passes *declare* values during resolution (into a
|
||||
provenance stash); `materialize_declarations()` applies them once at the very end of
|
||||
`__post_init__` (gate order, last writer wins). Consequences:
|
||||
**`ServerArgs` is a pristine seed. Business code never reads it for decisions —
|
||||
resolved configuration lives in the namespace bags.**
|
||||
|
||||
- **Reading config**: read fields directly, in any process, at any time after construction.
|
||||
For global access use `runtime_context.get_server_args()` (the blessed accessor —
|
||||
`get_global_server_args()` is a legacy shim over the same slot and its call-site count is
|
||||
ratcheted; do not add new ones).
|
||||
- **Mutating config after resolution**: the ONLY entry point is
|
||||
`ServerArgs.override(source, **fields)`. It records provenance (`_runtime_mutations`),
|
||||
keeps whitelisted resolvable fields consistent with the declaration stash (so a republish
|
||||
resolves the same values), and bypasses the strict guard. Bare `server_args.x = ...`
|
||||
after resolution **raises** under `SGLANG_STRICT_CONFIG_MUTATION=1`, which the test
|
||||
harness (`sglang.test.test_utils`) turns on by default.
|
||||
- **Mid-resolution code** (inside `server_args.py` / `arg_groups/` only): fields are
|
||||
read-only during resolution; handlers and hooks read the in-flight state through
|
||||
`resolved_view(server_args)` / `self._resolved()`. This is pipeline-internal — never use
|
||||
`resolved_view` outside the pipeline. (One sanctioned exception: helpers that the pipeline
|
||||
itself invokes mid-resolution, e.g. `adaptive_spec_params`.)
|
||||
- Every publishing process entry calls `publish(server_args, role=...)`
|
||||
(`run_scheduler_process`, the Ray `SchedulerActor`, the DP controller, tokenizer,
|
||||
encoder, weight-cache daemon, launcher, ...). The one deliberate exception is the
|
||||
detokenizer: its processes never publish and read only the raw config handed to
|
||||
their constructors — code that can run detokenizer-side must not use the
|
||||
namespace accessors. `publish` snapshots the resolved field
|
||||
values into the config bags; the accessors (`get_exec()` etc.) fail closed before it
|
||||
runs. `role` records which process type published, and keys per-role namespace
|
||||
enforcement: `SGLANG_ROLE_NAMESPACES=record` audits which namespaces each role's
|
||||
process actually reads (per-pair persisted via `SGLANG_ROLE_NAMESPACES_OUT`;
|
||||
reads inside torch.compile-traced code are NOT observed — audit with
|
||||
compilation disabled before restricting a role), and
|
||||
`=enforce` fails closed on bag reads outside the role's `ROLE_NAMESPACE_SETS` entry
|
||||
(`None` = full tree; only audited roles are restricted).
|
||||
- Bag membership is metadata on the dataclass: every `ServerArgs` field carries
|
||||
`NS("path")` (e.g. `NS("exec.moe")`); coverage is linted two-way
|
||||
(`test_server_args_namespaces.py`, `test_runtime_context_config_bags.py`).
|
||||
- **Reading config**: `get_<ns>()[.sub].field` — e.g.
|
||||
`get_exec().moe.moe_a2a_backend`, `get_schedule().max_running_requests`. Bag leaves
|
||||
are plain instance attributes, safe inside `torch.compile`-traced code.
|
||||
- **Mutating config after publish**: the ONLY entry point is
|
||||
`get_context().override(source, **fields)`. It writes the bag leaves in place
|
||||
(namespace readers see the new value) and records provenance in the overrides log.
|
||||
There is **no write-through** to the `ServerArgs` instance — it stays pristine.
|
||||
`ServerArgs.override(...)` (instance-only) is being retired; its call sites are
|
||||
ratcheted down and new ones are rejected.
|
||||
- **Nested publishes**: a construction step that must publish a private copy (the
|
||||
draft-worker build publishes the draft's rewritten config for the duration of the
|
||||
build) wraps itself in `get_context().preserve_config()` — the enclosing lifecycle,
|
||||
including its post-publish overrides, is value-snapshotted and reinstated on exit.
|
||||
|
||||
### Reads that legitimately stay on a `ServerArgs` instance
|
||||
|
||||
- **Per-runner (fork) fields** — fields the draft-worker deepcopy rewrites
|
||||
(`attention_backend`, `prefill/decode_attention_backend`,
|
||||
`speculative_draft_attention_backend`, `skip_tokenizer_init`, `context_length`,
|
||||
`load_format`, `json_model_override_args`, `kv_cache_dtype`): each runner's copy is
|
||||
authoritative for that runner, so runner code reads `self.server_args.X`, and
|
||||
*resolved* per-runner values live as runner attributes
|
||||
(`model_runner.kv_cache_dtype_str` is the pattern — threaded to consumers as
|
||||
constructor args, never backfilled onto shared objects).
|
||||
- **Per-instance boundaries** — the tokenizer-manager family, everything under
|
||||
`entrypoints/`, and the tokenizer-process multimodal processors read
|
||||
`self.server_args`: several `Engine`s can share one process, and the process-global
|
||||
bags are last-publish-wins across engines. (The mm-processor boundary is not yet
|
||||
airtight: `BaseMultimodalProcessor.process_mm_data` still reads `base_gpu_id` /
|
||||
`rl_on_policy_target` through `get_server_args()` — a known last-publish-wins gap,
|
||||
not a pattern to copy.)
|
||||
- **Whole-object passes** (`f(server_args)` handing the instance along) keep the
|
||||
supplied-instance contract; don't rewrite the parameter reads to bag reads unless the
|
||||
field is runtime-mutated (see the elastic-EP `ep_size` case in
|
||||
`eplb/expert_location.py`).
|
||||
|
||||
### `get_parallel()`: config leaves vs live topology
|
||||
|
||||
Config leaves (`nccl_port`, `enable_dp_attention`, `dp_size`, `ep_size`,
|
||||
`dwdp_size`, ...) resolve through the parallel bag; live topology (`tp_size`,
|
||||
`attn_tp_group`, ranks) are `@property` and **win on name collisions**. Five topology
|
||||
sizes are live-shadowed (`tp/pp/dcp/attn_cp/moe_dp_size`): a config-intent read of
|
||||
those must stay on `server_args.X` — the live property always wins on the accessor.
|
||||
Fail-loud is narrower: before dist init, any live size/group read raises; after it,
|
||||
only the DCP group is optional (`_DCP` exists only when `dcp_size > 1`; attn-CP and
|
||||
moe-DP always install, as size-1 aliases if unused). `ParallelContext.__getattr__` is deliberately dynamo-traceable (no
|
||||
`object.__getattribute__`); gate helpers like `enable_moe_dense_fully_dp()` run inside
|
||||
compiled model forwards (`test_parallel_config_leaves_trace_under_torch_compile` pins
|
||||
this).
|
||||
|
||||
### Mid-resolution reads (inside the pipeline only)
|
||||
|
||||
Resolution itself still runs in `__post_init__`: handlers and hooks read the
|
||||
in-flight state through `resolved_view(server_args)` / `self._resolved()`, fields are
|
||||
read-only during resolution, and declarations materialize once at the very end of
|
||||
`__post_init__` (gate order, last writer wins) — *then* `publish` snapshots the
|
||||
resolved values into the bags. `resolved_view` is pipeline-internal
|
||||
(`server_args.py` / `arg_groups/`, plus helpers the pipeline itself invokes
|
||||
mid-resolution, e.g. `adaptive_spec_params`); do not introduce new
|
||||
out-of-pipeline call sites.
|
||||
|
||||
### Adding a model-specific config adjustment
|
||||
|
||||
@@ -53,7 +114,13 @@ Never assign `server_args` fields from model code. Declare instead
|
||||
- Normalization that must see earlier declarations → a post-process pass invoked via
|
||||
`run_post_process_pass` at its slot (reads a view, returns a declaration dict).
|
||||
- Values only knowable at weight-load time → `declare_load_time_override(source, {...})`
|
||||
(validates the whitelist, routes through `override()`).
|
||||
— validates the whitelist, then routes through `get_context().override` (**bag-only**;
|
||||
the declaration lands on the published bags, not on any `ServerArgs` instance).
|
||||
Scope caveat for draft models: only a draft build that publishes a private copy
|
||||
under `preserve_config` discards its declarations with the scope. Draft loads
|
||||
that skip publish share the process bags, so their declarations land
|
||||
process-wide — declares reachable from a draft load must be draft-safe (guard
|
||||
or same-value).
|
||||
|
||||
Declarable fields form a whitelist: `Arg(..., resolvable=True)` in the `ServerArgs`
|
||||
dataclass. A declaration against a non-whitelisted field fails at its slot.
|
||||
@@ -63,14 +130,14 @@ dataclass. A declaration against a non-whitelisted field fails at its slot.
|
||||
`__post_init__` runs in the launcher process before any model/platform import. Logic that
|
||||
consults an **extensible registry** (e.g. out-of-tree platforms registering attention
|
||||
backends in `init_backend()`, which runs at `model_runner` import) must stay at load time
|
||||
(ModelRunner init), writing through `override()`. Before moving any load-time logic into
|
||||
resolution, verify everything it reads is already complete at construction time.
|
||||
(ModelRunner init), writing through `get_context().override()`. Before moving any
|
||||
load-time logic into resolution, verify everything it reads is already complete at
|
||||
construction time.
|
||||
|
||||
## Runtime flags (`get_flags()`)
|
||||
|
||||
For state that init-time code *derives* and runtime code reads — parsed enums, platform
|
||||
probes, swappable ACTIVE values. Not for config mirrors (those died with resolve-at-end:
|
||||
read the field).
|
||||
probes, swappable ACTIVE values. Not for config mirrors (read the bag leaf instead).
|
||||
|
||||
- Groups are typed dataclasses on `Flags` (`capture` / `moe` / `dp`): typo-safe writes,
|
||||
transactional test-only `override(**kw)` context manager.
|
||||
@@ -115,46 +182,65 @@ ONE thread — do not design for TBO threads that don't exist.
|
||||
## Testing idioms
|
||||
|
||||
- **Force a code path by overriding causes, not effects**: compose
|
||||
`get_context().override_server_args(**fields)` (config tier: publishes a fresh
|
||||
dummy-boundary `ServerArgs` carrying the overrides — `with`-scoped, or
|
||||
`install()`/`restore()` for fixture-lifetime use) + `get_parallel().override(...)` +
|
||||
`get_flags().<group>.override(...)` + `get_forward().scoped(...)` +
|
||||
`get_resources().override(...)`. All are scoped and transactional. Tests control
|
||||
execution through the context — do not hand-build and publish config objects.
|
||||
Note `override_server_args` is itself transitional (to be deprecated): it exists
|
||||
while production still branches on raw `server_args` fields at runtime; prefer the
|
||||
finer-grained tier overrides wherever they already cover the path you need.
|
||||
- **Never monkeypatch import bindings** (`module.get_x = lambda: ...`): production code may
|
||||
read a different accessor over the same slot and your patch silently stops intercepting.
|
||||
Publish/inject for real: `get_context().override_server_args(...)` for config;
|
||||
`monkeypatch.setattr(get_resources(), "slot", fake)` for resources.
|
||||
- Fixtures standing in for `ServerArgs` need an `override` method if the code under test
|
||||
mutates config (`_fake_server_args`-style: SimpleNamespace + write-through override).
|
||||
MagicMock swallows `override()` calls silently — prefer SimpleNamespace so misses raise.
|
||||
- `reset_context()` in teardown; `_IsolatedServerArgs`-style save/restore when a test
|
||||
publishes.
|
||||
`get_context().override_server_args(**fields)` (publishes a fresh dummy-boundary
|
||||
`ServerArgs` carrying the overrides AND projects the bags — `with`-scoped, or
|
||||
`install()`/`restore()` + `addCleanup` for fixture-lifetime use) +
|
||||
`get_<ns>().override(...)` (scoped override of one bag's own leaves) +
|
||||
`get_parallel().override(...)` (live topology) + `get_flags().<group>.override(...)` +
|
||||
`get_forward().scoped(...)`. All are scoped and transactional. Tests control execution
|
||||
through the context — do not hand-build and publish config objects.
|
||||
- **Never monkeypatch import bindings** (`module.get_x = lambda: ...`) and never fake a
|
||||
config source with a `SimpleNamespace` stand-in: production reads the published bags,
|
||||
so a faked accessor silently stops intercepting after any reader migration. Publish
|
||||
for real (`override_server_args(...)`), then adjust bag leaves with the scoped bag
|
||||
`override` where the constructed `ServerArgs` cannot carry the value (e.g.
|
||||
`get_device().override(device="meta")`).
|
||||
- Mocked runners/managers still need the **per-runner instance attributes** the code
|
||||
under test reads (`kv_cache_dtype_str`, `server_args` for whole-object passes) — set
|
||||
them explicitly on the mock; `MagicMock(spec=...)` raises on attributes that only
|
||||
exist post-`__init__`, which is the fastest way to find a missed stub.
|
||||
- `reset_context()` in teardown when a test publishes outside a scoped override.
|
||||
- `ServerArgs(model_path="dummy")` early-returns `__post_init__` (no materialization, no
|
||||
strict guard) — fine for lightweight fixtures.
|
||||
- **Run changed test files per-file** (own process), the way CI does: a monolithic local
|
||||
pytest run lets a context published by an earlier file mask a missing-publish bug in a
|
||||
later one.
|
||||
|
||||
## Guardrails (these fail CI; what to do when they fire)
|
||||
|
||||
1. **Strict mutation guard** (`SGLANG_STRICT_CONFIG_MUTATION=1`, default-on in tests):
|
||||
bare `server_args.x = ...` after resolution raises. Fix: route through
|
||||
`override(source, ...)`, or move genuinely config-decidable logic into resolution.
|
||||
1. **Strict mutation guard** (always on): bare `server_args.x = ...` after resolution
|
||||
raises unconditionally — `ServerArgs.__setattr__` no longer consults
|
||||
`SGLANG_STRICT_CONFIG_MUTATION` (the env var survives only as a legacy harness
|
||||
flag). Projected bags are sealed the same way (leaf assignment raises — write via
|
||||
`get_context().override`).
|
||||
2. **Mutation ratchet** (`test_server_args_mutation_ratchet.py`, exact pin 0 over the whole
|
||||
package minus the pipeline / multimodal_gen / the mock-fixture factory): textual scan
|
||||
for assignment forms. Never raise the baseline.
|
||||
3. **Legacy-accessor ratchet** (`test_legacy_global_ratchet.py`): `get_global_server_args`
|
||||
call sites must not grow — new code uses `runtime_context.get_server_args()`.
|
||||
4. **Module-state ratchet** (`test_module_state_ratchet.py`): `global` statements in the
|
||||
package minus the pipeline / multimodal_gen): textual scan for assignment forms. Never
|
||||
raise the baseline.
|
||||
3. **Writer ratchet** (`test_server_args_writer_ratchet.py`): `ServerArgs.override`
|
||||
call sites are pinned exactly and may only shrink — instance writes never reach the
|
||||
bags, so namespace readers desync from the writer. New post-publish writes go through
|
||||
`get_context().override`; rerouting a writer means flipping **all its readers to the
|
||||
bag in the same commit** (no transitional dual-write).
|
||||
4. **Legacy-accessor ratchet** (`test_legacy_global_ratchet.py`): `get_global_server_args`
|
||||
call sites must not grow — new code uses `runtime_context.get_server_args()` (and
|
||||
business decisions should read the bags).
|
||||
5. **Module-state ratchet** (`test_module_state_ratchet.py`): `global` statements in the
|
||||
flag-owning layers are pinned by name. A new module-level runtime global belongs on a
|
||||
flags group / resources slot instead; migrating a pinned survivor must shrink the pin.
|
||||
6. **Namespace coverage** (`test_server_args_namespaces.py`,
|
||||
`test_runtime_context_config_bags.py`): every `ServerArgs` field carries `NS(...)`
|
||||
metadata and the projected bags must cover the fields exactly (two-way).
|
||||
|
||||
Never module-skip a test "until the migration settles" — seed the context instead
|
||||
(the deferral ratchet that once pinned this is retired; the rule stands).
|
||||
|
||||
## Hard-won pitfalls (check these before/while refactoring)
|
||||
|
||||
- **Moving code drops first-line guards**: early returns (`if self.is_draft_worker: return`)
|
||||
are the easiest thing to lose when relocating a method body. Draft workers share the
|
||||
target's `server_args` object — a draft-side write poisons the target.
|
||||
are the easiest thing to lose when relocating a method body. Only drafts built through
|
||||
`build_draft_tp_worker()` get private bags (a preserved publish of the rewritten copy);
|
||||
drafts constructed directly with `is_draft_worker=True` skip publish and **share the
|
||||
target's bags** — a draft-side write there poisons the target.
|
||||
- **Registry-completeness timing**: a gate that consults an extensible list is only correct
|
||||
after the registrars ran (platform `init_backend()` at module import). See "load-time vs
|
||||
resolution-time".
|
||||
@@ -168,9 +254,11 @@ ONE thread — do not design for TBO threads that don't exist.
|
||||
recompile limit; **class/instance attributes are the only compile-friendly
|
||||
form** (attribute-source ints get automatic-dynamic after the first size
|
||||
change). Bools (≤2 values) are tolerable in any form — see
|
||||
`ForwardFlags._GRAPH_VISIBLE`. Before moving such state, prove its readers
|
||||
sit outside compile coverage; a piecewise-prefill boot of a small model is
|
||||
the fast check (recompile storms show as `torch._dynamo hit
|
||||
`ForwardFlags._GRAPH_VISIBLE`. Config-bag leaves are real instance attributes for
|
||||
exactly this reason, and `ParallelContext.__getattr__` must stay free of
|
||||
`object.__getattribute__` (dynamo graph-breaks on it). Before moving such state,
|
||||
prove its readers sit outside compile coverage; a piecewise-prefill boot of a small
|
||||
model is the fast check (recompile storms show as `torch._dynamo hit
|
||||
config.recompile_limit` during the compile pass).
|
||||
- **Engine-booting e2e tests are the only coverage for launcher-path code**; a child crash
|
||||
kills the process tree and pytest dies silently — run with `PYTHONUNBUFFERED=1` and read
|
||||
@@ -184,12 +272,13 @@ ONE thread — do not design for TBO threads that don't exist.
|
||||
|
||||
## Where to read the code
|
||||
|
||||
Key source files: `python/sglang/srt/runtime_context.py` (the container and every tier),
|
||||
Key source files: `python/sglang/srt/runtime_context.py` (the container, every tier,
|
||||
`publish`, `_ConfigBag`, `preserve_config`, `override_server_args`),
|
||||
`python/sglang/srt/arg_groups/overrides.py` (override registry, passes,
|
||||
`declare_load_time_override`, `resolved_view`), `python/sglang/srt/server_args.py`
|
||||
(`override`, `__setattr__`, `materialize_declarations` call,
|
||||
`_handle_model_capability_adjustments`), and the guardrail tests under
|
||||
`declare_load_time_override`), `python/sglang/srt/server_args.py` (`NS` metadata,
|
||||
`Arg(..., resolvable=True)`, `__setattr__` strict guard), and the guardrail tests under
|
||||
`test/registered/unit/` (`test_server_args_mutation_ratchet.py`,
|
||||
`test_legacy_global_ratchet.py`, `test_module_state_ratchet.py`,
|
||||
`test_runtime_context.py` — the last one doubles as executable documentation of
|
||||
every tier's semantics).
|
||||
`test_server_args_writer_ratchet.py`, `test_legacy_global_ratchet.py`,
|
||||
`test_module_state_ratchet.py`, `test_server_args_namespaces.py`,
|
||||
`test_runtime_context.py` — the last one doubles
|
||||
as executable documentation of every tier's semantics).
|
||||
|
||||
Reference in New Issue
Block a user