minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

config: retire the multi-engine accommodation in the runtime context (#35022)

Browse Source
This commit is contained in:
Cheng Wan
2026-08-17 16:15:33 -07:00
committed by GitHub
parent bc312d185d
commit 2b278b4ac4
8 changed files with 172 additions and 283 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/skills/sglang-runtime-context/SKILL.md
+25 -25
View File
@@ -10,7 +10,7 @@ One container owns process-static runtime state: `sglang.srt.runtime_context.Run
| Tier | Accessor | Holds | Lifecycle |
|------|----------|-------|-----------|
| raw config seed | `get_server_args()` | the published `ServerArgs` — the startup record, for debugging, dumps and provenance. **Business code does not read fields off it**: the read ratchet pins that at zero, and "Reading config: the seed is off limits" below says what to read instead, which forms the ratchet sees, and what is outside it by construction (a runtime-computed name; a whole-object hand-off) | published at process entry; re-publish is **last-publish-wins** (in-process tokenizer build, multi-Engine) and re-projects the bags; read-only |
| raw config seed | `get_server_args()` | the published `ServerArgs` — the startup record, for debugging, dumps and provenance. **Business code does not read fields off it**: the read ratchet pins that at zero, and "Reading config: the seed is off limits" below says what to read instead, which forms the ratchet sees, and what is outside it by construction (a runtime-computed name; a whole-object hand-off) | published at process entry; re-publish is **last-publish-wins** (the tokenizer publish in the launcher process; sequential engine rebuild in one process, e.g. unit tests) and re-projects the bags; read-only |
| resolved config | `get_exec()` `get_memory()` `get_schedule()` `get_model()` `get_spec()` `get_serving()` `get_observability()` `get_disagg()` `get_lora()` `get_mm()` `get_device()` | namespace **config bags** — the single source of truth for resolved config; leaves are real attributes (dynamo-traceable) | projected from `server_args` at `publish`; mutated only via `get_context().override` |
| runtime flags | `get_flags()` | state that is *not* a pure function of config: `capture` (cuda-graph lifecycle), `moe` (ACTIVE backends, swappable), `dp` (DP-attention runtime flags) | materialized at subsystem init; groups offer `override()` for tests |
| resources | `get_resources()`, `get_stream(name)`, `get_buffer(name, factory)` | process-level handles: graph pools, EPLB state, EP dispatcher state, named side streams, workspace buffers | lazy; cleared by `reset_context()` |
@@ -79,11 +79,6 @@ re-projects its own bags, so a parent-side override is lost. Values that feed
construction before any bag exists (group init reads `server_args.tp_size`) have no
bag to override at all.
- **Nested publishes**: `get_context().preserve_config()` snapshots the enclosing
lifecycle (including its post-publish overrides) and reinstates it on exit. No
production caller is left — the draft build was the last one, and per-runner
values are constructor arguments now — so it survives for tests and for a future
construction step that genuinely has to publish a private copy.
### Reads that legitimately stay on a `ServerArgs` instance
@@ -110,28 +105,31 @@ bag to override at all.
the scope. When there is a runner in hand, read its
stamp; that is a different rule from "read the instance".
- **Per-instance boundaries** — the tokenizer-manager family, everything under
`entrypoints/`, and the tokenizer-process multimodal processors read
`self.server_args`: several `Engine`s can share one process, and the process-global
bags are last-publish-wins across engines. `base_gpu_id` also differs per engine,
so no process-global value can stand in for it —
`BaseMultimodalProcessor._fast_image_processor_device` is the shape to copy. (The
encode-server DP workers used to specialize a config copy for the same reason;
their device now travels as `MMEncoder(gpu_id=...)`.)
`entrypoints/`, and the tokenizer-process multimodal processors still read
`self.server_args` today. The old justification ("several `Engine`s can share
one process, bags are last-publish-wins across them") is **retracted** — owner
ruling (2026-08-15): a process holds at most one live config at a time
(concurrent multi-Engine is unsupported; sequential rebuild stays legal, unit
tests rely on it). These reads are scheduled to become bag reads in the
bag-read series; treat them as pinned debt, not as a boundary to imitate. What
genuinely stays per-instance is what differs per *worker* within one engine:
`base_gpu_id` travels as a constructor argument (`MMEncoder(gpu_id=...)`;
`BaseMultimodalProcessor._fast_image_processor_device` is the shape to copy).
- **Whole-object passes** (`f(server_args)` handing the instance along) keep the
supplied-instance contract; don't rewrite the parameter reads unless the
field is runtime-mutated (see the elastic-EP `ep_size` case in
`eplb/expert_location.py`) — **or the field is one that resolution fills in
and the callee runs in a process that has published.** That second case is
step-12 debt, not a style question: the record is destined to carry the
pinned debt, not a style question: the record is destined to carry the
user's raw input, so `server_args.page_size` inside a runner-owned
constructor will read the raw pre-resolution value instead of the effective
one. Debt means a decision, not automatically a bag read: pick where the
value should come from — usually the `get_*()` bag, sometimes a runner stamp
or a constructor argument (the per-mode attention pair and the encode-server
`gpu_id` above are dispositions of exactly this debt). And the per-instance
boundaries above stay exempt from this unless-clause: a multi-Engine site
must not become a process-global bag read even for a resolution-filled
field. `test_supplied_instance_exposure_ratchet.py`
`gpu_id` above are dispositions of exactly this debt). The per-instance
boundaries above are **not** exempt from this unless-clause (the multi-Engine
exemption is retracted); each one gets its own disposition.
`test_supplied_instance_exposure_ratchet.py`
pins the remaining set — three spellings of the read: `server_args.field`,
literal-name `getattr(server_args, "field", default)`, and the parked form
(`self.x = server_args` in a method that takes the parameter, read as
@@ -224,13 +222,15 @@ this).
`self.server_args.field` is still right for handed per-instance config (see
"Reads that legitimately stay on a ServerArgs instance" above for the full set —
per-instance boundaries and whole-object passes; there are no per-runner config
copies to read any more). The allow-list is the
tokenizer-manager family, `entrypoints/`, the tokenizer-process multimodal
processors, `GrammarManager`, `MMEncoder` — but not for one single reason:
copies to read any more). The allow-list is `GrammarManager` and `MMEncoder`;
the tokenizer-manager family, `entrypoints/`, and the tokenizer-process
multimodal processors sit beside it only as pinned debt — not for one single
reason:
- the tokenizer-manager family and `entrypoints/` are the multi-Engine case
proper: several of them can live in one process, so a bag read would answer
from whichever Engine published last;
- the tokenizer-manager family and `entrypoints/` are **pinned debt awaiting
conversion to bag reads** (the old multi-Engine justification is retracted —
one process, one live config); the reads still work today because the
instance carries resolved values;
- `GrammarManager` is a handed instance — it is constructed with the config its
owner hands it and never assumes a published namespace;
- `MMEncoder` publishes the very instance it is handed (`publish(server_args,
@@ -474,7 +474,7 @@ Never module-skip a test "until the migration settles" — seed the context inst
## Where to read the code
Key source files: `python/sglang/srt/runtime_context.py` (the container, every tier,
`publish`, `_ConfigBag`, `preserve_config`, `override_server_args`),
`publish`, `_ConfigBag`, `override_server_args`),
`python/sglang/srt/arg_groups/overrides.py` (override registry, passes,
`declare_late_resolution`), `python/sglang/srt/server_args.py` (`NS` metadata,
`Arg(..., resolvable=True)`, `__setattr__` strict guard), and the guardrail tests under
python/sglang/srt/entrypoints/engine.py
+4 -5
View File
@@ -278,8 +278,8 @@ class Engine(EngineScoreMixin, EngineBase):
self.template_manager = template_manager
self._scheduler_init_result = scheduler_init_result
# Engine-spawned weight cache daemons owned by *this* instance (empty
# unless --weight-cache-mode daemon). Kept per-instance so two Engines
# in one process each reap only their own daemons in shutdown().
# unless --weight-cache-mode daemon), so shutdown() reaps exactly what
# this Engine spawned.
self._weight_cache_daemon_procs = weight_cache_daemon_procs
if tokenizer_manager is not None:
tokenizer_manager._subprocess_watchdog = subprocess_watchdog
@@ -1109,9 +1109,8 @@ class Engine(EngineScoreMixin, EngineBase):
):
resolve_auto_parsers(server_args)
# Launch daemons (daemon mode only). Handles are threaded back to the
# owning Engine instance (not a class attr) so two Engines in one process
# don't clobber each other's daemon list.
# Launch daemons (daemon mode only). The handles travel back to the
# Engine that spawned them; shutdown() reaps from there.
weight_cache_daemon_procs: List = []
if server_args.weight_cache_mode == "daemon":
weight_cache_daemon_procs = cls._launch_weight_cache_daemons(server_args)
python/sglang/srt/managers/tokenizer_manager.py
+5 -5
View File
@@ -2057,11 +2057,11 @@ class TokenizerManager(TokenizerControlMixin, TokenizerManagerScoreMixin):
def record_config_updates(self, source: str, **fields) -> None:
"""Record a control-plane config change for this engine.
Per-engine state: several ``Engine``s can share one tokenizer process.
The readback endpoints overlay these onto the startup config. The
process-global sibling is ``RuntimeContext.override`` /
``resolved_server_args_dict``, which writes the config bags every
process shares.
These are post-startup facts the config bags do not model (weight
version, model path, the tokenizer's HiCache mirror); the readback
endpoints overlay them onto the startup config. The process-global
sibling is ``RuntimeContext.override`` / ``resolved_server_args_dict``,
which writes the config bags.
"""
unknown = sorted(f for f in fields if f not in _SERVER_ARGS_FIELDS)
if unknown:
python/sglang/srt/runtime_context.py
+11 -71
View File
@@ -722,34 +722,6 @@ def _build_config_bags(server_args: Any) -> dict:
return tops
def _snapshot_bag_values(bags: dict | None) -> dict | None:
"""Per-leaf value snapshot of a config-bag tree (bags are mutated in
place by ``override``, so reference snapshots alias live state)."""
if bags is None:
return None
snap: dict = {}
def walk(prefix: str, bag) -> None:
snap[prefix] = dict(object.__getattribute__(bag, "_fields"))
for name, sub in object.__getattribute__(bag, "_subs").items():
walk(f"{prefix}.{name}", sub)
for name, bag in bags.items():
walk(name, bag)
return snap
def _restore_bag_values(bags: dict, snap: dict) -> None:
def walk(prefix: str, bag) -> None:
for key, value in snap[prefix].items():
bag._set(key, value)
for name, sub in object.__getattribute__(bag, "_subs").items():
walk(f"{prefix}.{name}", sub)
for name, bag in bags.items():
walk(name, bag)
class RuntimeContext:
"""Container for the structured runtime accessors; exposes ``parallel``,
``server_args``, the resolved config namespace bags, ``flags``,
@@ -953,12 +925,12 @@ class RuntimeContext:
``ServerArgs`` field names, so overlaying them onto the top level of
either base is exact.
This covers the process-global bags only. Per-engine control-plane
changes (weight version, model path, the tokenizer's HiCache mirror)
live on the tokenizer manager — several ``Engine``s can share one
process — and ``TokenizerManager.resolved_config_dict`` overlays those
for the top-level ``/server_info`` body. The two are separate logs, not
one merged dict.
This covers the process-global bags only. Control-plane facts the bags
do not model (weight version, model path, the tokenizer's HiCache
mirror) live on the tokenizer manager, and
``TokenizerManager.resolved_config_dict`` overlays those for the
top-level ``/server_info`` body. The two are separate logs, not one
merged dict.
"""
d = dict(vars(self.server_args)) if base is None else dict(base)
for _source, fields in self._overrides_log:
@@ -987,33 +959,6 @@ class RuntimeContext:
"""
return _ServerArgsOverride(self, fields)
@contextmanager
def preserve_config(self):
"""Snapshot the full config lifecycle and reinstate it verbatim on exit.
For nested construction steps that publish a private ``ServerArgs``
copy (e.g. a draft-worker build) and must leave the enclosing
lifecycle — including its post-publish overrides — untouched.
"""
prev_server_args = self._server_args
prev_bags = self._config_bags
prev_bag_values = _snapshot_bag_values(prev_bags)
prev_overrides_log = list(self._overrides_log)
prev_publish_role = self._publish_role
prev_parallel_config = self.parallel._config
prev_capture = self.flags.capture.enable_torch_compile
try:
yield
finally:
self._server_args = prev_server_args
self._config_bags = prev_bags
if prev_bags is not None:
_restore_bag_values(prev_bags, prev_bag_values)
self._overrides_log = prev_overrides_log
self._publish_role = prev_publish_role
self.parallel._config = prev_parallel_config
self.flags.capture.enable_torch_compile = prev_capture
class _ServerArgsOverride:
"""Scoped config override (see ``RuntimeContext.override_server_args``).
@@ -1198,7 +1143,6 @@ def get_observability() -> _ConfigBag:
ROLE_NAMESPACE_SETS: dict[str, frozenset[str] | None] = {
# Reads (almost) everything by design — the model-executing process.
"scheduler": None,
"launcher": None,
"test": None,
# Audited (record-mode smokes, plain + DP-attention): the DP controller
# reads only the elastic-EP gate; its module's static read set agrees.
@@ -1317,21 +1261,17 @@ def publish(server_args, *, role: str, hf_config: Any = None) -> RuntimeContext:
Records the process ``role`` (``tokenizer`` / ``scheduler`` /
``dp_controller`` / ``encoder`` / ``expert_backup`` /
``weight_cache_daemon`` / ``launcher`` / ``test``) and
``weight_cache_daemon`` / ``test``) and
projects the config bags. Draft workers skip publish (they must not clobber
the target). ``role`` is provenance, and — when ``SGLANG_ROLE_NAMESPACES``
is ``enforce`` — the key into ``ROLE_NAMESPACE_SETS`` for fail-closed
namespace-read enforcement (``record`` audits the reads instead).
``hf_config`` is accepted for forward-compat and currently unused.
Normally one call per process, but re-publish is allowed and is
**last-publish-wins** (bags re-projected, provenance reset, role
overwritten). Two sanctioned multi-publish shapes exist: the in-process
Engine builds its ``TokenizerManager`` inside the launcher process (the
process ends up with the tokenizer publish), and multiple Engines in one
process publish in sequence — which is exactly why per-instance managers
must read ``self.server_args`` for anything engine-specific rather than
the process-global bags.
A process holds at most one live config: the bags always describe the
engine running now. Re-publish is allowed and is **last-publish-wins**
(bags re-projected, provenance reset, role overwritten), which is what
lets one process rebuild an engine after shutting the previous one down.
"""
if _ROLE_NS_MODE == "enforce" and role not in ROLE_NAMESPACE_SETS:
# Fail closed at publish time, not at the first stray read.
test/registered/prefill_only/test_multi_item_scoring.py
+121 -112
View File
@@ -37,6 +37,20 @@ TEST_CLASSIFICATION_BASE_MODEL = os.environ.get(
_CLS_NUM_LABELS = AutoConfig.from_pretrained(TEST_CLASSIFICATION_BASE_MODEL).num_labels
def _collect_scores(engine_kwargs, calls):
"""Boot one engine, run ``calls`` through score(), shut it down.
A process holds one live config, so the reference engine must be gone
before the engine under test boots.
"""
engine = Engine(**engine_kwargs)
try:
return [engine.score(**call).scores for call in calls]
finally:
engine.shutdown()
torch.cuda.empty_cache()
class TestMISServerArgsValidation(unittest.TestCase):
"""Test ServerArgs defaults for MIS mode."""
@@ -50,8 +64,24 @@ class TestMISServerArgsValidation(unittest.TestCase):
class TestMultiItemScoringOptimization(CustomTestCase):
"""Test the Multi-Item Scoring (MIS) optimization with generation models."""
CONSISTENCY_CALL = dict(
query="Is this a fact?\n",
items=[" The sun rises in the east"],
label_token_ids=[9454, 2753],
apply_softmax=True,
)
@classmethod
def setUpClass(cls):
(cls.non_mis_consistency_scores,) = _collect_scores(
dict(
model_path=TEST_MODEL_NAME,
disable_radix_cache=True,
chunked_prefill_size=-1,
mem_fraction_static=0.15,
),
[cls.CONSISTENCY_CALL],
)
cls.engine = Engine(
model_path=TEST_MODEL_NAME,
disable_radix_cache=True,
@@ -60,19 +90,11 @@ class TestMultiItemScoringOptimization(CustomTestCase):
attention_backend="flashinfer",
mem_fraction_static=0.15,
)
cls.non_mis_engine = Engine(
model_path=TEST_MODEL_NAME,
disable_radix_cache=True,
chunked_prefill_size=-1,
mem_fraction_static=0.15,
)
@classmethod
def tearDownClass(cls):
if cls.engine is not None:
cls.engine.shutdown()
if cls.non_mis_engine is not None:
cls.non_mis_engine.shutdown()
torch.cuda.empty_cache()
def test_mis_basic(self):
@@ -98,23 +120,8 @@ class TestMultiItemScoringOptimization(CustomTestCase):
def test_mis_consistency_with_single_item(self):
"""MIS with one item should match non-MIS scoring closely."""
query = "Is this a fact?\n"
items = [" The sun rises in the east"]
label_token_ids = [9454, 2753]
mis_scores = self.engine.score(
query=query,
items=items,
label_token_ids=label_token_ids,
apply_softmax=True,
).scores
non_mis_scores = self.non_mis_engine.score(
query=query,
items=items,
label_token_ids=label_token_ids,
apply_softmax=True,
).scores
mis_scores = self.engine.score(**self.CONSISTENCY_CALL).scores
non_mis_scores = self.non_mis_consistency_scores
self.assertEqual(len(mis_scores), 1)
self.assertEqual(len(non_mis_scores), 1)
@@ -151,14 +158,29 @@ class TestMultiItemScoringClassification(CustomTestCase):
Pre-trained Qwen3ForSequenceClassification, so the head weights are
deterministic. One class rather than four because the CI harness demands an
idle GPU at every setUpClass -- splitting these means re-booting the same
two engines instead of sharing them. score() is stateless and the radix
cache is off, so sharing is safe.
engines instead of sharing them. score() is stateless and the radix cache
is off, so sharing is safe.
"""
NUM_LABELS = _CLS_NUM_LABELS
FALLBACK_CALL = dict(query="Test:", items=["A", "B"], apply_softmax=True)
SINGLE_VS_MIS_CALL = dict(
query="Rate this option:",
items=[" Option A", " Option B", " Option C"],
apply_softmax=True,
)
@classmethod
def setUpClass(cls):
cls.non_mis_fallback_scores, cls.non_mis_single_scores = _collect_scores(
dict(
model_path=TEST_CLASSIFICATION_BASE_MODEL,
disable_radix_cache=True,
mem_fraction_static=0.15,
),
[cls.FALLBACK_CALL, cls.SINGLE_VS_MIS_CALL],
)
cls.engine = Engine(
model_path=TEST_CLASSIFICATION_BASE_MODEL,
disable_radix_cache=True,
@@ -167,17 +189,11 @@ class TestMultiItemScoringClassification(CustomTestCase):
attention_backend="flashinfer",
mem_fraction_static=0.15,
)
cls.non_mis_engine = Engine(
model_path=TEST_CLASSIFICATION_BASE_MODEL,
disable_radix_cache=True,
mem_fraction_static=0.15,
)
@classmethod
def tearDownClass(cls):
for engine in (cls.engine, cls.non_mis_engine):
if engine is not None:
engine.shutdown()
if cls.engine is not None:
cls.engine.shutdown()
torch.cuda.empty_cache()
def test_classification_mis_basic(self):
@@ -215,9 +231,7 @@ class TestMultiItemScoringClassification(CustomTestCase):
def test_classification_non_mis_fallback(self):
"""Classification model works correctly without --enable-mis."""
scores = self.non_mis_engine.score(
query="Test:", items=["A", "B"], apply_softmax=True
).scores
scores = self.non_mis_fallback_scores
self.assertEqual(len(scores), 2)
for score_list in scores:
@@ -395,14 +409,8 @@ class TestMultiItemScoringClassification(CustomTestCase):
perturbs hidden states; after softmax the scores should still land
within places=1 (+-0.05).
"""
query = "Rate this option:"
items = [" Option A", " Option B", " Option C"]
non_mis_scores = self.non_mis_engine.score(
query=query, items=items, apply_softmax=True
).scores
mis_scores = self.engine.score(
query=query, items=items, apply_softmax=True
).scores
non_mis_scores = self.non_mis_single_scores
mis_scores = self.engine.score(**self.SINGLE_VS_MIS_CALL).scores
self.assertEqual(len(mis_scores), len(non_mis_scores))
for i, (ms, ns) in enumerate(zip(mis_scores, non_mis_scores)):
@@ -420,48 +428,73 @@ class TestMultiItemScoringParity(CustomTestCase):
"""Test that MIS produces the same results as single-item scoring."""
@classmethod
def setUpClass(cls):
cls.engine_single = Engine(
model_path=TEST_MODEL_NAME,
disable_radix_cache=True,
log_level="error",
mem_fraction_static=0.15,
)
cls.engine_mis = Engine(
model_path=TEST_MODEL_NAME,
disable_radix_cache=True,
chunked_prefill_size=-1,
log_level="error",
enable_mis=True,
attention_backend="flashinfer",
mem_fraction_static=0.15,
)
def _cases(cls):
"""The scoring calls both engines run, keyed by the test that reads them."""
tokenizer = AutoTokenizer.from_pretrained(TEST_MODEL_NAME)
def label_ids(labels):
return [tokenizer.encode(lb, add_special_tokens=False)[0] for lb in labels]
return {
"basic": dict(
query="Rate this option:",
items=[" Option A", " Option B", " Option C"],
label_token_ids=label_ids([" good", " bad"]),
apply_softmax=True,
),
"tokenized": dict(
query=tokenizer.encode("Rate this option:", add_special_tokens=False),
items=[
tokenizer.encode(item, add_special_tokens=False)
for item in [" Option X", " Option Y"]
],
label_token_ids=label_ids([" good", " bad"]),
apply_softmax=True,
),
"no_softmax": dict(
query="The weather today is",
items=[" sunny", " cloudy", " rainy"],
label_token_ids=label_ids([" nice", " bad"]),
apply_softmax=False,
),
"many_items": dict(
query="Rate this option from 1 to 5:",
items=[f" Option {i}" for i in range(10)],
label_token_ids=label_ids([" 1", " 2", " 3", " 4", " 5"]),
apply_softmax=True,
),
}
@classmethod
def tearDownClass(cls):
if cls.engine_single is not None:
cls.engine_single.shutdown()
if cls.engine_mis is not None:
cls.engine_mis.shutdown()
torch.cuda.empty_cache()
def setUpClass(cls):
cases = cls._cases()
names, calls = list(cases), list(cases.values())
base = dict(
model_path=TEST_MODEL_NAME,
disable_radix_cache=True,
log_level="error",
mem_fraction_static=0.15,
)
cls.single_scores = dict(zip(names, _collect_scores(base, calls)))
cls.mis_scores = dict(
zip(
names,
_collect_scores(
dict(
base,
chunked_prefill_size=-1,
enable_mis=True,
attention_backend="flashinfer",
),
calls,
),
)
)
def _compare_scores(
self, query, items, label_token_ids=None, apply_softmax=True, test_name=""
):
def _compare_scores(self, test_name):
"""Compare MIS vs single-item scoring results."""
single_scores = self.engine_single.score(
query=query,
items=items,
label_token_ids=label_token_ids,
apply_softmax=apply_softmax,
).scores
mis_scores = self.engine_mis.score(
query=query,
items=items,
label_token_ids=label_token_ids,
apply_softmax=apply_softmax,
).scores
single_scores = self.single_scores[test_name]
mis_scores = self.mis_scores[test_name]
self.assertEqual(
len(mis_scores), len(single_scores), f"{test_name}: count mismatch"
@@ -477,40 +510,16 @@ class TestMultiItemScoringParity(CustomTestCase):
)
def test_parity_basic(self):
tokenizer = AutoTokenizer.from_pretrained(TEST_MODEL_NAME)
query = "Rate this option:"
items = [" Option A", " Option B", " Option C"]
labels = [" good", " bad"]
label_ids = [tokenizer.encode(lb, add_special_tokens=False)[0] for lb in labels]
self._compare_scores(query, items, label_ids, test_name="basic")
self._compare_scores("basic")
def test_parity_tokenized_inputs(self):
tokenizer = AutoTokenizer.from_pretrained(TEST_MODEL_NAME)
query = "Rate this option:"
items = [" Option X", " Option Y"]
labels = [" good", " bad"]
query_ids = tokenizer.encode(query, add_special_tokens=False)
items_ids = [tokenizer.encode(i, add_special_tokens=False) for i in items]
label_ids = [tokenizer.encode(lb, add_special_tokens=False)[0] for lb in labels]
self._compare_scores(query_ids, items_ids, label_ids, test_name="tokenized")
self._compare_scores("tokenized")
def test_parity_without_softmax(self):
tokenizer = AutoTokenizer.from_pretrained(TEST_MODEL_NAME)
query = "The weather today is"
items = [" sunny", " cloudy", " rainy"]
labels = [" nice", " bad"]
label_ids = [tokenizer.encode(lb, add_special_tokens=False)[0] for lb in labels]
self._compare_scores(
query, items, label_ids, apply_softmax=False, test_name="no_softmax"
)
self._compare_scores("no_softmax")
def test_parity_many_items(self):
tokenizer = AutoTokenizer.from_pretrained(TEST_MODEL_NAME)
query = "Rate this option from 1 to 5:"
items = [f" Option {i}" for i in range(10)]
labels = [" 1", " 2", " 3", " 4", " 5"]
label_ids = [tokenizer.encode(lb, add_special_tokens=False)[0] for lb in labels]
self._compare_scores(query, items, label_ids, test_name="many_items")
self._compare_scores("many_items")
if __name__ == "__main__":
test/registered/unit/managers/test_tokenizer_config_updates.py
+2 -7
View File
@@ -2,8 +2,8 @@
Regression: runtime updates (weight version, model path, HiCache attach) were
written onto the manager's ServerArgs instance so that the readback endpoints
would show them. They are per-engine — several Engines can share a tokenizer
process — so they live on the manager and the endpoints overlay them.
would show them. The record stays pristine; the updates live in a separate log
that the endpoints overlay on top of it.
"""
import re
@@ -39,11 +39,6 @@ class TestTokenizerConfigUpdates(CustomTestCase):
manager.record_config_updates("test", weight_version="v2")
self.assertEqual(manager.server_args.weight_version, "v1")
def test_two_engines_keep_their_own_updates(self):
first, second = _manager(weight_version="v1"), _manager(weight_version="v1")
first.record_config_updates("test", weight_version="v2")
self.assertEqual(second.config_value("weight_version"), "v1")
def test_the_readback_dict_carries_the_updates(self):
manager = _manager(hicache_storage_backend=None)
manager.record_config_updates(
test/registered/unit/multimodal/test_processor_device_selection.py
+4 -4
View File
@@ -1,9 +1,9 @@
"""The fast-image-processor device comes from the processor's own ServerArgs.
Regression: the device decision read the published global ServerArgs, which is
last-publish-wins. Two engines in one tokenizer process then shared whichever
config published last, so one engine's images were preprocessed on the other
engine's GPU.
Regression: the device decision read the published global ServerArgs, so every
processor answered with one process-wide device. The encode-server DP workers
each drive their own GPU, which no process-global value can express — the
device has to come from what the worker was handed.
"""
import unittest
test/registered/unit/test_runtime_context_override.py
-54
View File
@@ -109,60 +109,6 @@ class TestContextOverride(CustomTestCase):
with self.assertRaises(AttributeError):
sa.page_size = 999
def test_preserve_config_keeps_post_publish_overrides(self):
# A nested build (e.g. a draft worker) publishes its own private copy;
# on exit the target's resolved bags — including post-publish
# overrides — must be reinstated verbatim, not re-projected from the
# pristine record (which would silently drop the overrides).
target = self._publish()
rc.get_context().override(
"ModelRunner.configure_kv_cache_dtype", kv_cache_dtype="fp8_e4m3"
)
draft = ServerArgs(model_path="dummy", kv_cache_dtype="bf16")
with rc.get_context().preserve_config():
rc.get_context().set_server_args(draft)
# Inside the scope the draft's bags are live...
self.assertEqual(rc.get_model().kv_cache_dtype, "bf16")
# ...and its own post-publish overrides work as usual.
rc.get_context().override("draft-load", kv_cache_dtype="fp8_e5m2")
self.assertEqual(rc.get_model().kv_cache_dtype, "fp8_e5m2")
# Target slot, bags, and provenance restored verbatim.
self.assertIs(rc.get_context().server_args, target)
self.assertEqual(rc.get_model().kv_cache_dtype, "fp8_e4m3")
self.assertEqual(
rc.get_context().overrides_log(),
[
(
"ModelRunner.configure_kv_cache_dtype",
{"kv_cache_dtype": "fp8_e4m3"},
)
],
)
def test_preserve_config_restores_in_scope_override_without_republish(self):
# An override inside the scope (no republish) mutates the live bags
# and provenance log in place; the scope must restore entry VALUES,
# not just reassign the aliased objects.
self._publish()
rc.get_context().override("srcA", page_size=16)
with rc.get_context().preserve_config():
rc.get_context().override("in-scope", page_size=64)
self.assertEqual(rc.get_schedule().page_size, 64)
self.assertEqual(rc.get_schedule().page_size, 16)
self.assertEqual(
rc.get_context().overrides_log(), [("srcA", {"page_size": 16})]
)
def test_preserve_config_restores_on_exception(self):
target = self._publish()
rc.get_context().override("srcA", page_size=16)
with self.assertRaises(RuntimeError):
with rc.get_context().preserve_config():
rc.get_context().set_server_args(ServerArgs(model_path="dummy"))
raise RuntimeError("nested build failed")
self.assertIs(rc.get_context().server_args, target)
self.assertEqual(rc.get_schedule().page_size, 16)
def test_publish_records_role(self):
rc.publish(ServerArgs(model_path="dummy"), role="scheduler")
self.assertEqual(rc.publish_role(), "scheduler")