diff --git a/.claude/skills/sglang-runtime-context/SKILL.md b/.claude/skills/sglang-runtime-context/SKILL.md index 0215c4110..bb5f5f96b 100644 --- a/.claude/skills/sglang-runtime-context/SKILL.md +++ b/.claude/skills/sglang-runtime-context/SKILL.md @@ -10,7 +10,7 @@ One container owns process-static runtime state: `sglang.srt.runtime_context.Run | Tier | Accessor | Holds | Lifecycle | |------|----------|-------|-----------| -| raw config seed | `get_server_args()` | the published `ServerArgs` — the startup record, for debugging, dumps and provenance. **Business code does not read fields off it**: the read ratchet pins that at zero, and "Reading config: the seed is off limits" below says what to read instead, which forms the ratchet sees, and what is outside it by construction (a runtime-computed name; a whole-object hand-off) | published at process entry; re-publish is **last-publish-wins** (in-process tokenizer build, multi-Engine) and re-projects the bags; read-only | +| raw config seed | `get_server_args()` | the published `ServerArgs` — the startup record, for debugging, dumps and provenance. **Business code does not read fields off it**: the read ratchet pins that at zero, and "Reading config: the seed is off limits" below says what to read instead, which forms the ratchet sees, and what is outside it by construction (a runtime-computed name; a whole-object hand-off) | published at process entry; re-publish is **last-publish-wins** (the tokenizer publish in the launcher process; sequential engine rebuild in one process, e.g. unit tests) and re-projects the bags; read-only | | resolved config | `get_exec()` `get_memory()` `get_schedule()` `get_model()` `get_spec()` `get_serving()` `get_observability()` `get_disagg()` `get_lora()` `get_mm()` `get_device()` | namespace **config bags** — the single source of truth for resolved config; leaves are real attributes (dynamo-traceable) | projected from `server_args` at `publish`; mutated only via `get_context().override` | | runtime flags | `get_flags()` | state that is *not* a pure function of config: `capture` (cuda-graph lifecycle), `moe` (ACTIVE backends, swappable), `dp` (DP-attention runtime flags) | materialized at subsystem init; groups offer `override()` for tests | | resources | `get_resources()`, `get_stream(name)`, `get_buffer(name, factory)` | process-level handles: graph pools, EPLB state, EP dispatcher state, named side streams, workspace buffers | lazy; cleared by `reset_context()` | @@ -79,11 +79,6 @@ re-projects its own bags, so a parent-side override is lost. Values that feed construction before any bag exists (group init reads `server_args.tp_size`) have no bag to override at all. -- **Nested publishes**: `get_context().preserve_config()` snapshots the enclosing - lifecycle (including its post-publish overrides) and reinstates it on exit. No - production caller is left — the draft build was the last one, and per-runner - values are constructor arguments now — so it survives for tests and for a future - construction step that genuinely has to publish a private copy. ### Reads that legitimately stay on a `ServerArgs` instance @@ -110,28 +105,31 @@ bag to override at all. the scope. When there is a runner in hand, read its stamp; that is a different rule from "read the instance". - **Per-instance boundaries** — the tokenizer-manager family, everything under - `entrypoints/`, and the tokenizer-process multimodal processors read - `self.server_args`: several `Engine`s can share one process, and the process-global - bags are last-publish-wins across engines. `base_gpu_id` also differs per engine, - so no process-global value can stand in for it — - `BaseMultimodalProcessor._fast_image_processor_device` is the shape to copy. (The - encode-server DP workers used to specialize a config copy for the same reason; - their device now travels as `MMEncoder(gpu_id=...)`.) + `entrypoints/`, and the tokenizer-process multimodal processors still read + `self.server_args` today. The old justification ("several `Engine`s can share + one process, bags are last-publish-wins across them") is **retracted** — owner + ruling (2026-08-15): a process holds at most one live config at a time + (concurrent multi-Engine is unsupported; sequential rebuild stays legal, unit + tests rely on it). These reads are scheduled to become bag reads in the + bag-read series; treat them as pinned debt, not as a boundary to imitate. What + genuinely stays per-instance is what differs per *worker* within one engine: + `base_gpu_id` travels as a constructor argument (`MMEncoder(gpu_id=...)`; + `BaseMultimodalProcessor._fast_image_processor_device` is the shape to copy). - **Whole-object passes** (`f(server_args)` handing the instance along) keep the supplied-instance contract; don't rewrite the parameter reads unless the field is runtime-mutated (see the elastic-EP `ep_size` case in `eplb/expert_location.py`) — **or the field is one that resolution fills in and the callee runs in a process that has published.** That second case is - step-12 debt, not a style question: the record is destined to carry the + pinned debt, not a style question: the record is destined to carry the user's raw input, so `server_args.page_size` inside a runner-owned constructor will read the raw pre-resolution value instead of the effective one. Debt means a decision, not automatically a bag read: pick where the value should come from — usually the `get_*()` bag, sometimes a runner stamp or a constructor argument (the per-mode attention pair and the encode-server - `gpu_id` above are dispositions of exactly this debt). And the per-instance - boundaries above stay exempt from this unless-clause: a multi-Engine site - must not become a process-global bag read even for a resolution-filled - field. `test_supplied_instance_exposure_ratchet.py` + `gpu_id` above are dispositions of exactly this debt). The per-instance + boundaries above are **not** exempt from this unless-clause (the multi-Engine + exemption is retracted); each one gets its own disposition. + `test_supplied_instance_exposure_ratchet.py` pins the remaining set — three spellings of the read: `server_args.field`, literal-name `getattr(server_args, "field", default)`, and the parked form (`self.x = server_args` in a method that takes the parameter, read as @@ -224,13 +222,15 @@ this). `self.server_args.field` is still right for handed per-instance config (see "Reads that legitimately stay on a ServerArgs instance" above for the full set — per-instance boundaries and whole-object passes; there are no per-runner config -copies to read any more). The allow-list is the -tokenizer-manager family, `entrypoints/`, the tokenizer-process multimodal -processors, `GrammarManager`, `MMEncoder` — but not for one single reason: +copies to read any more). The allow-list is `GrammarManager` and `MMEncoder`; +the tokenizer-manager family, `entrypoints/`, and the tokenizer-process +multimodal processors sit beside it only as pinned debt — not for one single +reason: -- the tokenizer-manager family and `entrypoints/` are the multi-Engine case - proper: several of them can live in one process, so a bag read would answer - from whichever Engine published last; +- the tokenizer-manager family and `entrypoints/` are **pinned debt awaiting + conversion to bag reads** (the old multi-Engine justification is retracted — + one process, one live config); the reads still work today because the + instance carries resolved values; - `GrammarManager` is a handed instance — it is constructed with the config its owner hands it and never assumes a published namespace; - `MMEncoder` publishes the very instance it is handed (`publish(server_args, @@ -474,7 +474,7 @@ Never module-skip a test "until the migration settles" — seed the context inst ## Where to read the code Key source files: `python/sglang/srt/runtime_context.py` (the container, every tier, -`publish`, `_ConfigBag`, `preserve_config`, `override_server_args`), +`publish`, `_ConfigBag`, `override_server_args`), `python/sglang/srt/arg_groups/overrides.py` (override registry, passes, `declare_late_resolution`), `python/sglang/srt/server_args.py` (`NS` metadata, `Arg(..., resolvable=True)`, `__setattr__` strict guard), and the guardrail tests under diff --git a/python/sglang/srt/entrypoints/engine.py b/python/sglang/srt/entrypoints/engine.py index 4943fc2ac..1b8ec6889 100644 --- a/python/sglang/srt/entrypoints/engine.py +++ b/python/sglang/srt/entrypoints/engine.py @@ -278,8 +278,8 @@ class Engine(EngineScoreMixin, EngineBase): self.template_manager = template_manager self._scheduler_init_result = scheduler_init_result # Engine-spawned weight cache daemons owned by *this* instance (empty - # unless --weight-cache-mode daemon). Kept per-instance so two Engines - # in one process each reap only their own daemons in shutdown(). + # unless --weight-cache-mode daemon), so shutdown() reaps exactly what + # this Engine spawned. self._weight_cache_daemon_procs = weight_cache_daemon_procs if tokenizer_manager is not None: tokenizer_manager._subprocess_watchdog = subprocess_watchdog @@ -1109,9 +1109,8 @@ class Engine(EngineScoreMixin, EngineBase): ): resolve_auto_parsers(server_args) - # Launch daemons (daemon mode only). Handles are threaded back to the - # owning Engine instance (not a class attr) so two Engines in one process - # don't clobber each other's daemon list. + # Launch daemons (daemon mode only). The handles travel back to the + # Engine that spawned them; shutdown() reaps from there. weight_cache_daemon_procs: List = [] if server_args.weight_cache_mode == "daemon": weight_cache_daemon_procs = cls._launch_weight_cache_daemons(server_args) diff --git a/python/sglang/srt/managers/tokenizer_manager.py b/python/sglang/srt/managers/tokenizer_manager.py index 41df6be03..3ab16fa57 100644 --- a/python/sglang/srt/managers/tokenizer_manager.py +++ b/python/sglang/srt/managers/tokenizer_manager.py @@ -2057,11 +2057,11 @@ class TokenizerManager(TokenizerControlMixin, TokenizerManagerScoreMixin): def record_config_updates(self, source: str, **fields) -> None: """Record a control-plane config change for this engine. - Per-engine state: several ``Engine``s can share one tokenizer process. - The readback endpoints overlay these onto the startup config. The - process-global sibling is ``RuntimeContext.override`` / - ``resolved_server_args_dict``, which writes the config bags every - process shares. + These are post-startup facts the config bags do not model (weight + version, model path, the tokenizer's HiCache mirror); the readback + endpoints overlay them onto the startup config. The process-global + sibling is ``RuntimeContext.override`` / ``resolved_server_args_dict``, + which writes the config bags. """ unknown = sorted(f for f in fields if f not in _SERVER_ARGS_FIELDS) if unknown: diff --git a/python/sglang/srt/runtime_context.py b/python/sglang/srt/runtime_context.py index a52a3a942..01a809ce9 100644 --- a/python/sglang/srt/runtime_context.py +++ b/python/sglang/srt/runtime_context.py @@ -722,34 +722,6 @@ def _build_config_bags(server_args: Any) -> dict: return tops -def _snapshot_bag_values(bags: dict | None) -> dict | None: - """Per-leaf value snapshot of a config-bag tree (bags are mutated in - place by ``override``, so reference snapshots alias live state).""" - if bags is None: - return None - snap: dict = {} - - def walk(prefix: str, bag) -> None: - snap[prefix] = dict(object.__getattribute__(bag, "_fields")) - for name, sub in object.__getattribute__(bag, "_subs").items(): - walk(f"{prefix}.{name}", sub) - - for name, bag in bags.items(): - walk(name, bag) - return snap - - -def _restore_bag_values(bags: dict, snap: dict) -> None: - def walk(prefix: str, bag) -> None: - for key, value in snap[prefix].items(): - bag._set(key, value) - for name, sub in object.__getattribute__(bag, "_subs").items(): - walk(f"{prefix}.{name}", sub) - - for name, bag in bags.items(): - walk(name, bag) - - class RuntimeContext: """Container for the structured runtime accessors; exposes ``parallel``, ``server_args``, the resolved config namespace bags, ``flags``, @@ -953,12 +925,12 @@ class RuntimeContext: ``ServerArgs`` field names, so overlaying them onto the top level of either base is exact. - This covers the process-global bags only. Per-engine control-plane - changes (weight version, model path, the tokenizer's HiCache mirror) - live on the tokenizer manager — several ``Engine``s can share one - process — and ``TokenizerManager.resolved_config_dict`` overlays those - for the top-level ``/server_info`` body. The two are separate logs, not - one merged dict. + This covers the process-global bags only. Control-plane facts the bags + do not model (weight version, model path, the tokenizer's HiCache + mirror) live on the tokenizer manager, and + ``TokenizerManager.resolved_config_dict`` overlays those for the + top-level ``/server_info`` body. The two are separate logs, not one + merged dict. """ d = dict(vars(self.server_args)) if base is None else dict(base) for _source, fields in self._overrides_log: @@ -987,33 +959,6 @@ class RuntimeContext: """ return _ServerArgsOverride(self, fields) - @contextmanager - def preserve_config(self): - """Snapshot the full config lifecycle and reinstate it verbatim on exit. - - For nested construction steps that publish a private ``ServerArgs`` - copy (e.g. a draft-worker build) and must leave the enclosing - lifecycle — including its post-publish overrides — untouched. - """ - prev_server_args = self._server_args - prev_bags = self._config_bags - prev_bag_values = _snapshot_bag_values(prev_bags) - prev_overrides_log = list(self._overrides_log) - prev_publish_role = self._publish_role - prev_parallel_config = self.parallel._config - prev_capture = self.flags.capture.enable_torch_compile - try: - yield - finally: - self._server_args = prev_server_args - self._config_bags = prev_bags - if prev_bags is not None: - _restore_bag_values(prev_bags, prev_bag_values) - self._overrides_log = prev_overrides_log - self._publish_role = prev_publish_role - self.parallel._config = prev_parallel_config - self.flags.capture.enable_torch_compile = prev_capture - class _ServerArgsOverride: """Scoped config override (see ``RuntimeContext.override_server_args``). @@ -1198,7 +1143,6 @@ def get_observability() -> _ConfigBag: ROLE_NAMESPACE_SETS: dict[str, frozenset[str] | None] = { # Reads (almost) everything by design — the model-executing process. "scheduler": None, - "launcher": None, "test": None, # Audited (record-mode smokes, plain + DP-attention): the DP controller # reads only the elastic-EP gate; its module's static read set agrees. @@ -1317,21 +1261,17 @@ def publish(server_args, *, role: str, hf_config: Any = None) -> RuntimeContext: Records the process ``role`` (``tokenizer`` / ``scheduler`` / ``dp_controller`` / ``encoder`` / ``expert_backup`` / - ``weight_cache_daemon`` / ``launcher`` / ``test``) and + ``weight_cache_daemon`` / ``test``) and projects the config bags. Draft workers skip publish (they must not clobber the target). ``role`` is provenance, and — when ``SGLANG_ROLE_NAMESPACES`` is ``enforce`` — the key into ``ROLE_NAMESPACE_SETS`` for fail-closed namespace-read enforcement (``record`` audits the reads instead). ``hf_config`` is accepted for forward-compat and currently unused. - Normally one call per process, but re-publish is allowed and is - **last-publish-wins** (bags re-projected, provenance reset, role - overwritten). Two sanctioned multi-publish shapes exist: the in-process - Engine builds its ``TokenizerManager`` inside the launcher process (the - process ends up with the tokenizer publish), and multiple Engines in one - process publish in sequence — which is exactly why per-instance managers - must read ``self.server_args`` for anything engine-specific rather than - the process-global bags. + A process holds at most one live config: the bags always describe the + engine running now. Re-publish is allowed and is **last-publish-wins** + (bags re-projected, provenance reset, role overwritten), which is what + lets one process rebuild an engine after shutting the previous one down. """ if _ROLE_NS_MODE == "enforce" and role not in ROLE_NAMESPACE_SETS: # Fail closed at publish time, not at the first stray read. diff --git a/test/registered/prefill_only/test_multi_item_scoring.py b/test/registered/prefill_only/test_multi_item_scoring.py index dc43fdbef..aa29486fc 100644 --- a/test/registered/prefill_only/test_multi_item_scoring.py +++ b/test/registered/prefill_only/test_multi_item_scoring.py @@ -37,6 +37,20 @@ TEST_CLASSIFICATION_BASE_MODEL = os.environ.get( _CLS_NUM_LABELS = AutoConfig.from_pretrained(TEST_CLASSIFICATION_BASE_MODEL).num_labels +def _collect_scores(engine_kwargs, calls): + """Boot one engine, run ``calls`` through score(), shut it down. + + A process holds one live config, so the reference engine must be gone + before the engine under test boots. + """ + engine = Engine(**engine_kwargs) + try: + return [engine.score(**call).scores for call in calls] + finally: + engine.shutdown() + torch.cuda.empty_cache() + + class TestMISServerArgsValidation(unittest.TestCase): """Test ServerArgs defaults for MIS mode.""" @@ -50,8 +64,24 @@ class TestMISServerArgsValidation(unittest.TestCase): class TestMultiItemScoringOptimization(CustomTestCase): """Test the Multi-Item Scoring (MIS) optimization with generation models.""" + CONSISTENCY_CALL = dict( + query="Is this a fact?\n", + items=[" The sun rises in the east"], + label_token_ids=[9454, 2753], + apply_softmax=True, + ) + @classmethod def setUpClass(cls): + (cls.non_mis_consistency_scores,) = _collect_scores( + dict( + model_path=TEST_MODEL_NAME, + disable_radix_cache=True, + chunked_prefill_size=-1, + mem_fraction_static=0.15, + ), + [cls.CONSISTENCY_CALL], + ) cls.engine = Engine( model_path=TEST_MODEL_NAME, disable_radix_cache=True, @@ -60,19 +90,11 @@ class TestMultiItemScoringOptimization(CustomTestCase): attention_backend="flashinfer", mem_fraction_static=0.15, ) - cls.non_mis_engine = Engine( - model_path=TEST_MODEL_NAME, - disable_radix_cache=True, - chunked_prefill_size=-1, - mem_fraction_static=0.15, - ) @classmethod def tearDownClass(cls): if cls.engine is not None: cls.engine.shutdown() - if cls.non_mis_engine is not None: - cls.non_mis_engine.shutdown() torch.cuda.empty_cache() def test_mis_basic(self): @@ -98,23 +120,8 @@ class TestMultiItemScoringOptimization(CustomTestCase): def test_mis_consistency_with_single_item(self): """MIS with one item should match non-MIS scoring closely.""" - query = "Is this a fact?\n" - items = [" The sun rises in the east"] - label_token_ids = [9454, 2753] - - mis_scores = self.engine.score( - query=query, - items=items, - label_token_ids=label_token_ids, - apply_softmax=True, - ).scores - - non_mis_scores = self.non_mis_engine.score( - query=query, - items=items, - label_token_ids=label_token_ids, - apply_softmax=True, - ).scores + mis_scores = self.engine.score(**self.CONSISTENCY_CALL).scores + non_mis_scores = self.non_mis_consistency_scores self.assertEqual(len(mis_scores), 1) self.assertEqual(len(non_mis_scores), 1) @@ -151,14 +158,29 @@ class TestMultiItemScoringClassification(CustomTestCase): Pre-trained Qwen3ForSequenceClassification, so the head weights are deterministic. One class rather than four because the CI harness demands an idle GPU at every setUpClass -- splitting these means re-booting the same - two engines instead of sharing them. score() is stateless and the radix - cache is off, so sharing is safe. + engines instead of sharing them. score() is stateless and the radix cache + is off, so sharing is safe. """ NUM_LABELS = _CLS_NUM_LABELS + FALLBACK_CALL = dict(query="Test:", items=["A", "B"], apply_softmax=True) + SINGLE_VS_MIS_CALL = dict( + query="Rate this option:", + items=[" Option A", " Option B", " Option C"], + apply_softmax=True, + ) + @classmethod def setUpClass(cls): + cls.non_mis_fallback_scores, cls.non_mis_single_scores = _collect_scores( + dict( + model_path=TEST_CLASSIFICATION_BASE_MODEL, + disable_radix_cache=True, + mem_fraction_static=0.15, + ), + [cls.FALLBACK_CALL, cls.SINGLE_VS_MIS_CALL], + ) cls.engine = Engine( model_path=TEST_CLASSIFICATION_BASE_MODEL, disable_radix_cache=True, @@ -167,17 +189,11 @@ class TestMultiItemScoringClassification(CustomTestCase): attention_backend="flashinfer", mem_fraction_static=0.15, ) - cls.non_mis_engine = Engine( - model_path=TEST_CLASSIFICATION_BASE_MODEL, - disable_radix_cache=True, - mem_fraction_static=0.15, - ) @classmethod def tearDownClass(cls): - for engine in (cls.engine, cls.non_mis_engine): - if engine is not None: - engine.shutdown() + if cls.engine is not None: + cls.engine.shutdown() torch.cuda.empty_cache() def test_classification_mis_basic(self): @@ -215,9 +231,7 @@ class TestMultiItemScoringClassification(CustomTestCase): def test_classification_non_mis_fallback(self): """Classification model works correctly without --enable-mis.""" - scores = self.non_mis_engine.score( - query="Test:", items=["A", "B"], apply_softmax=True - ).scores + scores = self.non_mis_fallback_scores self.assertEqual(len(scores), 2) for score_list in scores: @@ -395,14 +409,8 @@ class TestMultiItemScoringClassification(CustomTestCase): perturbs hidden states; after softmax the scores should still land within places=1 (+-0.05). """ - query = "Rate this option:" - items = [" Option A", " Option B", " Option C"] - non_mis_scores = self.non_mis_engine.score( - query=query, items=items, apply_softmax=True - ).scores - mis_scores = self.engine.score( - query=query, items=items, apply_softmax=True - ).scores + non_mis_scores = self.non_mis_single_scores + mis_scores = self.engine.score(**self.SINGLE_VS_MIS_CALL).scores self.assertEqual(len(mis_scores), len(non_mis_scores)) for i, (ms, ns) in enumerate(zip(mis_scores, non_mis_scores)): @@ -420,48 +428,73 @@ class TestMultiItemScoringParity(CustomTestCase): """Test that MIS produces the same results as single-item scoring.""" @classmethod - def setUpClass(cls): - cls.engine_single = Engine( - model_path=TEST_MODEL_NAME, - disable_radix_cache=True, - log_level="error", - mem_fraction_static=0.15, - ) - cls.engine_mis = Engine( - model_path=TEST_MODEL_NAME, - disable_radix_cache=True, - chunked_prefill_size=-1, - log_level="error", - enable_mis=True, - attention_backend="flashinfer", - mem_fraction_static=0.15, - ) + def _cases(cls): + """The scoring calls both engines run, keyed by the test that reads them.""" + tokenizer = AutoTokenizer.from_pretrained(TEST_MODEL_NAME) + + def label_ids(labels): + return [tokenizer.encode(lb, add_special_tokens=False)[0] for lb in labels] + + return { + "basic": dict( + query="Rate this option:", + items=[" Option A", " Option B", " Option C"], + label_token_ids=label_ids([" good", " bad"]), + apply_softmax=True, + ), + "tokenized": dict( + query=tokenizer.encode("Rate this option:", add_special_tokens=False), + items=[ + tokenizer.encode(item, add_special_tokens=False) + for item in [" Option X", " Option Y"] + ], + label_token_ids=label_ids([" good", " bad"]), + apply_softmax=True, + ), + "no_softmax": dict( + query="The weather today is", + items=[" sunny", " cloudy", " rainy"], + label_token_ids=label_ids([" nice", " bad"]), + apply_softmax=False, + ), + "many_items": dict( + query="Rate this option from 1 to 5:", + items=[f" Option {i}" for i in range(10)], + label_token_ids=label_ids([" 1", " 2", " 3", " 4", " 5"]), + apply_softmax=True, + ), + } @classmethod - def tearDownClass(cls): - if cls.engine_single is not None: - cls.engine_single.shutdown() - if cls.engine_mis is not None: - cls.engine_mis.shutdown() - torch.cuda.empty_cache() + def setUpClass(cls): + cases = cls._cases() + names, calls = list(cases), list(cases.values()) + base = dict( + model_path=TEST_MODEL_NAME, + disable_radix_cache=True, + log_level="error", + mem_fraction_static=0.15, + ) + cls.single_scores = dict(zip(names, _collect_scores(base, calls))) + cls.mis_scores = dict( + zip( + names, + _collect_scores( + dict( + base, + chunked_prefill_size=-1, + enable_mis=True, + attention_backend="flashinfer", + ), + calls, + ), + ) + ) - def _compare_scores( - self, query, items, label_token_ids=None, apply_softmax=True, test_name="" - ): + def _compare_scores(self, test_name): """Compare MIS vs single-item scoring results.""" - single_scores = self.engine_single.score( - query=query, - items=items, - label_token_ids=label_token_ids, - apply_softmax=apply_softmax, - ).scores - - mis_scores = self.engine_mis.score( - query=query, - items=items, - label_token_ids=label_token_ids, - apply_softmax=apply_softmax, - ).scores + single_scores = self.single_scores[test_name] + mis_scores = self.mis_scores[test_name] self.assertEqual( len(mis_scores), len(single_scores), f"{test_name}: count mismatch" @@ -477,40 +510,16 @@ class TestMultiItemScoringParity(CustomTestCase): ) def test_parity_basic(self): - tokenizer = AutoTokenizer.from_pretrained(TEST_MODEL_NAME) - query = "Rate this option:" - items = [" Option A", " Option B", " Option C"] - labels = [" good", " bad"] - label_ids = [tokenizer.encode(lb, add_special_tokens=False)[0] for lb in labels] - self._compare_scores(query, items, label_ids, test_name="basic") + self._compare_scores("basic") def test_parity_tokenized_inputs(self): - tokenizer = AutoTokenizer.from_pretrained(TEST_MODEL_NAME) - query = "Rate this option:" - items = [" Option X", " Option Y"] - labels = [" good", " bad"] - query_ids = tokenizer.encode(query, add_special_tokens=False) - items_ids = [tokenizer.encode(i, add_special_tokens=False) for i in items] - label_ids = [tokenizer.encode(lb, add_special_tokens=False)[0] for lb in labels] - self._compare_scores(query_ids, items_ids, label_ids, test_name="tokenized") + self._compare_scores("tokenized") def test_parity_without_softmax(self): - tokenizer = AutoTokenizer.from_pretrained(TEST_MODEL_NAME) - query = "The weather today is" - items = [" sunny", " cloudy", " rainy"] - labels = [" nice", " bad"] - label_ids = [tokenizer.encode(lb, add_special_tokens=False)[0] for lb in labels] - self._compare_scores( - query, items, label_ids, apply_softmax=False, test_name="no_softmax" - ) + self._compare_scores("no_softmax") def test_parity_many_items(self): - tokenizer = AutoTokenizer.from_pretrained(TEST_MODEL_NAME) - query = "Rate this option from 1 to 5:" - items = [f" Option {i}" for i in range(10)] - labels = [" 1", " 2", " 3", " 4", " 5"] - label_ids = [tokenizer.encode(lb, add_special_tokens=False)[0] for lb in labels] - self._compare_scores(query, items, label_ids, test_name="many_items") + self._compare_scores("many_items") if __name__ == "__main__": diff --git a/test/registered/unit/managers/test_tokenizer_config_updates.py b/test/registered/unit/managers/test_tokenizer_config_updates.py index d4950280b..712cf00ed 100644 --- a/test/registered/unit/managers/test_tokenizer_config_updates.py +++ b/test/registered/unit/managers/test_tokenizer_config_updates.py @@ -2,8 +2,8 @@ Regression: runtime updates (weight version, model path, HiCache attach) were written onto the manager's ServerArgs instance so that the readback endpoints -would show them. They are per-engine — several Engines can share a tokenizer -process — so they live on the manager and the endpoints overlay them. +would show them. The record stays pristine; the updates live in a separate log +that the endpoints overlay on top of it. """ import re @@ -39,11 +39,6 @@ class TestTokenizerConfigUpdates(CustomTestCase): manager.record_config_updates("test", weight_version="v2") self.assertEqual(manager.server_args.weight_version, "v1") - def test_two_engines_keep_their_own_updates(self): - first, second = _manager(weight_version="v1"), _manager(weight_version="v1") - first.record_config_updates("test", weight_version="v2") - self.assertEqual(second.config_value("weight_version"), "v1") - def test_the_readback_dict_carries_the_updates(self): manager = _manager(hicache_storage_backend=None) manager.record_config_updates( diff --git a/test/registered/unit/multimodal/test_processor_device_selection.py b/test/registered/unit/multimodal/test_processor_device_selection.py index 57905f1dd..49458d071 100644 --- a/test/registered/unit/multimodal/test_processor_device_selection.py +++ b/test/registered/unit/multimodal/test_processor_device_selection.py @@ -1,9 +1,9 @@ """The fast-image-processor device comes from the processor's own ServerArgs. -Regression: the device decision read the published global ServerArgs, which is -last-publish-wins. Two engines in one tokenizer process then shared whichever -config published last, so one engine's images were preprocessed on the other -engine's GPU. +Regression: the device decision read the published global ServerArgs, so every +processor answered with one process-wide device. The encode-server DP workers +each drive their own GPU, which no process-global value can express — the +device has to come from what the worker was handed. """ import unittest diff --git a/test/registered/unit/test_runtime_context_override.py b/test/registered/unit/test_runtime_context_override.py index 462c0b812..8b6c5cdfb 100644 --- a/test/registered/unit/test_runtime_context_override.py +++ b/test/registered/unit/test_runtime_context_override.py @@ -109,60 +109,6 @@ class TestContextOverride(CustomTestCase): with self.assertRaises(AttributeError): sa.page_size = 999 - def test_preserve_config_keeps_post_publish_overrides(self): - # A nested build (e.g. a draft worker) publishes its own private copy; - # on exit the target's resolved bags — including post-publish - # overrides — must be reinstated verbatim, not re-projected from the - # pristine record (which would silently drop the overrides). - target = self._publish() - rc.get_context().override( - "ModelRunner.configure_kv_cache_dtype", kv_cache_dtype="fp8_e4m3" - ) - draft = ServerArgs(model_path="dummy", kv_cache_dtype="bf16") - with rc.get_context().preserve_config(): - rc.get_context().set_server_args(draft) - # Inside the scope the draft's bags are live... - self.assertEqual(rc.get_model().kv_cache_dtype, "bf16") - # ...and its own post-publish overrides work as usual. - rc.get_context().override("draft-load", kv_cache_dtype="fp8_e5m2") - self.assertEqual(rc.get_model().kv_cache_dtype, "fp8_e5m2") - # Target slot, bags, and provenance restored verbatim. - self.assertIs(rc.get_context().server_args, target) - self.assertEqual(rc.get_model().kv_cache_dtype, "fp8_e4m3") - self.assertEqual( - rc.get_context().overrides_log(), - [ - ( - "ModelRunner.configure_kv_cache_dtype", - {"kv_cache_dtype": "fp8_e4m3"}, - ) - ], - ) - - def test_preserve_config_restores_in_scope_override_without_republish(self): - # An override inside the scope (no republish) mutates the live bags - # and provenance log in place; the scope must restore entry VALUES, - # not just reassign the aliased objects. - self._publish() - rc.get_context().override("srcA", page_size=16) - with rc.get_context().preserve_config(): - rc.get_context().override("in-scope", page_size=64) - self.assertEqual(rc.get_schedule().page_size, 64) - self.assertEqual(rc.get_schedule().page_size, 16) - self.assertEqual( - rc.get_context().overrides_log(), [("srcA", {"page_size": 16})] - ) - - def test_preserve_config_restores_on_exception(self): - target = self._publish() - rc.get_context().override("srcA", page_size=16) - with self.assertRaises(RuntimeError): - with rc.get_context().preserve_config(): - rc.get_context().set_server_args(ServerArgs(model_path="dummy")) - raise RuntimeError("nested build failed") - self.assertIs(rc.get_context().server_args, target) - self.assertEqual(rc.get_schedule().page_size, 16) - def test_publish_records_role(self): rc.publish(ServerArgs(model_path="dummy"), role="scheduler") self.assertEqual(rc.publish_role(), "scheduler")