Last of four; stacked on #38048. The record is the operator's input; the bags are what is in effect. A reader that takes the record and reads a field off it gets the input, which is the wrong one of the two whenever resolution decided something -- and the mistake is silent, because for most fields and most launches the two agree. Several of these files already read both ways, sometimes in the same expression: ```python get_tokenizer( get_serving().tokenizer_path, tokenizer_mode=server_args.tokenizer_mode, # the input, not the decision ... ) ``` Sixty-odd files convert. Record field reads in runtime code go from 199 to 11. Nine parameters that the conversion emptied are dropped along with the argument at every call site -- the dead-parameter ratchet is what names them. ### "Runs after its process publishes" is a per-entry-point claim Most converted reads sit in the serving and model-executor layers, which only exist after publication, or in the two subprocess entry points, which publish first thing. Three places are not like that, and they keep reading the record they were handed: - **`HttpServerEngineAdapter`** launches the server as a *child*. The parent resolves the record and never publishes, so the adapter's own reads -- the launch banner, the API key in its readiness loop, the TP width in `update_weights_from_tensor` -- are of `self.server_args`. A bag read here fails closed in a bare process, or answers for an unrelated engine in one that happens to have published. - **`serve_grpc`** reads its sidecar port before the integrated servicer builds the `Engine` that publishes. The comment above that line already said so and already bound `cfg = resolving_view(server_args)` for it; the sidecar port and the port it derives from read `cfg`. - **`initialize_dp_attention`** runs from callers whose publish is not guaranteed, so its one predicate stays on the resolution view. `ROLE_NAMESPACE_SETS["dp_controller"]` gains `observability` and `serving`, because the controller's metrics gate, tracing setup and worker-port broadcast now read those namespaces. Under `SGLANG_ROLE_NAMESPACES=enforce` that set is what the process may read, so a conversion that reaches a new namespace has to widen it in the same change. ## Three things worth a reviewer's attention **Eleven reads were `getattr(record, "field", default)`.** An AST scan for attribute access does not see those, so the census that said "43 readers" was counting the shape it could match rather than the thing it was after. `incremental_streaming_output` was read that way twice, and the transcription tests were the only reason it surfaced. **Not every record read is a bag read waiting to happen.** A multimodal processor's `base_gpu_id` is the instance's, not the process's: two engines in one process keep different ones, and `test_publishing_another_config_does_not_move_the_device` exists to say so. It stays on the record while `rl_on_policy_target` beside it moves. `RequestMetricsExporter` is the same shape -- it is handed the directory it writes to, and a test builds several with different ones. `configure_logger` is a third: 17 call sites, one of which passes an `argparse.Namespace`, so it is not a global-context reader at all. Those eleven remaining reads are the ones with a reason. **The fixtures move with the code.** Tests that hung config off a mock manager now publish a record, which is what the serving layer reads; where a test states a value it says so with `override_server_args` instead of assigning through the mock. `test_hisparse_unit` is the last of them: it stubbed a `server_args` onto a fake scheduler to say the decode radix cache was off, and the value it was standing in for is the published default, so the stub goes and the class publishes. ## Two things CI caught that a local sweep could not **`unittest.TestCase.enterContext` is Python 3.11+.** The converted fixtures used it at 18 sites; `requires-python` is `>=3.10` and CI runs 3.10, so every one of them raised `AttributeError` there while passing on a newer local interpreter. They call `enter_override(self, ...)` now -- a four-line helper in `sglang/test/test_utils.py` over the override's own `install()` / `restore()`. **A batched sweep cannot see a missing publish.** Three fixtures needed a published config and did not have one; each *passed* inside a shard where some other file had published, and failed when run alone. The affected cases are `test_serving_completions` (which set `incremental_streaming_output` on the mock manager's record, where nothing reads it now), `test_qwen3_vl_feature_materialization` (same shape for `mm_enable_dp_encoder`), and the two Qwen Rust tests -- whose fixture already carried the comment `# Non-auto: get_resolved_model_impl would choke on a SimpleNamespace` next to the `model_impl` it sets, which is exactly what happened once `get_mm_processor_cls` started reading that value from the bag. Its `publish` mirrors `model_impl` now, like the four fields it already mirrored. ## Verification A full registered-unit sweep (648 files) against this stack's merge-base: 19 failures on both sides, the same 19, none of them config. That sweep is what caught 23 failures the file-scoped runs missed -- and, later, that the narrower 139-file list did not even contain the files this change reaches. It is also what caught the `test_hisparse_unit` fixture above: the file passes inside a shard where something else published, and fails when it is run on its own, which is why every failing file is re-run alone before it is counted.
272 lines
11 KiB
Python
272 lines
11 KiB
Python
"""
|
|
Unit tests for OpenAI-compatible LoRA API support.
|
|
|
|
Tests the model parameter parsing and LoRA adapter resolution logic
|
|
that enables OpenAI-compatible LoRA adapter selection.
|
|
"""
|
|
|
|
import unittest
|
|
from unittest.mock import MagicMock
|
|
|
|
from sglang.srt.entrypoints.openai.serving_base import OpenAIServingBase
|
|
from sglang.srt.runtime_context import publish, reset_context
|
|
from sglang.srt.server_args import ServerArgs
|
|
from sglang.test.ci.ci_register import register_amd_ci, register_cpu_ci
|
|
|
|
register_amd_ci(est_time=30, suite="nightly-amd-1-gpu", nightly=True)
|
|
register_cpu_ci(est_time=7, suite="stage-b-test-cpu-intel")
|
|
|
|
|
|
def publish_config(case):
|
|
"""`OpenAIServingBase.__init__` reads `get_observability()`, so a case that
|
|
builds one needs a published config. The mock manager cannot stand in for
|
|
it: `MagicMock(spec=ServerArgs)` passes the `isinstance` guard, so the read
|
|
happens and there is no bag to answer from."""
|
|
reset_context()
|
|
case.addCleanup(reset_context)
|
|
publish(ServerArgs(model_path="dummy"), role="tokenizer")
|
|
|
|
|
|
class MockTokenizerManager:
|
|
"""Mock TokenizerManager for testing."""
|
|
|
|
def __init__(self, enable_lora=False):
|
|
self.server_args = MagicMock(spec=ServerArgs)
|
|
self.server_args.enable_lora = enable_lora
|
|
|
|
|
|
class ConcreteServingBase(OpenAIServingBase):
|
|
"""Concrete implementation for testing abstract base class."""
|
|
|
|
def _request_id_prefix(self) -> str:
|
|
return "test-"
|
|
|
|
def _convert_to_internal_request(self, request, raw_request=None):
|
|
pass
|
|
|
|
def _validate_request(self, request):
|
|
pass
|
|
|
|
|
|
class TestParseModelParameter(unittest.TestCase):
|
|
"""Test _parse_model_parameter method."""
|
|
|
|
def setUp(self):
|
|
publish_config(self)
|
|
self.tokenizer_manager = MockTokenizerManager(enable_lora=True)
|
|
self.serving = ConcreteServingBase(self.tokenizer_manager)
|
|
|
|
def test_model_without_adapter(self):
|
|
"""Test parsing model without adapter returns None for adapter."""
|
|
base_model, adapter = self.serving._parse_model_parameter("llama-3.1-8B")
|
|
self.assertEqual(base_model, "llama-3.1-8B")
|
|
self.assertIsNone(adapter)
|
|
|
|
def test_model_with_adapter(self):
|
|
"""Test parsing model with adapter extracts both parts."""
|
|
base_model, adapter = self.serving._parse_model_parameter(
|
|
"llama-3.1-8B:sql-expert"
|
|
)
|
|
self.assertEqual(base_model, "llama-3.1-8B")
|
|
self.assertEqual(adapter, "sql-expert")
|
|
|
|
def test_model_with_path_and_adapter(self):
|
|
"""Test parsing model path with slashes and adapter."""
|
|
base_model, adapter = self.serving._parse_model_parameter(
|
|
"meta-llama/Llama-3.1-8B-Instruct:adapter-name"
|
|
)
|
|
self.assertEqual(base_model, "meta-llama/Llama-3.1-8B-Instruct")
|
|
self.assertEqual(adapter, "adapter-name")
|
|
|
|
def test_model_with_multiple_colons(self):
|
|
"""Test that only first colon is used for splitting."""
|
|
base_model, adapter = self.serving._parse_model_parameter("model:adapter:extra")
|
|
self.assertEqual(base_model, "model")
|
|
self.assertEqual(adapter, "adapter:extra")
|
|
|
|
def test_model_with_whitespace(self):
|
|
"""Test that whitespace is stripped from both parts."""
|
|
base_model, adapter = self.serving._parse_model_parameter(
|
|
" model-name : adapter-name "
|
|
)
|
|
self.assertEqual(base_model, "model-name")
|
|
self.assertEqual(adapter, "adapter-name")
|
|
|
|
def test_model_with_empty_adapter(self):
|
|
"""Test model ending with colon returns None for adapter."""
|
|
base_model, adapter = self.serving._parse_model_parameter("model-name:")
|
|
self.assertEqual(base_model, "model-name")
|
|
self.assertIsNone(adapter)
|
|
|
|
def test_model_with_only_spaces_after_colon(self):
|
|
"""Test model with only whitespace after colon returns None for adapter."""
|
|
base_model, adapter = self.serving._parse_model_parameter("model-name: ")
|
|
self.assertEqual(base_model, "model-name")
|
|
self.assertIsNone(adapter)
|
|
|
|
|
|
class TestResolveLoraPath(unittest.TestCase):
|
|
"""Test _resolve_lora_path method."""
|
|
|
|
def setUp(self):
|
|
publish_config(self)
|
|
self.tokenizer_manager = MockTokenizerManager(enable_lora=True)
|
|
self.serving = ConcreteServingBase(self.tokenizer_manager)
|
|
|
|
def test_no_adapter_specified(self):
|
|
"""Test when neither model nor explicit lora_path has adapter."""
|
|
result = self.serving._resolve_lora_path("model-name", None)
|
|
self.assertIsNone(result)
|
|
|
|
def test_adapter_in_model_only(self):
|
|
"""Test adapter from model parameter when no explicit path."""
|
|
result = self.serving._resolve_lora_path("model:sql-expert", None)
|
|
self.assertEqual(result, "sql-expert")
|
|
|
|
def test_adapter_in_explicit_only(self):
|
|
"""Test adapter from explicit lora_path when not in model."""
|
|
result = self.serving._resolve_lora_path("model-name", "python-expert")
|
|
self.assertEqual(result, "python-expert")
|
|
|
|
def test_model_parameter_takes_precedence(self):
|
|
"""Test model parameter adapter takes precedence over explicit."""
|
|
result = self.serving._resolve_lora_path("model:sql-expert", "python-expert")
|
|
self.assertEqual(result, "sql-expert")
|
|
|
|
def test_with_list_explicit_lora_path(self):
|
|
"""Test that explicit list is returned when no model adapter."""
|
|
explicit = ["adapter1", "adapter2", None]
|
|
result = self.serving._resolve_lora_path("model-name", explicit)
|
|
self.assertEqual(result, explicit)
|
|
|
|
def test_model_adapter_overrides_list(self):
|
|
"""Test model adapter overrides even when explicit is a list."""
|
|
result = self.serving._resolve_lora_path(
|
|
"model:sql-expert", ["adapter1", "adapter2"]
|
|
)
|
|
self.assertEqual(result, "sql-expert")
|
|
|
|
def test_complex_model_name_with_adapter(self):
|
|
"""Test resolution with complex model name."""
|
|
result = self.serving._resolve_lora_path(
|
|
"org/model-v2.1:adapter-name", "other-adapter"
|
|
)
|
|
self.assertEqual(result, "adapter-name")
|
|
|
|
|
|
class TestIntegrationScenarios(unittest.TestCase):
|
|
"""Integration tests for common usage scenarios."""
|
|
|
|
def setUp(self):
|
|
publish_config(self)
|
|
self.tokenizer_manager = MockTokenizerManager(enable_lora=True)
|
|
self.serving = ConcreteServingBase(self.tokenizer_manager)
|
|
|
|
def test_openai_compatible_usage(self):
|
|
"""Test typical OpenAI-compatible usage pattern."""
|
|
# User specifies adapter in model parameter
|
|
model = "meta-llama/Llama-3.1-8B:sql-expert"
|
|
explicit_lora = None
|
|
|
|
lora_path = self.serving._resolve_lora_path(model, explicit_lora)
|
|
self.assertEqual(lora_path, "sql-expert")
|
|
|
|
def test_backward_compatible_usage(self):
|
|
"""Test backward-compatible usage with explicit lora_path."""
|
|
model = "meta-llama/Llama-3.1-8B"
|
|
explicit_lora = "sql-expert"
|
|
|
|
lora_path = self.serving._resolve_lora_path(model, explicit_lora)
|
|
self.assertEqual(lora_path, "sql-expert")
|
|
|
|
def test_base_model_usage(self):
|
|
"""Test using base model without any adapter."""
|
|
model = "meta-llama/Llama-3.1-8B"
|
|
explicit_lora = None
|
|
|
|
lora_path = self.serving._resolve_lora_path(model, explicit_lora)
|
|
self.assertIsNone(lora_path)
|
|
|
|
# No validation needed when no adapter
|
|
|
|
def test_batch_request_scenario(self):
|
|
"""Test batch request with list of adapters."""
|
|
model = "meta-llama/Llama-3.1-8B" # No adapter in model
|
|
explicit_lora = ["sql-expert", "python-expert", None]
|
|
|
|
lora_path = self.serving._resolve_lora_path(model, explicit_lora)
|
|
self.assertEqual(lora_path, explicit_lora)
|
|
|
|
def test_adapter_in_model_overrides_batch_list(self):
|
|
"""Test that adapter in model parameter overrides batch list."""
|
|
model = "meta-llama/Llama-3.1-8B:preferred-adapter"
|
|
explicit_lora = ["adapter1", "adapter2"]
|
|
|
|
lora_path = self.serving._resolve_lora_path(model, explicit_lora)
|
|
self.assertEqual(lora_path, "preferred-adapter")
|
|
|
|
|
|
class TestEdgeCases(unittest.TestCase):
|
|
"""Test edge cases and error conditions."""
|
|
|
|
def setUp(self):
|
|
publish_config(self)
|
|
self.tokenizer_manager = MockTokenizerManager(enable_lora=True)
|
|
self.serving = ConcreteServingBase(self.tokenizer_manager)
|
|
|
|
def test_empty_string_model(self):
|
|
"""Test handling of empty string model."""
|
|
base, adapter = self.serving._parse_model_parameter("")
|
|
self.assertEqual(base, "")
|
|
self.assertIsNone(adapter)
|
|
|
|
def test_only_colon(self):
|
|
"""Test model parameter that is just a colon."""
|
|
base, adapter = self.serving._parse_model_parameter(":")
|
|
self.assertEqual(base, "")
|
|
self.assertIsNone(adapter)
|
|
|
|
def test_empty_list_lora_path(self):
|
|
"""Test validation with empty list doesn't crash."""
|
|
lora_path = self.serving._resolve_lora_path("model-name", [])
|
|
# Empty list is falsy, so validation won't be called
|
|
self.assertEqual(lora_path, [])
|
|
|
|
def test_list_with_none_first(self):
|
|
"""Test validation finds first non-None adapter in list."""
|
|
lora_path = self.serving._resolve_lora_path("model-name", [None, "adapter2"])
|
|
self.assertEqual(lora_path, [None, "adapter2"])
|
|
# In actual usage, validation would find "adapter2"
|
|
|
|
def test_list_all_none(self):
|
|
"""Test validation with list of all None values."""
|
|
lora_path = self.serving._resolve_lora_path("model-name", [None, None])
|
|
self.assertEqual(lora_path, [None, None])
|
|
# In actual usage, no validation would occur (no non-None adapters)
|
|
|
|
def test_unicode_in_adapter_name(self):
|
|
"""Test Unicode characters in adapter name."""
|
|
base, adapter = self.serving._parse_model_parameter("model:adapter-名前")
|
|
self.assertEqual(base, "model")
|
|
self.assertEqual(adapter, "adapter-名前")
|
|
|
|
def test_special_characters_in_adapter(self):
|
|
"""Test special characters in adapter name."""
|
|
base, adapter = self.serving._parse_model_parameter("model:adapter_v2.1-final")
|
|
self.assertEqual(base, "model")
|
|
self.assertEqual(adapter, "adapter_v2.1-final")
|
|
|
|
def test_none_as_explicit_lora_path(self):
|
|
"""Test None as explicit lora_path is handled correctly."""
|
|
result = self.serving._resolve_lora_path("model:adapter", None)
|
|
self.assertEqual(result, "adapter")
|
|
|
|
def test_empty_string_as_explicit_lora_path(self):
|
|
"""Test empty string as explicit lora_path."""
|
|
result = self.serving._resolve_lora_path("model-name", "")
|
|
self.assertEqual(result, "")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
unittest.main()
|