minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity
Files
1f60ddef5dc2ae3bbfbe0c5cea45690c4b60a251
sglang/test/registered/unit/entrypoints/openai/test_serving_completions.py
T
Cheng Wan b99175dc7d [Config] Round 6.4: the runtime reads the bags, not the record (#38049) 
Last of four; stacked on #38048.

The record is the operator's input; the bags are what is in effect. A reader
that takes the record and reads a field off it gets the input, which is the
wrong one of the two whenever resolution decided something -- and the mistake is
silent, because for most fields and most launches the two agree. Several of
these files already read both ways, sometimes in the same expression:

```python
get_tokenizer(
    get_serving().tokenizer_path,
    tokenizer_mode=server_args.tokenizer_mode,   # the input, not the decision
    ...
)
```

Sixty-odd files convert. Record field reads in runtime code go from 199 to 11.
Nine parameters that the conversion emptied are dropped along with the argument
at every call site -- the dead-parameter ratchet is what names them.

### "Runs after its process publishes" is a per-entry-point claim

Most converted reads sit in the serving and model-executor layers, which only
exist after publication, or in the two subprocess entry points, which publish
first thing. Three places are not like that, and they keep reading the record
they were handed:

- **`HttpServerEngineAdapter`** launches the server as a *child*. The parent
  resolves the record and never publishes, so the adapter's own reads -- the
  launch banner, the API key in its readiness loop, the TP width in
  `update_weights_from_tensor` -- are of `self.server_args`. A bag read here
  fails closed in a bare process, or answers for an unrelated engine in one that
  happens to have published.
- **`serve_grpc`** reads its sidecar port before the integrated servicer builds
  the `Engine` that publishes. The comment above that line already said so and
  already bound `cfg = resolving_view(server_args)` for it; the sidecar port and
  the port it derives from read `cfg`.
- **`initialize_dp_attention`** runs from callers whose publish is not
  guaranteed, so its one predicate stays on the resolution view.

`ROLE_NAMESPACE_SETS["dp_controller"]` gains `observability` and `serving`,
because the controller's metrics gate, tracing setup and worker-port broadcast
now read those namespaces. Under `SGLANG_ROLE_NAMESPACES=enforce` that set is
what the process may read, so a conversion that reaches a new namespace has to
widen it in the same change.

## Three things worth a reviewer's attention

**Eleven reads were `getattr(record, "field", default)`.** An AST scan for
attribute access does not see those, so the census that said "43 readers" was
counting the shape it could match rather than the thing it was after.
`incremental_streaming_output` was read that way twice, and the transcription
tests were the only reason it surfaced.

**Not every record read is a bag read waiting to happen.** A multimodal
processor's `base_gpu_id` is the instance's, not the process's: two engines in
one process keep different ones, and
`test_publishing_another_config_does_not_move_the_device` exists to say so. It
stays on the record while `rl_on_policy_target` beside it moves.
`RequestMetricsExporter` is the same shape -- it is handed the directory it
writes to, and a test builds several with different ones. `configure_logger` is
a third: 17 call sites, one of which passes an `argparse.Namespace`, so it is
not a global-context reader at all. Those eleven remaining reads are the ones
with a reason.

**The fixtures move with the code.** Tests that hung config off a mock manager
now publish a record, which is what the serving layer reads; where a test states
a value it says so with `override_server_args` instead of assigning through the
mock. `test_hisparse_unit` is the last of them: it stubbed a `server_args` onto
a fake scheduler to say the decode radix cache was off, and the value it was
standing in for is the published default, so the stub goes and the class
publishes.

## Two things CI caught that a local sweep could not

**`unittest.TestCase.enterContext` is Python 3.11+.** The converted fixtures used
it at 18 sites; `requires-python` is `>=3.10` and CI runs 3.10, so every one of
them raised `AttributeError` there while passing on a newer local interpreter.
They call `enter_override(self, ...)` now -- a four-line helper in
`sglang/test/test_utils.py` over the override's own `install()` / `restore()`.

**A batched sweep cannot see a missing publish.** Three fixtures needed a
published config and did not have one; each *passed* inside a shard where some
other file had published, and failed when run alone. The affected cases are
`test_serving_completions` (which set `incremental_streaming_output` on the mock
manager's record, where nothing reads it now), `test_qwen3_vl_feature_materialization`
(same shape for `mm_enable_dp_encoder`), and the two Qwen Rust tests -- whose
fixture already carried the comment `# Non-auto: get_resolved_model_impl would
choke on a SimpleNamespace` next to the `model_impl` it sets, which is exactly
what happened once `get_mm_processor_cls` started reading that value from the
bag. Its `publish` mirrors `model_impl` now, like the four fields it already
mirrored.

## Verification

A full registered-unit sweep (648 files) against this stack's merge-base:
19 failures on both sides, the same 19, none of them config. That sweep is what
caught 23 failures the file-scoped runs missed -- and, later, that the narrower
139-file list did not even contain the files this change reaches. It is also
what caught the `test_hisparse_unit` fixture above: the file passes inside a
shard where something else published, and fails when it is run on its own,
which is why every failing file is re-run alone before it is counted.
2026-09-06 21:41:46 -07:00

605 lines
22 KiB
Python
Raw Blame History

"""
Unit-tests for the refactored completions-serving handler (no pytest).
Run with:
python -m unittest tests.test_serving_completions_unit -v
"""
from sglang.test.test_utils import maybe_stub_sgl_kernel
maybe_stub_sgl_kernel() # must precede any import that pulls in sgl_kernel
import json
import unittest
from http import HTTPStatus
from typing import Optional
from unittest.mock import AsyncMock, Mock
from fastapi import Request
from sglang.srt.entrypoints.openai.protocol import CompletionRequest
from sglang.srt.entrypoints.openai.serving_completions import OpenAIServingCompletion
from sglang.srt.managers.tokenizer_manager import TokenizerManager
from sglang.srt.runtime_context import get_context, publish, reset_context
from sglang.srt.server_args import ServerArgs
from sglang.srt.utils import get_or_create_event_loop
from sglang.test.ci.ci_register import register_cpu_ci
register_cpu_ci(est_time=11, suite="base-a-test-cpu")
def _spec_result(index):
return {
"text": f"choice-{index}",
"meta_info": {
"id": "cmpl-spec-test",
"prompt_tokens": 10,
"completion_tokens": 2,
"cached_tokens": 0,
"finish_reason": {"type": "stop"},
"weight_version": "default",
"spec_accept_rate": 0.5,
"spec_accept_length": 2.0,
"spec_cap_length": index + 1.0,
"spec_block_accept_length": index + 0.5,
"spec_num_correct_drafts": 1,
"spec_num_proposed_drafts": 2,
"spec_verify_ct": 1,
"spec_correct_drafts_histogram": [0, 1],
"spec_cap_lens_histogram": [index, 1],
},
"index": index,
}
class _MockTemplateManager:
"""Minimal mock for TemplateManager."""
def __init__(self):
self.chat_template_name: Optional[str] = None
self.jinja_template_content_format: Optional[str] = None
self.completion_template_name: Optional[str] = (
None # Set to None to avoid template processing
)
self.jinja_template_may_reorder_tool_results = False
class ServingCompletionTestCase(unittest.TestCase):
"""Bundle all prompt/echo tests in one TestCase."""
# ---------- shared test fixtures ----------
def setUp(self):
reset_context()
self.addCleanup(reset_context)
publish(ServerArgs(model_path="dummy"), role="tokenizer")
# build the mock TokenizerManager once for every test
tm = Mock(spec=TokenizerManager)
tm.tokenizer = Mock()
tm.tokenizer.encode.return_value = [1, 2, 3, 4]
tm.tokenizer.decode.return_value = "decoded text"
tm.tokenizer.bos_token_id = 1
tm.model_config = Mock(is_multimodal=False)
tm.server_args = Mock(enable_cache_report=False)
tm.generate_request = AsyncMock()
tm.create_abort_task = Mock()
self.template_manager = _MockTemplateManager()
self.sc = OpenAIServingCompletion(tm, self.template_manager)
self.fastapi_request = Mock(spec=Request)
# ---------- prompt-handling ----------
def test_single_token_ids_prompt(self):
req = CompletionRequest(model="x", prompt=[1, 2, 3, 4], max_tokens=100)
internal, _ = self.sc._convert_to_internal_request(req)
self.assertEqual(internal.input_ids, [1, 2, 3, 4])
def test_cache_salt_and_extra_key_remain_distinct(self):
req = CompletionRequest(
model="x",
prompt=[1, 2, 3, 4],
max_tokens=1,
cache_salt="tenant-a",
extra_key="classification",
)
internal, _ = self.sc._convert_to_internal_request(req)
self.assertEqual(internal.cache_salt, "tenant-a")
self.assertEqual(internal.extra_key, "classification")
def test_single_request_rejects_batched_cache_salt(self):
req = CompletionRequest(
model="x",
prompt=[1, 2, 3, 4],
max_tokens=1,
cache_salt=["tenant-a"],
)
internal, _ = self.sc._convert_to_internal_request(req)
with self.assertRaisesRegex(ValueError, "single request"):
internal.normalize_batch_and_arguments()
# ---------- echo-handling ----------
def test_echo_with_list_of_strings_streaming(self):
req = CompletionRequest(
model="x", prompt=["A", "B"], max_tokens=1, echo=True, n=1
)
self.assertEqual(self.sc._get_echo_text(req, 0), "A")
self.assertEqual(self.sc._get_echo_text(req, 1), "B")
def test_echo_with_token_ids_streaming(self):
req = CompletionRequest(model="x", prompt=[1, 2, 3], max_tokens=1, echo=True)
self.sc.tokenizer_manager.tokenizer.decode.return_value = "decoded_prompt"
self.assertEqual(self.sc._get_echo_text(req, 0), "decoded_prompt")
def test_echo_with_multiple_token_ids_streaming(self):
req = CompletionRequest(
model="x", prompt=[[1, 2], [3, 4]], max_tokens=1, echo=True, n=1
)
self.sc.tokenizer_manager.tokenizer.decode.return_value = "decoded"
self.assertEqual(self.sc._get_echo_text(req, 0), "decoded")
def test_prepare_echo_prompts_non_streaming(self):
# single string
req = CompletionRequest(model="x", prompt="Hi", echo=True)
self.assertEqual(self.sc._prepare_echo_prompts(req), ["Hi"])
# list of strings
req = CompletionRequest(model="x", prompt=["Hi", "Yo"], echo=True)
self.assertEqual(self.sc._prepare_echo_prompts(req), ["Hi", "Yo"])
# token IDs
req = CompletionRequest(model="x", prompt=[1, 2, 3], echo=True)
self.sc.tokenizer_manager.tokenizer.decode.return_value = "decoded"
self.assertEqual(self.sc._prepare_echo_prompts(req), ["decoded"])
# ---------- response_format handling ----------
def test_response_format_json_object(self):
"""Test that response_format json_object is correctly processed in sampling params."""
req = CompletionRequest(
model="x",
prompt="Generate a JSON object:",
max_tokens=100,
response_format={"type": "json_object"},
)
sampling_params = self.sc._build_sampling_params(req)
self.assertEqual(sampling_params["json_schema"], '{"type": "object"}')
def test_response_format_json_schema(self):
"""Test that response_format json_schema is correctly processed in sampling params."""
schema = {
"type": "object",
"properties": {"name": {"type": "string"}, "age": {"type": "integer"}},
}
req = CompletionRequest(
model="x",
prompt="Generate a JSON object:",
max_tokens=100,
response_format={
"type": "json_schema",
"json_schema": {"name": "person", "schema": schema},
},
)
sampling_params = self.sc._build_sampling_params(req)
# The schema should be converted to string by convert_json_schema_to_str
self.assertIn("json_schema", sampling_params)
self.assertIsInstance(sampling_params["json_schema"], str)
def test_response_format_json_schema_missing_schema(self):
"""Test that json_schema response_format without a schema raises a ValueError."""
req = CompletionRequest(
model="x",
prompt="Generate a JSON object:",
max_tokens=100,
response_format={"type": "json_schema"},
)
with self.assertRaises(ValueError):
self.sc._build_sampling_params(req)
def test_response_format_structural_tag(self):
"""Test that response_format structural_tag is correctly processed in sampling params."""
req = CompletionRequest(
model="x",
prompt="Generate structured output:",
max_tokens=100,
response_format={
"type": "structural_tag",
"structures": [{"begin": "<data>", "end": "</data>"}],
"triggers": ["<data>"],
},
)
sampling_params = self.sc._build_sampling_params(req)
# The structural_tag should be processed
self.assertIn("structural_tag", sampling_params)
self.assertIsInstance(sampling_params["structural_tag"], str)
def test_response_format_none(self):
"""Test that no response_format doesn't add extra constraints."""
req = CompletionRequest(model="x", prompt="Generate text:", max_tokens=100)
sampling_params = self.sc._build_sampling_params(req)
# Should not have json_schema or structural_tag from response_format
# (but might have json_schema from the legacy json_schema field)
self.assertIsNone(sampling_params.get("structural_tag"))
def test_non_streaming_response(self):
req = CompletionRequest(
model="x",
prompt="Hello",
max_tokens=10,
logprobs=False,
return_token_ids=True,
)
mock_ret = [
{
"text": " world",
"output_ids": [3, 4],
"prompt_token_ids": [1, 2],
"meta_info": {
"id": "test-id",
"prompt_tokens": 1,
"completion_tokens": 2,
"finish_reason": {"type": "stop"},
"weight_version": "v1",
},
}
]
response = self.sc._build_completion_response(req, mock_ret, 1234567890)
self.assertEqual(len(response.choices), 1)
self.assertEqual(response.choices[0].text, " world")
self.assertEqual(len(response.choices[0].logprobs.top_logprobs), 0)
self.assertEqual(response.choices[0].token_ids, [3, 4])
self.assertEqual(response.choices[0].prompt_token_ids, [1, 2])
def test_streaming_abort_yields_error(self):
"""Test that an abort finish reason during streaming correctly yields an error and stops."""
err_msg = "Aborted by scheduler"
err_code = HTTPStatus.INTERNAL_SERVER_ERROR
async def _mock_generate_abort(*args, **kwargs):
yield {
"text": "Partial ",
"meta_info": {
"id": "cmpl-test",
"prompt_tokens": 10,
"completion_tokens": 2,
"cached_tokens": 0,
"finish_reason": {
"type": "abort",
"status_code": err_code,
"message": err_msg,
},
"output_token_logprobs": None,
"output_top_logprobs": None,
},
"index": 0,
}
self.sc.tokenizer_manager.generate_request = _mock_generate_abort
req = CompletionRequest(
model="x",
prompt="Hello world",
max_tokens=100,
stream=True,
)
adapted_request, _ = self.sc._convert_to_internal_request(req)
async def run_stream():
chunks = []
try:
async for chunk in self.sc._generate_completion_stream(
adapted_request, req, self.fastapi_request
):
chunks.append(chunk)
except Exception as e:
print(f"Error during stream iteration: {e}")
return chunks
loop = get_or_create_event_loop()
chunks = loop.run_until_complete(run_stream())
error_chunk_data = None
for c in chunks:
if "error" in c:
error_chunk_data = json.loads(c[len("data: ") :])
break
self.assertIsNotNone(error_chunk_data, "Error chunk not found in stream")
self.assertEqual(error_chunk_data["error"]["message"], err_msg)
self.assertEqual(error_chunk_data["error"]["code"], err_code.value)
# Ensure the stream stops after the abort error
# The last chunk should be "data: [DONE]\n\n"
self.assertEqual(chunks[-1], "data: [DONE]\n\n")
# Check that there is an error chunk and a DONE chunk, and possibly a role chunk
self.assertGreaterEqual(len(chunks), 2)
self.assertIn("error", chunks[0])
def test_streaming_token_ids_deltas_cover_output_exactly(self):
req = CompletionRequest(
model="x",
prompt="Hi",
max_tokens=10,
stream=True,
return_token_ids=True,
)
adapted_request, _ = self.sc._convert_to_internal_request(req)
for incremental in (False, True):
# Both of these are read through `get_serving()` now, so assigning
# them on the mock manager's record has no effect on what the code
# under test sees. State them where the code reads them.
with (
self.subTest(incremental_streaming_output=incremental),
get_context().override_server_args(
stream_response_default_include_usage=False,
incremental_streaming_output=incremental,
),
):
texts = ("a", "b", "c") if incremental else ("a", "ab", "abc")
output_ids = (
([5], [6], [7]) if incremental else ([5], [5, 6], [5, 6, 7])
)
chunks = [
{
"text": text,
"output_ids": ids,
"prompt_token_ids": [1, 2],
"meta_info": {
"id": "cmpl-test",
"prompt_tokens": 2,
"completion_tokens": i + 1,
"finish_reason": {"type": "stop"} if i == 2 else None,
},
"index": 0,
}
for i, (text, ids) in enumerate(zip(texts, output_ids))
]
async def _mock_generate(*args, _chunks=chunks, **kwargs):
for chunk in _chunks:
yield chunk
self.sc.tokenizer_manager.generate_request = _mock_generate
async def run_stream():
return [
chunk
async for chunk in self.sc._generate_completion_stream(
adapted_request, req, self.fastapi_request
)
]
loop = get_or_create_event_loop()
raw_chunks = loop.run_until_complete(run_stream())
choices = []
for raw in raw_chunks:
if not raw.startswith("data: ") or raw.strip() == "data: [DONE]":
continue
data = json.loads(raw[len("data: ") :])
choices.extend(data.get("choices", []))
token_ids = [tid for c in choices for tid in c.get("token_ids", [])]
text = "".join(c["text"] for c in choices)
self.assertEqual(text, "abc")
self.assertEqual(token_ids, [5, 6, 7])
self.assertEqual(choices[0]["prompt_token_ids"], [1, 2])
for choice in choices[1:]:
self.assertNotIn("prompt_token_ids", choice)
def test_non_streaming_cached_tokens_details_emits_sglext(self):
"""Test that non-streaming completion responses emit cached token details in sglext."""
req = CompletionRequest(
model="x",
prompt="Hello world",
max_tokens=100,
return_cached_tokens_details=True,
)
ret = [
{
"text": "Cached response",
"meta_info": {
"id": "cmpl-cache-test",
"prompt_tokens": 10,
"completion_tokens": 2,
"cached_tokens": 6,
"cached_tokens_details": {
"device": 4,
"host": 1,
"storage": 1,
"storage_backend": "file",
},
"finish_reason": {"type": "stop", "matched": None},
"weight_version": "default",
},
}
]
response = self.sc._build_completion_response(req, ret, 1234567890)
self.assertIsNotNone(response.sglext)
self.assertEqual(
response.sglext.cached_tokens_details.model_dump(exclude_none=True),
{
"device": 4,
"host": 1,
"storage": 1,
"storage_backend": "file",
},
)
def test_parallel_sampling_returns_spec_details_per_choice(self):
req = CompletionRequest(
model="x",
prompt="Hello world",
max_tokens=100,
n=2,
return_spec_tokens_details=True,
)
ret = [_spec_result(index) for index in range(2)]
response = self.sc._build_completion_response(req, ret, 1234567890)
details = response.sglext.spec_tokens_details
self.assertEqual(len(details), 2)
self.assertEqual(details[0].spec_cap_length, 1.0)
self.assertEqual(details[0].spec_block_accept_length, 0.5)
self.assertEqual(details[0].spec_cap_lens_histogram, [0, 1])
self.assertEqual(details[1].spec_cap_length, 2.0)
self.assertEqual(details[1].spec_block_accept_length, 1.5)
self.assertEqual(details[1].spec_cap_lens_histogram, [1, 1])
single_req = req.model_copy(update={"n": 1})
single_response = self.sc._build_completion_response(
single_req, ret[:1], 1234567890
)
self.assertEqual(
single_response.sglext.spec_tokens_details.spec_cap_length,
1.0,
)
disabled_req = single_req.model_copy(
update={"return_spec_tokens_details": False}
)
disabled_response = self.sc._build_completion_response(
disabled_req, ret[:1], 1234567890
)
self.assertIsNone(disabled_response.sglext)
def test_streaming_parallel_sampling_orders_spec_details_by_choice(self):
async def mock_generate(*args, **kwargs):
for index in (1, 0):
yield _spec_result(index)
self.sc.tokenizer_manager.generate_request = mock_generate
req = CompletionRequest(
model="x",
prompt="Hello world",
max_tokens=100,
n=2,
stream=True,
return_spec_tokens_details=True,
)
adapted_request, _ = self.sc._convert_to_internal_request(req)
async def run_stream(request):
return [
chunk
async for chunk in self.sc._generate_completion_stream(
adapted_request, request, self.fastapi_request
)
]
chunks = get_or_create_event_loop().run_until_complete(run_stream(req))
parsed = [
json.loads(chunk[len("data: ") :])
for chunk in chunks
if chunk.startswith("data: ") and chunk.strip() != "data: [DONE]"
]
details = next(chunk["sglext"] for chunk in parsed if "sglext" in chunk)[
"spec_tokens_details"
]
self.assertEqual([item["spec_cap_length"] for item in details], [1.0, 2.0])
self.assertEqual(
[item["spec_cap_lens_histogram"] for item in details],
[[0, 1], [1, 1]],
)
async def mock_single_generate(*args, **kwargs):
async for content in mock_generate():
if content["index"] == 0:
yield content
self.sc.tokenizer_manager.generate_request = mock_single_generate
single_req = req.model_copy(update={"n": 1})
single_chunks = get_or_create_event_loop().run_until_complete(
run_stream(single_req)
)
single_parsed = [
json.loads(chunk[len("data: ") :])
for chunk in single_chunks
if chunk.startswith("data: ") and chunk.strip() != "data: [DONE]"
]
single_details = next(
chunk["sglext"] for chunk in single_parsed if "sglext" in chunk
)["spec_tokens_details"]
self.assertIsInstance(single_details, dict)
def test_streaming_cached_tokens_details_emits_sglext(self):
"""Test that streaming completion responses emit cached token details in sglext."""
async def _mock_generate_with_cached_tokens_details(*args, **kwargs):
yield {
"text": "Cached response",
"meta_info": {
"id": "cmpl-cache-test",
"prompt_tokens": 10,
"completion_tokens": 2,
"cached_tokens": 6,
"cached_tokens_details": {
"device": 4,
"host": 1,
"storage": 1,
"storage_backend": "file",
},
"finish_reason": {"type": "stop", "matched": None},
"output_token_logprobs": None,
"output_top_logprobs": None,
},
"index": 0,
}
self.sc.tokenizer_manager.generate_request = (
_mock_generate_with_cached_tokens_details
)
req = CompletionRequest(
model="x",
prompt="Hello world",
max_tokens=100,
stream=True,
return_cached_tokens_details=True,
)
adapted_request, _ = self.sc._convert_to_internal_request(req)
async def run_stream():
chunks = []
async for chunk in self.sc._generate_completion_stream(
adapted_request, req, self.fastapi_request
):
chunks.append(chunk)
return chunks
loop = get_or_create_event_loop()
chunks = loop.run_until_complete(run_stream())
sglext_chunks = []
for chunk in chunks:
if not chunk.startswith("data: ") or chunk.strip() == "data: [DONE]":
continue
data = json.loads(chunk[len("data: ") :])
if "sglext" in data:
sglext_chunks.append(data)
self.assertEqual(len(sglext_chunks), 1)
self.assertEqual(sglext_chunks[0]["choices"], [])
self.assertEqual(
sglext_chunks[0]["sglext"]["cached_tokens_details"],
{
"device": 4,
"host": 1,
"storage": 1,
"storage_backend": "file",
},
)
if __name__ == "__main__":
unittest.main(verbosity=2)
Reference in New Issue View Git Blame Copy Permalink