Files
sglang/python/sglang/srt/entrypoints/grpc_server.py
T
Cheng Wan b99175dc7d [Config] Round 6.4: the runtime reads the bags, not the record (#38049)
Last of four; stacked on #38048.

The record is the operator's input; the bags are what is in effect. A reader
that takes the record and reads a field off it gets the input, which is the
wrong one of the two whenever resolution decided something -- and the mistake is
silent, because for most fields and most launches the two agree. Several of
these files already read both ways, sometimes in the same expression:

```python
get_tokenizer(
    get_serving().tokenizer_path,
    tokenizer_mode=server_args.tokenizer_mode,   # the input, not the decision
    ...
)
```

Sixty-odd files convert. Record field reads in runtime code go from 199 to 11.
Nine parameters that the conversion emptied are dropped along with the argument
at every call site -- the dead-parameter ratchet is what names them.

### "Runs after its process publishes" is a per-entry-point claim

Most converted reads sit in the serving and model-executor layers, which only
exist after publication, or in the two subprocess entry points, which publish
first thing. Three places are not like that, and they keep reading the record
they were handed:

- **`HttpServerEngineAdapter`** launches the server as a *child*. The parent
  resolves the record and never publishes, so the adapter's own reads -- the
  launch banner, the API key in its readiness loop, the TP width in
  `update_weights_from_tensor` -- are of `self.server_args`. A bag read here
  fails closed in a bare process, or answers for an unrelated engine in one that
  happens to have published.
- **`serve_grpc`** reads its sidecar port before the integrated servicer builds
  the `Engine` that publishes. The comment above that line already said so and
  already bound `cfg = resolving_view(server_args)` for it; the sidecar port and
  the port it derives from read `cfg`.
- **`initialize_dp_attention`** runs from callers whose publish is not
  guaranteed, so its one predicate stays on the resolution view.

`ROLE_NAMESPACE_SETS["dp_controller"]` gains `observability` and `serving`,
because the controller's metrics gate, tracing setup and worker-port broadcast
now read those namespaces. Under `SGLANG_ROLE_NAMESPACES=enforce` that set is
what the process may read, so a conversion that reaches a new namespace has to
widen it in the same change.

## Three things worth a reviewer's attention

**Eleven reads were `getattr(record, "field", default)`.** An AST scan for
attribute access does not see those, so the census that said "43 readers" was
counting the shape it could match rather than the thing it was after.
`incremental_streaming_output` was read that way twice, and the transcription
tests were the only reason it surfaced.

**Not every record read is a bag read waiting to happen.** A multimodal
processor's `base_gpu_id` is the instance's, not the process's: two engines in
one process keep different ones, and
`test_publishing_another_config_does_not_move_the_device` exists to say so. It
stays on the record while `rl_on_policy_target` beside it moves.
`RequestMetricsExporter` is the same shape -- it is handed the directory it
writes to, and a test builds several with different ones. `configure_logger` is
a third: 17 call sites, one of which passes an `argparse.Namespace`, so it is
not a global-context reader at all. Those eleven remaining reads are the ones
with a reason.

**The fixtures move with the code.** Tests that hung config off a mock manager
now publish a record, which is what the serving layer reads; where a test states
a value it says so with `override_server_args` instead of assigning through the
mock. `test_hisparse_unit` is the last of them: it stubbed a `server_args` onto
a fake scheduler to say the decode radix cache was off, and the value it was
standing in for is the published default, so the stub goes and the class
publishes.

## Two things CI caught that a local sweep could not

**`unittest.TestCase.enterContext` is Python 3.11+.** The converted fixtures used
it at 18 sites; `requires-python` is `>=3.10` and CI runs 3.10, so every one of
them raised `AttributeError` there while passing on a newer local interpreter.
They call `enter_override(self, ...)` now -- a four-line helper in
`sglang/test/test_utils.py` over the override's own `install()` / `restore()`.

**A batched sweep cannot see a missing publish.** Three fixtures needed a
published config and did not have one; each *passed* inside a shard where some
other file had published, and failed when run alone. The affected cases are
`test_serving_completions` (which set `incremental_streaming_output` on the mock
manager's record, where nothing reads it now), `test_qwen3_vl_feature_materialization`
(same shape for `mm_enable_dp_encoder`), and the two Qwen Rust tests -- whose
fixture already carried the comment `# Non-auto: get_resolved_model_impl would
choke on a SimpleNamespace` next to the `model_impl` it sets, which is exactly
what happened once `get_mm_processor_cls` started reading that value from the
bag. Its `publish` mirrors `model_impl` now, like the four fields it already
mirrored.

## Verification

A full registered-unit sweep (648 files) against this stack's merge-base:
19 failures on both sides, the same 19, none of them config. That sweep is what
caught 23 failures the file-scoped runs missed -- and, later, that the narrower
139-file list did not even contain the files this change reaches. It is also
what caught the `test_hisparse_unit` fixture above: the file passes inside a
shard where something else published, and fails when it is run on its own,
which is why every failing file is re-run alone before it is counted.
2026-09-06 21:41:46 -07:00

272 lines
10 KiB
Python

"""
Thin gRPC server wrapper — delegates to smg-grpc-servicer package.
A lightweight HTTP sidecar is started alongside the gRPC server to expose:
- /metrics (Prometheus, when --enable-metrics is set)
- /start_profile, /stop_profile (profiling control)
The sidecar is started on --smg-http-sidecar-port (default: --port + 1)
once the gRPC request manager is ready, regardless of whether --enable-metrics
is set.
"""
import inspect
import json
import logging
import time
from aiohttp import web
from sglang.srt.arg_groups.overrides import resolving_view
from sglang.srt.managers.io_struct import ProfileReq, ProfileReqType
from sglang.srt.utils.common import get_bool_env_var
logger = logging.getLogger(__name__)
async def _start_sidecar_server(host: str, port: int, app):
"""Start the aiohttp sidecar and return the runner for cleanup."""
runner = web.AppRunner(app)
await runner.setup()
try:
site = web.TCPSite(runner, host, port)
await site.start()
except BaseException:
await runner.cleanup()
raise
logger.info("HTTP sidecar server started on http://%s:%d", host, port)
return runner
def _add_metrics_routes(app):
"""Add Prometheus /metrics endpoint to the aiohttp app."""
from prometheus_client import (
CollectorRegistry,
multiprocess,
)
from prometheus_client.openmetrics.exposition import (
CONTENT_TYPE_LATEST,
generate_latest,
)
async def metrics_handler(request):
try:
registry = CollectorRegistry()
multiprocess.MultiProcessCollector(registry)
data = generate_latest(registry)
return web.Response(
body=data,
headers={"Content-Type": CONTENT_TYPE_LATEST},
)
except Exception:
logger.exception("Failed to generate Prometheus metrics")
return web.Response(status=500, text="Failed to generate metrics")
app.router.add_get("/metrics", metrics_handler)
def _check_communicator_results(results, action):
"""Return a web.Response error if results indicate failure, else None."""
if not results:
return web.Response(status=500, text="No response from scheduler\n")
failures = [r for r in results if not r.success]
if failures:
msgs = " | ".join(r.message for r in failures)
return web.Response(status=500, text=f"{action} failed: {msgs}\n")
return None
def _add_admin_routes(app, request_manager):
"""Add admin endpoints to the aiohttp app.
Endpoints: /start_profile, /stop_profile.
Business logic (request construction, env var handling, response interpretation)
lives here; request_manager only provides the transport to the scheduler.
"""
async def start_profile_handler(request):
try:
if request.content_length and request.content_length > 0:
try:
body = await request.json()
except json.JSONDecodeError as e:
return web.Response(
status=400,
text=f"Invalid JSON in request body: {e}",
)
else:
body = {}
# Build ProfileReq with env var overrides (same as tokenizer_communicator_mixin)
with_stack = body.get("with_stack")
env_with_stack = get_bool_env_var("SGLANG_PROFILE_WITH_STACK", "true")
with_stack = (with_stack is not False) and env_with_stack
record_shapes = body.get("record_shapes")
env_record_shapes = get_bool_env_var("SGLANG_PROFILE_RECORD_SHAPES", "true")
record_shapes = (record_shapes is not False) and env_record_shapes
req = ProfileReq(
req_type=ProfileReqType.START_PROFILE,
output_dir=body.get("output_dir"),
start_step=body.get("start_step"),
num_steps=body.get("num_steps"),
activities=body.get("activities"),
with_stack=with_stack,
record_shapes=record_shapes,
profile_by_stage=body.get("profile_by_stage", False),
profile_id=str(time.time()),
merge_profiles=body.get("merge_profiles", False),
profile_prefix=body.get("profile_prefix"),
profile_stages=body.get("profile_stages"),
)
results = await request_manager.send_communicator_req(
req, "profile_communicator", timeout=600.0
)
err = _check_communicator_results(results, "Start Profile")
if err:
return err
return web.Response(text="Start profiling.\n")
except Exception as e:
logger.exception("Failed to start profile")
return web.Response(
status=500,
text=f"Internal error: {type(e).__name__}. Check server logs.\n",
)
async def stop_profile_handler(request):
try:
req = ProfileReq(req_type=ProfileReqType.STOP_PROFILE)
results = await request_manager.send_communicator_req(
req, "profile_communicator", timeout=600.0
)
err = _check_communicator_results(results, "Stop profile")
if err:
return err
return web.Response(text="Stop profiling. This will take some time.\n")
except Exception as e:
logger.exception("Failed to stop profile")
return web.Response(
status=500,
text=f"Internal error: {type(e).__name__}. Check server logs.\n",
)
app.router.add_post("/start_profile", start_profile_handler)
app.router.add_post("/stop_profile", stop_profile_handler)
async def serve_grpc(server_args, model_info=None):
"""Start the standalone gRPC server with integrated scheduler."""
try:
from smg_grpc_servicer.sglang.server import serve_grpc as _serve_grpc
except ImportError as e:
raise ImportError(
"gRPC mode requires the smg-grpc-servicer package. "
"If not installed, run: pip install smg-grpc-servicer[sglang]. "
"If already installed, there may be a broken import due to a "
"version mismatch — see the chained exception above for details."
) from e
# The integrated servicer builds an `Engine`, which validates and publishes
# on its own. Validating here would run `check_server_args` twice, and the
# LoRA normalization is not idempotent -- the second pass sees the `LoRARef`
# objects the first one declared and rejects them. So this entry reads the
# declarations for what it needs before the engine exists.
cfg = resolving_view(server_args)
sidecar_app = web.Application()
sidecar_runner = None
sidecar_port = (
cfg.smg_http_sidecar_port
if cfg.smg_http_sidecar_port is not None
else cfg.port + 1
)
# Metrics setup: must set PROMETHEUS_MULTIPROC_DIR before scheduler
# processes import prometheus_client, since the env var is inherited
# at fork time.
if cfg.enable_metrics:
try:
from sglang.srt.observability.func_timer import enable_func_timer
from sglang.srt.utils import set_prometheus_multiproc_dir
set_prometheus_multiproc_dir()
enable_func_timer()
_add_metrics_routes(sidecar_app)
except Exception as e:
logger.error(
"Failed to set up metrics: %s. Continuing without metrics.",
e,
exc_info=True,
)
async def _on_request_manager_ready(request_manager, srv_args, sched_info):
nonlocal sidecar_runner
try:
_add_admin_routes(sidecar_app, request_manager)
except Exception as e:
logger.error(
"Failed to set up admin routes: %s. "
"Continuing without admin endpoints.",
e,
exc_info=True,
)
try:
sidecar_runner = await _start_sidecar_server(
cfg.host, sidecar_port, sidecar_app
)
except OSError as e:
logger.error(
"Failed to start HTTP sidecar server: %s. "
"Continuing without metrics/profile endpoints.",
e,
exc_info=True,
)
except Exception as e:
logger.error(
"Unexpected error starting HTTP sidecar server: %s. "
"Continuing without metrics/profile endpoints.",
e,
exc_info=True,
)
# Older smg-grpc-servicer releases (≤ 0.5.2) accept only (server_args,
# model_info) and reject the on_request_manager_ready hook. The hook is
# what calls _start_sidecar_server, so dropping the kwarg disables the
# entire HTTP sidecar (Prometheus /metrics and /start_profile +
# /stop_profile). Core gRPC serving still works without it.
serve_kwargs: dict = {}
sidecar_supported = (
"on_request_manager_ready" in inspect.signature(_serve_grpc).parameters
)
if sidecar_supported:
serve_kwargs["on_request_manager_ready"] = _on_request_manager_ready
elif cfg.enable_metrics:
# User explicitly asked for metrics but the installed servicer can't
# start the sidecar that serves them — fail loud rather than silently
# produce a server with no /metrics endpoint.
raise RuntimeError(
"--enable-metrics requires smg-grpc-servicer ≥ 0.5.3 (the version "
"that accepts 'on_request_manager_ready'); installed version "
"lacks the hook so the HTTP sidecar would never start. Upgrade "
"smg-grpc-servicer or remove --enable-metrics."
)
else:
logger.warning(
"Installed smg-grpc-servicer does not accept "
"'on_request_manager_ready'; HTTP sidecar disabled "
"(no /metrics, /start_profile, /stop_profile). "
"Upgrade smg-grpc-servicer to ≥ 0.5.3 to enable it."
)
try:
await _serve_grpc(server_args, model_info, **serve_kwargs)
finally:
if sidecar_runner is not None:
try:
await sidecar_runner.cleanup()
except Exception as e:
logger.exception(
"Failed to cleanly shut down HTTP sidecar server: %s",
e,
)