Last of four; stacked on #38048. The record is the operator's input; the bags are what is in effect. A reader that takes the record and reads a field off it gets the input, which is the wrong one of the two whenever resolution decided something -- and the mistake is silent, because for most fields and most launches the two agree. Several of these files already read both ways, sometimes in the same expression: ```python get_tokenizer( get_serving().tokenizer_path, tokenizer_mode=server_args.tokenizer_mode, # the input, not the decision ... ) ``` Sixty-odd files convert. Record field reads in runtime code go from 199 to 11. Nine parameters that the conversion emptied are dropped along with the argument at every call site -- the dead-parameter ratchet is what names them. ### "Runs after its process publishes" is a per-entry-point claim Most converted reads sit in the serving and model-executor layers, which only exist after publication, or in the two subprocess entry points, which publish first thing. Three places are not like that, and they keep reading the record they were handed: - **`HttpServerEngineAdapter`** launches the server as a *child*. The parent resolves the record and never publishes, so the adapter's own reads -- the launch banner, the API key in its readiness loop, the TP width in `update_weights_from_tensor` -- are of `self.server_args`. A bag read here fails closed in a bare process, or answers for an unrelated engine in one that happens to have published. - **`serve_grpc`** reads its sidecar port before the integrated servicer builds the `Engine` that publishes. The comment above that line already said so and already bound `cfg = resolving_view(server_args)` for it; the sidecar port and the port it derives from read `cfg`. - **`initialize_dp_attention`** runs from callers whose publish is not guaranteed, so its one predicate stays on the resolution view. `ROLE_NAMESPACE_SETS["dp_controller"]` gains `observability` and `serving`, because the controller's metrics gate, tracing setup and worker-port broadcast now read those namespaces. Under `SGLANG_ROLE_NAMESPACES=enforce` that set is what the process may read, so a conversion that reaches a new namespace has to widen it in the same change. ## Three things worth a reviewer's attention **Eleven reads were `getattr(record, "field", default)`.** An AST scan for attribute access does not see those, so the census that said "43 readers" was counting the shape it could match rather than the thing it was after. `incremental_streaming_output` was read that way twice, and the transcription tests were the only reason it surfaced. **Not every record read is a bag read waiting to happen.** A multimodal processor's `base_gpu_id` is the instance's, not the process's: two engines in one process keep different ones, and `test_publishing_another_config_does_not_move_the_device` exists to say so. It stays on the record while `rl_on_policy_target` beside it moves. `RequestMetricsExporter` is the same shape -- it is handed the directory it writes to, and a test builds several with different ones. `configure_logger` is a third: 17 call sites, one of which passes an `argparse.Namespace`, so it is not a global-context reader at all. Those eleven remaining reads are the ones with a reason. **The fixtures move with the code.** Tests that hung config off a mock manager now publish a record, which is what the serving layer reads; where a test states a value it says so with `override_server_args` instead of assigning through the mock. `test_hisparse_unit` is the last of them: it stubbed a `server_args` onto a fake scheduler to say the decode radix cache was off, and the value it was standing in for is the published default, so the stub goes and the class publishes. ## Two things CI caught that a local sweep could not **`unittest.TestCase.enterContext` is Python 3.11+.** The converted fixtures used it at 18 sites; `requires-python` is `>=3.10` and CI runs 3.10, so every one of them raised `AttributeError` there while passing on a newer local interpreter. They call `enter_override(self, ...)` now -- a four-line helper in `sglang/test/test_utils.py` over the override's own `install()` / `restore()`. **A batched sweep cannot see a missing publish.** Three fixtures needed a published config and did not have one; each *passed* inside a shard where some other file had published, and failed when run alone. The affected cases are `test_serving_completions` (which set `incremental_streaming_output` on the mock manager's record, where nothing reads it now), `test_qwen3_vl_feature_materialization` (same shape for `mm_enable_dp_encoder`), and the two Qwen Rust tests -- whose fixture already carried the comment `# Non-auto: get_resolved_model_impl would choke on a SimpleNamespace` next to the `model_impl` it sets, which is exactly what happened once `get_mm_processor_cls` started reading that value from the bag. Its `publish` mirrors `model_impl` now, like the four fields it already mirrored. ## Verification A full registered-unit sweep (648 files) against this stack's merge-base: 19 failures on both sides, the same 19, none of them config. That sweep is what caught 23 failures the file-scoped runs missed -- and, later, that the narrower 139-file list did not even contain the files this change reaches. It is also what caught the `test_hisparse_unit` fixture above: the file passes inside a shard where something else published, and fails when it is run on its own, which is why every failing file is re-run alone before it is counted.
272 lines
10 KiB
Python
272 lines
10 KiB
Python
"""
|
|
Thin gRPC server wrapper — delegates to smg-grpc-servicer package.
|
|
|
|
A lightweight HTTP sidecar is started alongside the gRPC server to expose:
|
|
- /metrics (Prometheus, when --enable-metrics is set)
|
|
- /start_profile, /stop_profile (profiling control)
|
|
|
|
The sidecar is started on --smg-http-sidecar-port (default: --port + 1)
|
|
once the gRPC request manager is ready, regardless of whether --enable-metrics
|
|
is set.
|
|
"""
|
|
|
|
import inspect
|
|
import json
|
|
import logging
|
|
import time
|
|
|
|
from aiohttp import web
|
|
|
|
from sglang.srt.arg_groups.overrides import resolving_view
|
|
from sglang.srt.managers.io_struct import ProfileReq, ProfileReqType
|
|
from sglang.srt.utils.common import get_bool_env_var
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
|
|
async def _start_sidecar_server(host: str, port: int, app):
|
|
"""Start the aiohttp sidecar and return the runner for cleanup."""
|
|
runner = web.AppRunner(app)
|
|
await runner.setup()
|
|
try:
|
|
site = web.TCPSite(runner, host, port)
|
|
await site.start()
|
|
except BaseException:
|
|
await runner.cleanup()
|
|
raise
|
|
logger.info("HTTP sidecar server started on http://%s:%d", host, port)
|
|
return runner
|
|
|
|
|
|
def _add_metrics_routes(app):
|
|
"""Add Prometheus /metrics endpoint to the aiohttp app."""
|
|
from prometheus_client import (
|
|
CollectorRegistry,
|
|
multiprocess,
|
|
)
|
|
from prometheus_client.openmetrics.exposition import (
|
|
CONTENT_TYPE_LATEST,
|
|
generate_latest,
|
|
)
|
|
|
|
async def metrics_handler(request):
|
|
try:
|
|
registry = CollectorRegistry()
|
|
multiprocess.MultiProcessCollector(registry)
|
|
data = generate_latest(registry)
|
|
return web.Response(
|
|
body=data,
|
|
headers={"Content-Type": CONTENT_TYPE_LATEST},
|
|
)
|
|
except Exception:
|
|
logger.exception("Failed to generate Prometheus metrics")
|
|
return web.Response(status=500, text="Failed to generate metrics")
|
|
|
|
app.router.add_get("/metrics", metrics_handler)
|
|
|
|
|
|
def _check_communicator_results(results, action):
|
|
"""Return a web.Response error if results indicate failure, else None."""
|
|
if not results:
|
|
return web.Response(status=500, text="No response from scheduler\n")
|
|
failures = [r for r in results if not r.success]
|
|
if failures:
|
|
msgs = " | ".join(r.message for r in failures)
|
|
return web.Response(status=500, text=f"{action} failed: {msgs}\n")
|
|
return None
|
|
|
|
|
|
def _add_admin_routes(app, request_manager):
|
|
"""Add admin endpoints to the aiohttp app.
|
|
|
|
Endpoints: /start_profile, /stop_profile.
|
|
Business logic (request construction, env var handling, response interpretation)
|
|
lives here; request_manager only provides the transport to the scheduler.
|
|
"""
|
|
|
|
async def start_profile_handler(request):
|
|
try:
|
|
if request.content_length and request.content_length > 0:
|
|
try:
|
|
body = await request.json()
|
|
except json.JSONDecodeError as e:
|
|
return web.Response(
|
|
status=400,
|
|
text=f"Invalid JSON in request body: {e}",
|
|
)
|
|
else:
|
|
body = {}
|
|
|
|
# Build ProfileReq with env var overrides (same as tokenizer_communicator_mixin)
|
|
with_stack = body.get("with_stack")
|
|
env_with_stack = get_bool_env_var("SGLANG_PROFILE_WITH_STACK", "true")
|
|
with_stack = (with_stack is not False) and env_with_stack
|
|
record_shapes = body.get("record_shapes")
|
|
env_record_shapes = get_bool_env_var("SGLANG_PROFILE_RECORD_SHAPES", "true")
|
|
record_shapes = (record_shapes is not False) and env_record_shapes
|
|
|
|
req = ProfileReq(
|
|
req_type=ProfileReqType.START_PROFILE,
|
|
output_dir=body.get("output_dir"),
|
|
start_step=body.get("start_step"),
|
|
num_steps=body.get("num_steps"),
|
|
activities=body.get("activities"),
|
|
with_stack=with_stack,
|
|
record_shapes=record_shapes,
|
|
profile_by_stage=body.get("profile_by_stage", False),
|
|
profile_id=str(time.time()),
|
|
merge_profiles=body.get("merge_profiles", False),
|
|
profile_prefix=body.get("profile_prefix"),
|
|
profile_stages=body.get("profile_stages"),
|
|
)
|
|
results = await request_manager.send_communicator_req(
|
|
req, "profile_communicator", timeout=600.0
|
|
)
|
|
err = _check_communicator_results(results, "Start Profile")
|
|
if err:
|
|
return err
|
|
return web.Response(text="Start profiling.\n")
|
|
except Exception as e:
|
|
logger.exception("Failed to start profile")
|
|
return web.Response(
|
|
status=500,
|
|
text=f"Internal error: {type(e).__name__}. Check server logs.\n",
|
|
)
|
|
|
|
async def stop_profile_handler(request):
|
|
try:
|
|
req = ProfileReq(req_type=ProfileReqType.STOP_PROFILE)
|
|
results = await request_manager.send_communicator_req(
|
|
req, "profile_communicator", timeout=600.0
|
|
)
|
|
err = _check_communicator_results(results, "Stop profile")
|
|
if err:
|
|
return err
|
|
return web.Response(text="Stop profiling. This will take some time.\n")
|
|
except Exception as e:
|
|
logger.exception("Failed to stop profile")
|
|
return web.Response(
|
|
status=500,
|
|
text=f"Internal error: {type(e).__name__}. Check server logs.\n",
|
|
)
|
|
|
|
app.router.add_post("/start_profile", start_profile_handler)
|
|
app.router.add_post("/stop_profile", stop_profile_handler)
|
|
|
|
|
|
async def serve_grpc(server_args, model_info=None):
|
|
"""Start the standalone gRPC server with integrated scheduler."""
|
|
try:
|
|
from smg_grpc_servicer.sglang.server import serve_grpc as _serve_grpc
|
|
except ImportError as e:
|
|
raise ImportError(
|
|
"gRPC mode requires the smg-grpc-servicer package. "
|
|
"If not installed, run: pip install smg-grpc-servicer[sglang]. "
|
|
"If already installed, there may be a broken import due to a "
|
|
"version mismatch — see the chained exception above for details."
|
|
) from e
|
|
|
|
# The integrated servicer builds an `Engine`, which validates and publishes
|
|
# on its own. Validating here would run `check_server_args` twice, and the
|
|
# LoRA normalization is not idempotent -- the second pass sees the `LoRARef`
|
|
# objects the first one declared and rejects them. So this entry reads the
|
|
# declarations for what it needs before the engine exists.
|
|
cfg = resolving_view(server_args)
|
|
|
|
sidecar_app = web.Application()
|
|
sidecar_runner = None
|
|
sidecar_port = (
|
|
cfg.smg_http_sidecar_port
|
|
if cfg.smg_http_sidecar_port is not None
|
|
else cfg.port + 1
|
|
)
|
|
|
|
# Metrics setup: must set PROMETHEUS_MULTIPROC_DIR before scheduler
|
|
# processes import prometheus_client, since the env var is inherited
|
|
# at fork time.
|
|
if cfg.enable_metrics:
|
|
try:
|
|
from sglang.srt.observability.func_timer import enable_func_timer
|
|
from sglang.srt.utils import set_prometheus_multiproc_dir
|
|
|
|
set_prometheus_multiproc_dir()
|
|
enable_func_timer()
|
|
_add_metrics_routes(sidecar_app)
|
|
except Exception as e:
|
|
logger.error(
|
|
"Failed to set up metrics: %s. Continuing without metrics.",
|
|
e,
|
|
exc_info=True,
|
|
)
|
|
|
|
async def _on_request_manager_ready(request_manager, srv_args, sched_info):
|
|
nonlocal sidecar_runner
|
|
try:
|
|
_add_admin_routes(sidecar_app, request_manager)
|
|
except Exception as e:
|
|
logger.error(
|
|
"Failed to set up admin routes: %s. "
|
|
"Continuing without admin endpoints.",
|
|
e,
|
|
exc_info=True,
|
|
)
|
|
try:
|
|
sidecar_runner = await _start_sidecar_server(
|
|
cfg.host, sidecar_port, sidecar_app
|
|
)
|
|
except OSError as e:
|
|
logger.error(
|
|
"Failed to start HTTP sidecar server: %s. "
|
|
"Continuing without metrics/profile endpoints.",
|
|
e,
|
|
exc_info=True,
|
|
)
|
|
except Exception as e:
|
|
logger.error(
|
|
"Unexpected error starting HTTP sidecar server: %s. "
|
|
"Continuing without metrics/profile endpoints.",
|
|
e,
|
|
exc_info=True,
|
|
)
|
|
|
|
# Older smg-grpc-servicer releases (≤ 0.5.2) accept only (server_args,
|
|
# model_info) and reject the on_request_manager_ready hook. The hook is
|
|
# what calls _start_sidecar_server, so dropping the kwarg disables the
|
|
# entire HTTP sidecar (Prometheus /metrics and /start_profile +
|
|
# /stop_profile). Core gRPC serving still works without it.
|
|
serve_kwargs: dict = {}
|
|
sidecar_supported = (
|
|
"on_request_manager_ready" in inspect.signature(_serve_grpc).parameters
|
|
)
|
|
if sidecar_supported:
|
|
serve_kwargs["on_request_manager_ready"] = _on_request_manager_ready
|
|
elif cfg.enable_metrics:
|
|
# User explicitly asked for metrics but the installed servicer can't
|
|
# start the sidecar that serves them — fail loud rather than silently
|
|
# produce a server with no /metrics endpoint.
|
|
raise RuntimeError(
|
|
"--enable-metrics requires smg-grpc-servicer ≥ 0.5.3 (the version "
|
|
"that accepts 'on_request_manager_ready'); installed version "
|
|
"lacks the hook so the HTTP sidecar would never start. Upgrade "
|
|
"smg-grpc-servicer or remove --enable-metrics."
|
|
)
|
|
else:
|
|
logger.warning(
|
|
"Installed smg-grpc-servicer does not accept "
|
|
"'on_request_manager_ready'; HTTP sidecar disabled "
|
|
"(no /metrics, /start_profile, /stop_profile). "
|
|
"Upgrade smg-grpc-servicer to ≥ 0.5.3 to enable it."
|
|
)
|
|
|
|
try:
|
|
await _serve_grpc(server_args, model_info, **serve_kwargs)
|
|
finally:
|
|
if sidecar_runner is not None:
|
|
try:
|
|
await sidecar_runner.cleanup()
|
|
except Exception as e:
|
|
logger.exception(
|
|
"Failed to cleanly shut down HTTP sidecar server: %s",
|
|
e,
|
|
)
|