minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[Router] Log every request at one site; derive its outcome from the final status (3/3) (#39465)

Browse Source 
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
This commit is contained in:
Kangyan-Zhou
2026-09-19 03:11:00 +08:00
committed by GitHub
co-authored by Kangyan Zhou Claude Opus 5 Shangming Cai
parent 248c202b46
commit aed3fb1cdd
10 changed files with 923 additions and 95 deletions
Download Patch File Download Diff File Expand all files Collapse all files
experimental/sgl-router/monitoring/README.md
+41 -1
View File
@@ -23,7 +23,7 @@ Families the router emits. The dashboard graphs all of them except the
|---|---|---|
| `sgl_router_requests_total` | Counter | **Edge intake** — every request received at the router HTTP boundary, by `route`, `method`, counted before worker dispatch (true intake) |
| `sgl_router_responses_total` | Counter | **Edge responses** — every response returned, by `route`, `method`, `status_code` (incl. early-exit 400/413/503). `requests_total - responses_total` = received-but-not-answered |
| `sgl_router_worker_requests_total` | Counter | Per-worker **dispatches** by `worker_url`, `model_id`, `mode`, `outcome` (recorded after dispatch; blind to pre-dispatch drops) |
| `sgl_router_worker_requests_total` | Counter | Per-worker **dispatches** by `worker_url`, `model_id`, `mode`, `outcome` (recorded after dispatch; blind to pre-dispatch drops). See [Dispatch outcomes](#dispatch-outcomes) |
| `sgl_router_request_duration_seconds` | Histogram | End-to-end request latency by `model_id` |
| `sgl_router_ttft_seconds` | Histogram | Time to first token (streaming) by `model_id` |
| `sgl_router_stream_outcome_total` | Counter | Streaming outcomes by `worker_url`, `model_id`, and `outcome` (`ok`, `stream_error_event`, `upstream_error`, or `client_disconnect`). Counts committed 2xx streams only — non-2xx responses are counted by status in `responses_total` |
@@ -103,3 +103,43 @@ default to *All*) to scope the panels.
The JSON is generated programmatically to keep the ~20 panels consistent. If
the metric surface changes, update the generator and overwrite the JSON
rather than hand-editing — hand-edits drift from the panel conventions.
## Dispatch outcomes
`sgl_router_worker_requests_total{outcome}` is derived from the status the
client saw, not from whether the router's internal dispatch returned `Ok` — a
worker error the router forwards is a successful *proxy* operation and a failed
*request*.
| `outcome` | Source | Counts as a worker fault? |
|---|---|---|
| `success` | 2xx | no |
| `client_error` | 4xx except 429 | no — the caller sent something invalid |
| `backpressure` | 429, 503 | no — responsive but at capacity |
| `error` | 5xx except 503, plus transport failures, timeouts and incomplete bodies | **yes** |
| `cancelled` | the router's own stale-request deadline | no |
`error` is the only bucket that means *this worker failed*, which is why the
Error-ratio panel uses it alone. The split matters during an incident: a
saturated fleet answering with its own queue-full 503s registers as
`backpressure`, and the circuit breaker likewise declines to open on those
statuses — so the two agree, and the error ratio keeps pointing at genuine
faults instead of pegging at 100% exactly when it is being read.
A hung worker surfaces as `error` (the router's upstream timeout), *not* as
`cancelled`. Only the stale-request deadline produces `cancelled`;
`sgl_router_stale_requests_total{outcome="expired"}` counts the same events.
## Access log
The router emits one `http_request` event per request from a single middleware,
so requests that never reach a handler (a body-limit 413, an unrouted 404, a
panic-500) are logged too. Fields: `pod_id`, `request_id`, `method`, `path`,
`status`, `outcome`, `worker`, `model`, `stream`, `latency_ms`.
`worker` and `model` are empty when the request was rejected before dispatch or
hit a route that does not dispatch — that is normal, not a gap. Successful infra
polls (`/healthz`, `/readyz`, `/metrics`) log at DEBUG so they do not bury real
traffic; a *failing* probe keeps the INFO line. For a stream the line is written
when the response head is ready, so `status=200` there does not mean the stream
finished — `sgl_router_stream_outcome_total` carries that.