[router] Add request/TTFT/worker metrics + Grafana dashboard to experimental sgl-router (#27591)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-09 07:25:56 -07:00
parent 1368717248
commit badab6b136
10 changed files with 3042 additions and 48 deletions
@@ -0,0 +1,69 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 The SGLang Authors
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# sgl-router (experimental) monitoring
+
+Grafana dashboard for the experimental router's Prometheus metrics, exposed
+on `/metrics` (text/plain, version 0.0.4) on the router's serving port
+(default `30000`).
+
+## Files
+
+- `grafana-dashboard.json` — importable Grafana dashboard, **SGLang Router
+  (experimental)** (uid `sgl-router-experimental`).
+
+## Metrics covered
+
+The dashboard graphs every family the router emits:
+
+| Metric | Type | What it shows |
+|---|---|---|
+| `sgl_router_requests_total` | Counter | Dispatches by `worker_url`, `model_id`, `mode`, `outcome` |
+| `sgl_router_request_duration_seconds` | Histogram | End-to-end request latency by `model_id` |
+| `sgl_router_ttft_seconds` | Histogram | Time to first token (streaming) by `model_id` |
+| `sgl_router_responses_total` | Counter | Client-visible HTTP `status_code` |
+| `sgl_router_overlap_blocks` | Histogram | Cache-aware-zmq overlap blocks by `model_id` |
+| `sgl_router_active_load` | Gauge | Per-worker prefill-token / decode-block load |
+| `sgl_router_workers` | Gauge | Registered worker count by `mode` |
+| `sgl_router_worker_health` | Gauge | Per-worker health (1=breaker admits, 0=open) |
+| `sgl_router_worker_cb_state` | Gauge | Per-worker circuit breaker state (0=closed, 1=open, 2=half_open) |
+| `sgl_router_worker_inflight_requests` | Gauge | In-flight requests per worker |
+| `sgl_router_stale_requests_total` | Counter | Stale-request cancellations |
+| `sgl_router_decode_affinity_total` | Counter | PD decode-affinity outcomes |
+| `sgl_router_sticky_total` | Counter | Sticky-session selection outcomes |
+
+The `sgl_router_workers` / `sgl_router_worker_*` gauges are sampled from the
+live worker registry on every scrape, so a removed worker stops emitting
+series immediately rather than leaving a stale value.
+
+## Prometheus scrape config
+
+Point Prometheus at the router's `/metrics` endpoint:
+
+```yaml
+scrape_configs:
+  - job_name: sgl-router
+    metrics_path: /metrics
+    static_configs:
+      - targets:
+          - '127.0.0.1:30000'   # router host:port
+```
+
+## Import into Grafana
+
+1. **Dashboards → New → Import**.
+2. Upload `grafana-dashboard.json` (or paste its contents).
+3. When prompted, select your Prometheus data source for the `Datasource`
+   variable. The dashboard uses a templated data source, so it imports into
+   any Grafana without editing the JSON.
+
+The top bar exposes `model_id` and `worker_url` template variables (both
+default to *All*) to scope the panels.
+
+## Regenerating
+
+The JSON is generated programmatically to keep the ~20 panels consistent. If
+the metric surface changes, update the generator and overwrite the JSON
+rather than hand-editing — hand-edits drift from the panel conventions.