Files
sglang/python
minke.yu 7f42a0806a observability: per-rank request counter for P/D RPS
The frontend sglang:num_requests_total has no dp_rank label (verified from the
series registry), so per-rank RPS was not derivable from any existing metric.

Add sglang:batch_requests_total{phase,dp_rank}, incremented by the scheduler
process that actually completed the request:

  phase=prefill  prefill compute finished and the request entered the PD
                 transfer queue (not a guarantee that the transfer succeeded)
  phase=decode   request finished decoding

Only the canonical scheduler rank reports (attn_tp_rank==0, attn_cp_rank==0,
last PP stage) — every TP/CP rank runs a scheduler and sees the same request,
so an ungated counter would be multiplied by TP*CP in the multiprocess
registry. Both hooks count each request exactly once: the decode path counts
after _handle_finish_state_updated_req and on the sampling-mask abort branch,
skipping rows that finished on an earlier step; the prefill path counts after
the request enters the disagg inflight queue (aborted/retried requests never
reach it). The per-request finished() check is skipped entirely when
SGLANG_BATCH_TELEMETRY is off.
2026-11-12 10:30:56 +08:00
..