Files
sglang/python
minke.yu cb7988f849 observability: measure storage prefetch ready latency (hit/miss)
Under --hicache-storage-prefetch-policy wait_complete a request is not
admitted until its L3 lookup resolves, and a miss pays the same query round
trip as a hit. The scheduler only exposed a skip counter, so the actual wait
was unknown.

Add sglang:storage_prefetch_ready_seconds{dp_rank, result=hit|miss}:
  * stamp cache_request_handle on its first in-flight check,
  * observe at the moment check_prefetch_progress() turns usable, split by
    whether prefetch_loaded_tokens_by_reqid ended up > 0,
  * clear the stamp on revoke / abort / accounting discard.

Buckets cover 1 ms .. 30 s. Gated by SGLANG_BATCH_STAGE_TIMING (off by
default). The measurement is ~2 dict ops + one observe per request, not per
pass.

Tests: test/registered/unit/observability/test_batch_telemetry_stage_timing.py
adds the hit/miss split assertion (9 tests, all passing).
2026-11-04 17:26:41 +08:00
..