Motivation: on the PD prefill node the wall-clock budget is dominated by
get_next_batch_to_run (~50% of the hour on 2026-11-02 16:00-17:00), and
forward_overlap=partial is only 21-87 s/h, i.e. it does not overlap the GPU
forward. We know the total but not which segment inside it costs: radix
match, PrefillAdder admission, the HiCache event collective, the bootstrap
all-reduce, or the DP metadata gather.
Measurement design (one sample per scheduling pass, including passes that
produce no batch; see handoff/prefill-batch-stage-timing-20261103.md v2/v3):
L1 7 per-pass histograms:
sglang:batch_prefill_{get_next,match,adder,hicache_events,
bootstrap,mlp_sync,other}_seconds
other = total - (match + adder + hicache + bootstrap + mlp_sync),
clamped to >= 0. Nested stages are not double counted.
L3 5 per-pass counters:
candidates / skipped_prefetch / admitted / resumed_chunk / new_chunk
L2 a dedicated per-pass span scheduler.prefill.pass (not the batch span),
emitted from end_pass() on every pass. Its sampling decision is a pure
function of mlp_sync_seq, so a sampled gather round is sampled on all
DP ranks. Idle ranks and ranks without a local batch still emit, which
is exactly what the cross-rank skew analysis needs. No links.
Hooks are installed in both scheduling loops:
* get_next_disagg_prefill_batch_to_run() for --disaggregation-mode prefill
(hicache / bootstrap / mlp_sync segments), and
* get_next_batch_to_run() for standalone prefill (hicache / mlp_sync),
so the instrumentation can be validated on ds41-prefill-dp4 without a decode
peer. match/adder/counters live in the shared get_new_batch_prefill path.
Semantics:
* match times only init_next_round_input; check_prefetch_progress counts as
skipped_prefetch, never as match.
* bootstrap and hicache_events are cross-rank sync points measured at the
outer call, not attributed to local CPU work.
* dp_gather_ms comes from the existing MLPSyncBatchInfo.sync_wait_seconds;
sglang:dp_attention_sync_wait_seconds is reused, no duplicate metric.
The value is cleared before each outer call and mlp_sync_seq only
increments when a gather actually ran, so stale values cannot leak.
* resumed vs new chunked requests are captured separately: had_chunked is
recorded at pass start (before process_prefill_chunk/admission can
consume the last chunk) and new_chunk from adder.new_chunked_req. This
fixes both the first-chunk and last-chunk mislabels.
Opt-in:
* SGLANG_BATCH_STAGE_TIMING=1 (default 0);
SGLANG_BATCH_PASS_SPAN=1 / SGLANG_BATCH_PASS_SPAN_SAMPLE=0.05.
* When off every hook is one module-level bool check; no logging, no CUDA
calls, no new collectives.
Tests: test/registered/unit/observability/test_batch_telemetry_stage_timing.py
covers first/middle/last chunk labelling, all-prefetch-skip passes without a
batch, stale-gather clearing, rank-deterministic pass-span sampling, and the
disabled no-op path. py_compile passes on all touched files.