Runtime accounting on the prefill service (64 x 14.6k-token requests,
8-way concurrency) showed the non-forward wall time is dominated by the
scheduler receive path, not by radix match:
forward (extend+idle) 58.2% | get_next 6.9% | recv_requests 25.6%
process_batch_result 1.8% | run_batch overhead 0.9% | other ~6%
The existing pass span only covers get_next, so it cannot explain that
25.6%. Split it with the same opt-in gate:
* sglang:recv_requests_seconds{dp_rank, part} (total/pull/broadcast/
unwrap/mm), 5 new series per rank
* sglang:recv_requests_msgs_total, sglang:recv_requests_prompt_tokens_total
* OTLP span scheduler.recv_requests with the same breakdown, sampled
deterministically by a per-process counter (SGLANG_RECV_SPAN_SAMPLE,
default 0.05; SGLANG_RECV_SPAN=0 disables the span only)
Note on scope: tokenization itself does NOT run in the scheduler. The
tokenizer workers (--tokenizer-worker-num) produce TokenizedGenerateReqInput
and the scheduler receives the already-tokenized ids. What this measures is
ZMQ pull + decode, the cross-rank broadcast of the pickled objects, and
unwrap_pickle_fields() of the token-id buffers -- which is why large prompts
make recv_requests expensive.
Tests: test/registered/unit/observability/test_batch_telemetry_stage_timing.py
now also covers the recv histogram parts, counters, zero-message passes and
deterministic recv-span sampling (8 tests total, all passing).