Implements M2 of handoff/prefill-cpu-kv-overlap-design-20261106.md, behind
SGLANG_TRACE_MOONCAKE_DETAIL=1 (default off; reuses the existing trace stack).
- TransferKVChunk gains optional observation fields: chunk_seq, loop_seq,
send_kind, enqueue/last_enqueue monotonic+wall stamps, queue_wait_accum_ns,
queue_id, requeue_count, num_pages, approx_bytes, observe
- chunk_seq and enqueue timestamps are assigned at the real queue.put()
point (after every early return), so skipped paths cannot leave holes
- Mooncake transfer worker measures queue wait, source-event wait, indices
wait, KV/state/aux transfer, worker busy and residual; emits one
mooncake.kv_chunk span per attempt (+ queue/source/indices wait children)
and updates a per-room join summary
- request completion emits kv.request_join with chunk_count,
last_chunk_queue_wait/worker, final_exposed_ms, completion_poll_ms,
request_join_ms and first-enqueue-to-completion. First enqueue is stamped
by the Mooncake manager at the real queue.put(); no ReqTimeStats field is
added and the legacy queue_entry semantics are unchanged.
- nixl/mori add_transfer_request accept the observation kwarg (no-op there)
No scheduling/transfer semantics change; queue wait is excluded from worker
busy, and worker_busy excludes source/indices event waits. Unit tests: test_mooncake_chunk_detail.py (15 cases, including the
abort/worker interleaving and attempt-vs-total queue wait windows).
Implements M1 of handoff/prefill-cpu-kv-overlap-design-20261106.md:
per-loop detail phases behind SGLANG_TRACE_LOOP_DETAIL=1 (requires
SGLANG_BATCH_STAGE_TIMING=1 + SGLANG_TRACE_LOOP_SPAN=1, only sampled loops).
- run_batch: early_send_host, resolve_inputs, model_forward_host,
copy_result_submit (+ run_residual_ms)
- process_result: copy_done_wait, auxiliary_and_logprob, per_request_result,
kv_state_payload/page_indices/sender_send by send_kind, report_stats
(+ process_residual_ms)
- war_barrier is measured as its own loop phase (outside run_batch's span)
- repeated/per-request phases are summed attributes; single phases are child
spans, so span count never grows with n_reqs/n_segments
- DP-skew join fields: pass_end_wall_ns, arrival_est_wall_ns = pass_end -
mlp_sync, previous-loop phase summary (sglang.prev_*, published only for
consecutive loop_seq; prev_detail_missing marks a gap)
No CUDA syncs added; when disabled every hook is a module-level bool check.
Unit tests: test_loop_detail_phases.py (10 cases).
Review follow-up. match_kv_cache also runs on waiting-queue probes
(calc_priority), PD decode _match_prefix_and_lock and HiCache rematches.
Emitting the counter there counted the same gap once per scheduling round
and attributed "re-prefilled" tokens to decode engines.
- match_kv_cache now only records req.full_kv_hit_length (plus the match-time
gap); it no longer emits.
- the metric is emitted once in PrefillAdder._commit_prefill_admission, where
the accepted prefix is final after any host load-back.
- rename the request counter to swa_gate_reject_admissions_total and document
that a retracted and re-admitted request is counted again (it is re-prefilled).
- pre-create zero-valued children so gate-off deployments expose the series
before the first rejection.
- tests: repeated match does not emit, admission emits once, a retry counts
again, and host load-back closes the gap.
Not runtime verified (no GPU run on this build).
- prefetch_io_aux_func: allow up to SGLANG_HICACHE_PREFETCH_IO_THREADS
transfers in flight (default 1 = legacy behaviour); PrefetchAck is still
emitted strictly FIFO so the per-rank ack stream (and the TP all_reduce in
prefetch_sync_thread_func) is unchanged.
- backup_thread_func: same pipelining for L2->L3 writes via
SGLANG_HICACHE_BACKUP_IO_THREADS.
- SGLANG_HICACHE_TIMING=1: emit [HC_TIMING] prefetch/backup queue, inflight,
p50/p95 per 100 ops (the prefetch/backup lanes were previously single
threaded; this is what caps L3<->L2 at ~1 op at a time).
The unified-cache SWA component gates prefix matching on a live sliding
window: when the window is missing, the match stops before an otherwise
available Full-KV prefix. The existing hit counters do not show the tokens
lost to that gate, which makes the gate-on/gate-off trade-off invisible.
Record the gap between the longest Full-KV match and the accepted prefix
(full_kv_hit_length - (device_prefix_len + host_hit_length)) on the Req and
export it as:
sglang:swa_gate_reject_tokens_total
sglang:swa_gate_reject_reqs_total
This is the token volume that will be re-prefilled because a component
validator (SWA window; Mamba where present) rejected the match. With the
gate enabled it measures the hit length lost to a missing window; with the
gate disabled it only captures the host-only / load-back-gap fallback.
Includes unit tests for the gap and the host-hit case. Not runtime verified
yet (no GPU run on this build).
Adds an opt-in scheduler.loop span (SGLANG_TRACE_LOOP_SPAN) with recv /
get_next / run_batch / process_result / kv_send children. The span is only
created for sampled loops that have pending work; fully idle loops stay in
the counters so the 714 pass/s idle spin never floods Jaeger.
No-batch accounting is always on with SGLANG_BATCH_STAGE_TIMING:
- sglang:scheduler_no_batch_{seconds,total}{dp_rank,reason,forward_overlap}
- sglang:scheduler_dp_lockstep_idle_seconds_total{dp_rank,forward_overlap}
with the design's reason priority (result_backpressure, deferred_input,
waiting_bootstrap, waiting_prefetch, admission_blocked, waiting_pd,
fully_idle, no_ready_other) and local_batch_kind/global_outcome from the DP
sync. DP lockstep idle is counted separately from pure no-batch wait.
loop gpu_ms reuses the existing DeviceTimer intervals (no new CUDA events);
the forward.gpu lane is optional and approximate. Pass/recv spans become
children of the loop span when it is active.
Tests: test_loop_span_accounting (5) and the existing stage-timing suite (8)
pass.
_handle_prefetch_result() retires the operation with a direct
del ongoing_prefetch[request] instead of _retire_ongoing_prefetch(), so
the ready-latency hook only ever saw miss/revoke paths: with a cache-hit
workload storage_prefetch_ready_seconds{result="hit"} stayed at 0 even
though storage_prefetch_hit_tokens_total was moving ~312k tok/s.
Observe right after the del, classifying by loaded_from_storage.
The first version stamped the request on its first in-flight
check_prefetch_progress() call. A fast L3 query can be drained by
_process_hicache_events before that check runs (the op is already gone
from ongoing_prefetch), so those requests produced no sample -- exactly
the common case.
Stamp at issue time in prefetch_from_storage() and observe once at
_retire_ongoing_prefetch(), classifying by operation.storage_hit_count.
Retire is the single exit for hit completion, below-threshold revoke,
anchor loss and host-capacity fallback, so hit and miss are both covered.
No behavior change; the hook is still behind SGLANG_BATCH_STAGE_TIMING.
Under --hicache-storage-prefetch-policy wait_complete a request is not
admitted until its L3 lookup resolves, and a miss pays the same query round
trip as a hit. The scheduler only exposed a skip counter, so the actual wait
was unknown.
Add sglang:storage_prefetch_ready_seconds{dp_rank, result=hit|miss}:
* stamp cache_request_handle on its first in-flight check,
* observe at the moment check_prefetch_progress() turns usable, split by
whether prefetch_loaded_tokens_by_reqid ended up > 0,
* clear the stamp on revoke / abort / accounting discard.
Buckets cover 1 ms .. 30 s. Gated by SGLANG_BATCH_STAGE_TIMING (off by
default). The measurement is ~2 dict ops + one observe per request, not per
pass.
Tests: test/registered/unit/observability/test_batch_telemetry_stage_timing.py
adds the hit/miss split assertion (9 tests, all passing).
Runtime accounting on the prefill service (64 x 14.6k-token requests,
8-way concurrency) showed the non-forward wall time is dominated by the
scheduler receive path, not by radix match:
forward (extend+idle) 58.2% | get_next 6.9% | recv_requests 25.6%
process_batch_result 1.8% | run_batch overhead 0.9% | other ~6%
The existing pass span only covers get_next, so it cannot explain that
25.6%. Split it with the same opt-in gate:
* sglang:recv_requests_seconds{dp_rank, part} (total/pull/broadcast/
unwrap/mm), 5 new series per rank
* sglang:recv_requests_msgs_total, sglang:recv_requests_prompt_tokens_total
* OTLP span scheduler.recv_requests with the same breakdown, sampled
deterministically by a per-process counter (SGLANG_RECV_SPAN_SAMPLE,
default 0.05; SGLANG_RECV_SPAN=0 disables the span only)
Note on scope: tokenization itself does NOT run in the scheduler. The
tokenizer workers (--tokenizer-worker-num) produce TokenizedGenerateReqInput
and the scheduler receives the already-tokenized ids. What this measures is
ZMQ pull + decode, the cross-rank broadcast of the pickled objects, and
unwrap_pickle_fields() of the token-id buffers -- which is why large prompts
make recv_requests expensive.
Tests: test/registered/unit/observability/test_batch_telemetry_stage_timing.py
now also covers the recv histogram parts, counters, zero-message passes and
deterministic recv-span sampling (8 tests total, all passing).
Motivation: on the PD prefill node the wall-clock budget is dominated by
get_next_batch_to_run (~50% of the hour on 2026-11-02 16:00-17:00), and
forward_overlap=partial is only 21-87 s/h, i.e. it does not overlap the GPU
forward. We know the total but not which segment inside it costs: radix
match, PrefillAdder admission, the HiCache event collective, the bootstrap
all-reduce, or the DP metadata gather.
Measurement design (one sample per scheduling pass, including passes that
produce no batch; see handoff/prefill-batch-stage-timing-20261103.md v2/v3):
L1 7 per-pass histograms:
sglang:batch_prefill_{get_next,match,adder,hicache_events,
bootstrap,mlp_sync,other}_seconds
other = total - (match + adder + hicache + bootstrap + mlp_sync),
clamped to >= 0. Nested stages are not double counted.
L3 5 per-pass counters:
candidates / skipped_prefetch / admitted / resumed_chunk / new_chunk
L2 a dedicated per-pass span scheduler.prefill.pass (not the batch span),
emitted from end_pass() on every pass. Its sampling decision is a pure
function of mlp_sync_seq, so a sampled gather round is sampled on all
DP ranks. Idle ranks and ranks without a local batch still emit, which
is exactly what the cross-rank skew analysis needs. No links.
Hooks are installed in both scheduling loops:
* get_next_disagg_prefill_batch_to_run() for --disaggregation-mode prefill
(hicache / bootstrap / mlp_sync segments), and
* get_next_batch_to_run() for standalone prefill (hicache / mlp_sync),
so the instrumentation can be validated on ds41-prefill-dp4 without a decode
peer. match/adder/counters live in the shared get_new_batch_prefill path.
Semantics:
* match times only init_next_round_input; check_prefetch_progress counts as
skipped_prefetch, never as match.
* bootstrap and hicache_events are cross-rank sync points measured at the
outer call, not attributed to local CPU work.
* dp_gather_ms comes from the existing MLPSyncBatchInfo.sync_wait_seconds;
sglang:dp_attention_sync_wait_seconds is reused, no duplicate metric.
The value is cleared before each outer call and mlp_sync_seq only
increments when a gather actually ran, so stale values cannot leak.
* resumed vs new chunked requests are captured separately: had_chunked is
recorded at pass start (before process_prefill_chunk/admission can
consume the last chunk) and new_chunk from adder.new_chunked_req. This
fixes both the first-chunk and last-chunk mislabels.
Opt-in:
* SGLANG_BATCH_STAGE_TIMING=1 (default 0);
SGLANG_BATCH_PASS_SPAN=1 / SGLANG_BATCH_PASS_SPAN_SAMPLE=0.05.
* When off every hook is one module-level bool check; no logging, no CUDA
calls, no new collectives.
Tests: test/registered/unit/observability/test_batch_telemetry_stage_timing.py
covers first/middle/last chunk labelling, all-prefetch-skip passes without a
batch, stale-gather clearing, rank-deterministic pass-span sampling, and the
disabled no-op path. py_compile passes on all touched files.
These three files were already running in the gist production image
(gist-nvlink-ee9170e7bb-pzcfix) but were never committed: the image was
produced by docker-commit with the btelem4/5/6 files overlaid on top of
the built source tree.
Port them so the tree matches the image before layering new instrumentation:
* observability/batch_telemetry.py: per-batch histograms, per-DP-step
shape histograms and the sampled OTLP span scheduler.batch.{prefill,decode}.
* managers/scheduler_components/metrics_reporter.py: observe_prefill/
observe_decode call sites and the batch-trace log suffix.
* managers/scheduler_components/dp_attn.py: observe_dp_step().
Content is identical to the image; only line endings were normalised to LF
(the image copy was mixed CRLF/LF). No behavior change when
SGLANG_BATCH_TELEMETRY is unset (default off).
Sender and executor threads can enter synchronous IntraNode transfers with
another current CUDA device. Keep the explicit bootstrap ordinal, check it
against the published model device, and scope the native batch call to that
device while restoring the calling thread on success or failure.
Ordinary transports, force-TCP and non-CUDA devices keep their existing
path. Missing or inconsistent CUDA identity is rejected before native IO.
Add 13 CPU regressions importing the production wrapper, including real
constructor calls with a mocked SDK, concurrent thread-local device states,
error restoration and transport/device compatibility. All 13 pass; two
selected tests fail on the unchanged wrapper at behavior assertions. Native
and CUDA boundaries are mocked, so this does not certify physical CUDA IPC
or attribute the separate container-PID collision to this change.
Use the request radix-key cap for L3 prefetch so bounded replay cannot
split a queried trailing-state checkpoint at the later rematch. Reject
Host-only SWA-gap candidates before they mask a complete ancestor, while
preserving device matching and the final load-back completeness guard.
Add CPU regressions that import the real scheduler, cache and TreeCore
methods. All 13 test methods pass; the same tests against the unmodified
scheduler/SWA code fail six methods (ten assertions including subtests).
The checks use synthetic cache state and controller boundaries, not GPU
or live Store execution.
Do not invent shorter checkpoints when a trailing pool has no restorable-prefix inventory. Reject partial allocation and vote no common endpoint while preserving the collective shape and plain-KV contiguous behavior.
Add negative controls for missing inventory and retain pure-KV coverage. Validation: 33 cache/PP CPU tests passed with an external Mac native-hash import adapter using the repository Python hash oracle; repository pre-commit checks passed.
Choose a common validated trailing-state endpoint before partial host allocation, retain that endpoint set across query synchronization, and size auxiliary staging from the selected length. Assumed trailing hits prove only their original endpoint.
Add controller/cache CPU regressions and a PP checkpoint-vote control. Validation: 14 targeted CPU tests and repository pre-commit checks passed.
With long prompts and L3 under pressure, a matched prefix can contain a node
whose SWA component has neither a device nor a host value (host tombstone
evicted between match and load-back). build_hicache_transfers asserted on
this and killed the prefill scheduler.
Add SWAComponent.has_load_back_gap() and have build_load_back_spec return an
empty transfer spec when a gap is found, so the caller backs off and the
request is recomputed. Measured on peak15-v6 (1M, cap 64): 0 scheduler
exceptions, hit 59.4%->94.0%, TTFT p95 47.0s->3.4s with the SWA prefix-match
gate disabled.
The unified-cache SWA component gates prefix matching on a live SWA window,
so a long prefix only matches up to the last sliding window and the tail is
recomputed. Add SGLANG_DSV4_SWA_GATES_PREFIX_MATCH (default True) so the gate
can be turned off for measurement; the SWA window is then rebuilt/recovered
downstream as for the req-ring paths. Measured on 4xB300 DP4 at 160k prefix:
c32 hit tokens 40.6% -> ~100% (with HiCache L2/L3 enabled), needle test
(cold vs hit) identical.
The ctypes-based probing loaded a second libcudart copy into the process,
which corrupted CUDA/runtime state and segfaulted the scheduler processes
shortly after registration (reproducible on prefill ranks, intermittent on
decode). cuda.bindings.runtime.cudaPointerGetAttributes is a properly
typed binding and boots cleanly on both roles.
IntraNodeNvlinkTransport can only register/reach device memory. sglang's
register list includes host regions (aux buffers, some state components);
one host region failed the whole registerLocalMemoryBatch and the engine
rolled back every region, leaving the segment descriptor empty so all KV
transfers failed with 'Requested address ... not found'.
When the intra-node NVLink transport is active (MC_INTRANODE_NVLINK etc.):
- register only device-memory regions (probe via cudaPointerGetAttributes)
- route transfer blocks whose local source is host memory over the ordered
per-endpoint zmq channel (new STATE_DATA message) instead of the engine;
the receiver validates and writes the block into its buffer, relying on
the same ordering guarantee as the aux TCP path
- force aux over the TCP socket path automatically