Promote the validated ds41-gist-exp baseline:
- image recipe-ns-l3ssd-433257c7b-baked-20261115 (SGLang + Mooncake 0.3.14 + L3 SSD)
- P: cache / no-optimistic / fastokens / tokenizer-worker 16 / L3 SSD 4T
- D: fastokens / tokenizer-worker 16
- gateway: normal health/circuit, no P admission caps
- c580 baseline 24.045 rps; c620 and buffer_only results recorded
- Feishu baseline doc and deploy manifests included
This is the base for the next Dynamo layer.
Backport the upstream namespace-tool stack onto the B300 recipe base:
- #46370: Function.namespace / ChatMessage.action protocol fields.
- #49625: Responses/Messages native recipe rendering (convert_native),
tool_search/defer_loading, namespace identity in/out/history replay.
Adapted to this base: skips serving_kimi_k3.py (not present), drops the
Minimax/Beam/wire-only dependencies added between 10-23 and the PR base,
and keeps the existing tool-parsing fallback where the newer
`parses_without_tools()` helper is unavailable.
Also includes the PR's Anthropic tool-reference plumbing and unit tests.
A 2 ms phase whose CPU clock reads 0.1 us over its wall clock is clock
granularity, not an accounting bug; the anomaly flag fired on nearly every
short phase in the live verification run.
Only flag (and record non_cpu_raw_ms) when the deficit exceeds both 100 us and
1% of the phase wall. Larger real anomalies keep the previous behaviour.
The frontend sglang:num_requests_total has no dp_rank label (verified from the
series registry), so per-rank RPS was not derivable from any existing metric.
Add sglang:batch_requests_total{phase,dp_rank}, incremented by the scheduler
process that actually completed the request:
phase=prefill prefill compute finished and the request entered the PD
transfer queue (not a guarantee that the transfer succeeded)
phase=decode request finished decoding
Only the canonical scheduler rank reports (attn_tp_rank==0, attn_cp_rank==0,
last PP stage) — every TP/CP rank runs a scheduler and sees the same request,
so an ungated counter would be multiplied by TP*CP in the multiprocess
registry. Both hooks count each request exactly once: the decode path counts
after _handle_finish_state_updated_req and on the sampling-mask abort branch,
skipping rows that finished on an earlier step; the prefill path counts after
the request enters the disagg inflight queue (aborted/retried requests never
reach it). The per-request finished() check is skipped entirely when
SGLANG_BATCH_TELEMETRY is off.
Covers the design's unit-level acceptance points:
* disabled/unsampled loops never read the thread clock;
* wall/CPU/non-CPU semantics, unmeasured != zero, partial measurement omits
non-CPU, busy vs sleeping windows are distinguishable;
* CPU > wall flags an accounting anomaly with a raw negative delta;
* child spans carry the CPU split;
* repeated phases (prefix indices) stay attribute-only, inclusive phases are
not summed, and eviction demand/shortfall/plan plus Mamba plan/zero results
are recorded as separate counters.
Parent span scheduler.loop.prefill_batch_init records wall + the scheduler
thread's CPU at both existing close points (empty admission and normal
return); the CPU clock is only read when loop detail is enabled and sampled,
through the safe batch_telemetry.thread_cpu_ns() wrapper so a missing clock
degrades to 'not measured' instead of failing the scheduling path.
Fixed children, in call order:
schedule_batch_init, ready_to_load_host_cache, prepare_for_extend,
alloc_for_extend (+ prefix_device_indices attribute-only, sampling_batch_info),
evict_to_free_tokens, evict_for_alloc, prefill_stats (+ pending_tokens),
mixed_chunk_tail.
* prefill_stats wraps the whole assignment so _get_num_pending_tokens() is
included; mixed_chunk_tail only exists when that branch is taken.
* prefill-only shape counters (n_reqs/n_new/n_cached) are snapshotted before
mixed-chunk folding; a mixed batch additionally publishes decode_reqs and
merged totals under explicit *_merged_* names.
* eviction is timed at the call site (outside the allocator wrapper) so
rank-consensus waits are included, and covers base, SWA (x2), UnifiedSWA
(x3) and the extend Mamba reclaim through evict_for_alloc_observed().
Demand, shortfall and plan are separate counters; Mamba plan and realized
counts are both recorded (including zero).
All of it is gated by detail on a sampled loop.
Adds an optional thread CPU clock next to the existing wall clock:
* detail_begin/end record time.thread_time_ns() only when detail is on and the
loop is sampled; detail_add() accepts an explicit thread_cpu_ns for callers
that already hold absolute start/end points (the prefill_batch_init parent).
* phase records carry [wall, count, start_wall, end_wall, cpu_sum, cpu_count].
thread_cpu_ms and thread_cpu_measured_count are emitted whenever any
occurrence was measured; non_cpu_ms only when every occurrence was, so a
partial measurement never attributes unmeasured wall to non-CPU time. A
phase whose CPU exceeds its wall keeps the clamped value, adds
non_cpu_raw_ms and accounting_anomaly, and bumps a per-loop anomaly count.
* new detail_attr_sum()/detail_attr_setdefault() accumulate low-cardinality
numeric loop attributes (eviction tokens, free-token snapshots).
* prefix_device_indices joins the attribute-only prefixes so per-request
repeats never create spans.
* child spans carry phase_thread_cpu_ms / phase_cpu_measured_count /
phase_non_cpu_ms, so the CPU split is visible without the loop attributes.
Default-off path is unchanged: no thread clock, no new dicts, no spans.
The detail-phase refactor made residual a required positional argument while
test_loop_span_accounting.py still calls _emit_pass_span with four arguments,
so that test failed on the base commit. Default it to 0.0.
Split out of the prefill_batch_init observability series to keep that review
scope focused.
Deriving each candidate's uncached estimate from its own recent hit rate is
self-reinforcing: a rank that falls behind sees its hit rate drop (fewer
chances to hit), gets charged more for every new request, and falls further
behind. Measured on c580: one rank ended 16% below the others, 30 s cross-rank
dispersion max 952x, and 9.6% lower rps than the total_tokens arm.
Charge the fleet-average uncached size instead and rank by the published
num_waiting_uncached_tokens, which already carries each rank's exact prefix
match.
DP balancing on total_tokens charges each request's whole prompt, but with
prefix caching only the unmatched part costs prefill compute. On a workload
with a 95% hit rate the prompt is ~95% noise: per request the uncached part
is a few percent of the prompt yet varies by more than an order of magnitude
(p50 1.9k / p99 59k tokens against a 95k prompt), so one cold request
outweighs many hot ones and per-rank prefill work drifts apart.
Add --load-balance-method uncached_tokens: DPBudget tracks the published
LoadSnapshot.num_waiting_uncached_tokens (prefix match minus accepted storage
hits) plus the rank's recent cache_hit_rate for a newly dispatched request.
total_tokens DP balancing replaces its per-rank estimate with the rank's
latest load snapshot, so a snapshot that misses work already sent to the
rank lets the controller keep sending to it. Close three gaps:
- count grammar-queue requests in num_waiting_reqs and num_total_tokens
- count the in-progress chunked request's unprefilled tokens
- publish a snapshot once newly received requests are queued, since the
one published after a forward predates the requests sent during it
Copies the three files that the live cw mounts over the image:
- managers/load_snapshot.py: queued_total_input_tokens + drain_rate_tokens_per_s
- managers/scheduler_components/load_inquirer.py: fills the two new fields
- models/deepseek_v4.py: env-gated MM trace (SGLANG_MM_TRACE, off by default)
After this commit every runtime file the P/D pods need is in the tree, so the
hostPath file mounts can be dropped once the image is rebuilt.
SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE=<base> makes MooncakeStore.setup()
pass enable_http_server=True and client_http_port=base+dp_rank, so each DP
rank's embedded Mooncake client exposes /metrics (needs MC_STORE_CLIENT_METRIC=1).
Off by default.
Phase 1/3 of the async-backup handoff instrumentation plan:
* new fwd_timeline probe (SGLANG_FWD_TIMELINE=1): one CUDA event pair around
the model execute call per forward, resolved without blocking and written as
wall-clock JSONL per DP rank. Joined with loop spans it separates host time
spent while the GPU is busy (overlapped) from GPU-idle host stalls.
* detail phases: fwd_init_new, fwd_prepare, fwd_execute, fwd_graph_load,
fwd_graph_replay, fwd_post, fwd_runner_tail,
prefill_policy, prefill_adder_init, prefill_scan, prefill_batch_init.
All spans ride the existing SGLANG_TRACE_LOOP_DETAIL path; the GPU probe is
off by default.
Adds the Phase A instrumentation from the async-backup handoff (§7.1):
* PassTimings: pending_abort / chunk / new_batch / set_sched (+ chunk_count)
* prefill.get_next_disagg_prefill_batch_to_run: time
process_pending_chunked_abort, process_prefill_chunk,
get_new_batch_prefill and set_schedule_time_batch
* scheduler.prefill.pass gains sglang.{pending_abort,chunk,new_batch,set_sched}_ms,
sglang.get_next_residual_ms (= other - these four) and sglang.chunk_count;
the existing sglang.other_ms is unchanged for dashboard compatibility.
~5 monotonic calls per pass; enabled by SGLANG_BATCH_STAGE_TIMING (already on).
Net async change (from exp-async-backup-perop-20261107) applied onto
b300-pick-46884-20261107. Runtime files applied cleanly; the new tests and
the timeline probe are added as-is.
Interaction to validate on hardware: #46884 publishes prefill read-done before
the replay, so the scheduler-stream fence that the index copy waits on is
shorter; the async worker's wait should shrink and the copy may overlap more of
the in-flight forward. Re-baseline the 4-arm A/B on this branch.
The prefill shared-read-done fastpath lets the schedule stream run ahead of
the forward that writes a stashed chunk's KV instead of waiting for the whole
forward. It was disabled whenever hierarchical cache or a storage backend was
enabled, because host and storage offload copies used to be ordered only
behind the schedule stream and could have read unfinished KV.
The cache controller now orders its device-to-host copies behind the forward
stream through write_fence_stream, and its load-back H2D copies through
load_fence_stream, so that reason no longer holds when every KV offload copy
carries that fence. The scheduler records on the model runner whether that is
the case: hierarchical cache with a cache controller and no decode offload
manager, which has no such fence. The publish helper keeps the coarse
whole-forward barrier for LMCache and for unfenced offload, and publishes
read-done otherwise.
With the coarse barrier, a synchronous page-index copy issued while preparing
the next batch inherited the wait on the entire in-flight forward, leaving the
GPU idle between batches. With read-done published, that copy only waits for
the shared-table reads to finish; KV ordering is unchanged because the backup
copies still wait on the forward stream.
Tests: the CPU runner test covers the fenced hierarchical-cache and storage
cases publishing, and LMCache keeping the barrier even when hierarchical cache
is fenced. The GPU ordering test checks that a write-through backup of a chunk
stashed before its forward finished still copies the final KV.
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com>
(cherry picked from commit bb1cb2dded6fb9352d98076280342b10684e061b)
* Normalize the host-pool layout to a string (a None attribute would make
the OTel SDK drop the whole flush span).
* Drop any remaining None-valued flush attributes for the same reason.
Adds loop-detail phases outside run_batch/process_result/get_next:
* inflight_poll: process_disagg_prefill_inflight_queue() (overlap loop)
* inflight_release: release_kv_cache(checkpoint=True) + tree_cache.finish()
inside it, so the checkpoint/insert cost is separated from the poll
* launch_sample: launch_batch_sample_if_needed()
The get_next side already publishes match/adder/hicache/bootstrap/mlp_sync
and a pass residual (sglang.other_ms) on scheduler.prefill.pass.
Adds the phase-3/phase-4 instrumentation from
handoff/prefill-flush-backups-validation-20261107.md:
* process_batch_result outer phases: flush_trace, load_snapshot,
flush_backups, step_counters, metrics_report. process_residual now
subtracts them, so the 37.5% residual is attributable.
* SGLANG_TRACE_BACKUP_DETAIL=1 (default 0) traces one flush:
merge_ops / move_write (anchor + per-sidecar indices) / indices_cpu /
index_sort / index_select / write_fence / build_l2 / submit_d2h
(start-event wait, prepare, per-pool launch, ack events) / ack_enqueue,
plus token/op/pool/device dimensions and ack queue depth.
* Per-flush resolution: one scheduler.cache_flush_backup span per
non-empty flush, child of the loop span. Sub-phases never enter the
loop aggregate, so the residual identity cannot double count.
* Empty vs non-empty flushes are counted on the loop.
All timings are scheduler host wall, not device transfer time.
Review follow-up for the v2 async state payload:
- P2: new tests use CustomTestCase (with a local fallback when sglang is not
importable), matching the disaggregation/Mooncake test convention.
- P3: CP gate docstring now says all-CP-rank transfer *without DSA cache
layer split* keeps the synchronous path.
- Add real call-chain coverage:
* _send_kv_chunk -> sender.send -> add_transfer_request -> transfer_worker
-> materialize_state_indices -> maybe_send_extra, with DSV4's four
components (DSV4_REQUEST_STATE, REQUEST_ROW, SWA, SWA);
* the worker materializes pinned tensors only after indices_event and
fails loudly when a tensor has no event;
* a second materialization without an event (staging retry) is a no-op;
* real CUDA ordering: the pinned helper returns before a long kernel and
the event-protected values match kv_to_page_indices.
- Verified on b300 (P container, overlay source): 9 passed in
test_async_state_payload.py, 2 passed in the new
TestMooncakeAsyncStateMaterialization class; the old batching suite has
11 pre-existing failures that also fail on the image source.
The test-only batching suite requires the detail envs to be off when run in
the live pod (the inherited SGLANG_TRACE_MOONCAKE_DETAIL=1 exposes an existing
test fixture omission); noted, not changed here.
Implements the v2 plan in handoff/prefill-state-payload-async-review-20261106.md.
- mem_cache/common.py: add kv_to_page_indices_pinned() (pinned CPU buffer,
non-blocking copy; no synchronous .cpu()).
- prefill.py: gate via should_async_state_payload() (last chunk, independent
supports_async_state_indices, kv_shard_size==1, CP guard); only the
_swa_payload request_window-is-None hot branch uses the pinned helper.
Final segment asserts async state has an indices_event. Other state
builders (mamba/full/DSA) stay synchronous.
- mooncake/conn.py: MooncakeKVSender advertises
supports_async_state_indices; the worker materializes pinned state tensors
to numpy only after indices_event.synchronize(), and fails loudly if a
tensor arrives without an event.
- tests: guard/event/materialize/pinned-body unit tests + CUDA ordering test
(skipped unless torch+CUDA+sglang are available).
No scheduler/transfer semantics change; rollback is the helper switch or
reverting this commit. 49 tests pass, 1 skipped locally.
Implements M2 of handoff/prefill-cpu-kv-overlap-design-20261106.md, behind
SGLANG_TRACE_MOONCAKE_DETAIL=1 (default off; reuses the existing trace stack).
- TransferKVChunk gains optional observation fields: chunk_seq, loop_seq,
send_kind, enqueue/last_enqueue monotonic+wall stamps, queue_wait_accum_ns,
queue_id, requeue_count, num_pages, approx_bytes, observe
- chunk_seq and enqueue timestamps are assigned at the real queue.put()
point (after every early return), so skipped paths cannot leave holes
- Mooncake transfer worker measures queue wait, source-event wait, indices
wait, KV/state/aux transfer, worker busy and residual; emits one
mooncake.kv_chunk span per attempt (+ queue/source/indices wait children)
and updates a per-room join summary
- request completion emits kv.request_join with chunk_count,
last_chunk_queue_wait/worker, final_exposed_ms, completion_poll_ms,
request_join_ms and first-enqueue-to-completion. First enqueue is stamped
by the Mooncake manager at the real queue.put(); no ReqTimeStats field is
added and the legacy queue_entry semantics are unchanged.
- nixl/mori add_transfer_request accept the observation kwarg (no-op there)
No scheduling/transfer semantics change; queue wait is excluded from worker
busy, and worker_busy excludes source/indices event waits. Unit tests: test_mooncake_chunk_detail.py (15 cases, including the
abort/worker interleaving and attempt-vs-total queue wait windows).
Implements M1 of handoff/prefill-cpu-kv-overlap-design-20261106.md:
per-loop detail phases behind SGLANG_TRACE_LOOP_DETAIL=1 (requires
SGLANG_BATCH_STAGE_TIMING=1 + SGLANG_TRACE_LOOP_SPAN=1, only sampled loops).
- run_batch: early_send_host, resolve_inputs, model_forward_host,
copy_result_submit (+ run_residual_ms)
- process_result: copy_done_wait, auxiliary_and_logprob, per_request_result,
kv_state_payload/page_indices/sender_send by send_kind, report_stats
(+ process_residual_ms)
- war_barrier is measured as its own loop phase (outside run_batch's span)
- repeated/per-request phases are summed attributes; single phases are child
spans, so span count never grows with n_reqs/n_segments
- DP-skew join fields: pass_end_wall_ns, arrival_est_wall_ns = pass_end -
mlp_sync, previous-loop phase summary (sglang.prev_*, published only for
consecutive loop_seq; prev_detail_missing marks a gap)
No CUDA syncs added; when disabled every hook is a module-level bool check.
Unit tests: test_loop_detail_phases.py (10 cases).
Review follow-up. match_kv_cache also runs on waiting-queue probes
(calc_priority), PD decode _match_prefix_and_lock and HiCache rematches.
Emitting the counter there counted the same gap once per scheduling round
and attributed "re-prefilled" tokens to decode engines.
- match_kv_cache now only records req.full_kv_hit_length (plus the match-time
gap); it no longer emits.
- the metric is emitted once in PrefillAdder._commit_prefill_admission, where
the accepted prefix is final after any host load-back.
- rename the request counter to swa_gate_reject_admissions_total and document
that a retracted and re-admitted request is counted again (it is re-prefilled).
- pre-create zero-valued children so gate-off deployments expose the series
before the first rejection.
- tests: repeated match does not emit, admission emits once, a retry counts
again, and host load-back closes the gap.
Not runtime verified (no GPU run on this build).
- prefetch_io_aux_func: allow up to SGLANG_HICACHE_PREFETCH_IO_THREADS
transfers in flight (default 1 = legacy behaviour); PrefetchAck is still
emitted strictly FIFO so the per-rank ack stream (and the TP all_reduce in
prefetch_sync_thread_func) is unchanged.
- backup_thread_func: same pipelining for L2->L3 writes via
SGLANG_HICACHE_BACKUP_IO_THREADS.
- SGLANG_HICACHE_TIMING=1: emit [HC_TIMING] prefetch/backup queue, inflight,
p50/p95 per 100 ops (the prefetch/backup lanes were previously single
threaded; this is what caps L3<->L2 at ~1 op at a time).
The unified-cache SWA component gates prefix matching on a live sliding
window: when the window is missing, the match stops before an otherwise
available Full-KV prefix. The existing hit counters do not show the tokens
lost to that gate, which makes the gate-on/gate-off trade-off invisible.
Record the gap between the longest Full-KV match and the accepted prefix
(full_kv_hit_length - (device_prefix_len + host_hit_length)) on the Req and
export it as:
sglang:swa_gate_reject_tokens_total
sglang:swa_gate_reject_reqs_total
This is the token volume that will be re-prefilled because a component
validator (SWA window; Mamba where present) rejected the match. With the
gate enabled it measures the hit length lost to a missing window; with the
gate disabled it only captures the host-only / load-back-gap fallback.
Includes unit tests for the gap and the host-hit case. Not runtime verified
yet (no GPU run on this build).