Copies the three files that the live cw mounts over the image:
- managers/load_snapshot.py: queued_total_input_tokens + drain_rate_tokens_per_s
- managers/scheduler_components/load_inquirer.py: fills the two new fields
- models/deepseek_v4.py: env-gated MM trace (SGLANG_MM_TRACE, off by default)
After this commit every runtime file the P/D pods need is in the tree, so the
hostPath file mounts can be dropped once the image is rebuilt.
Review follow-up for the v2 async state payload:
- P2: new tests use CustomTestCase (with a local fallback when sglang is not
importable), matching the disaggregation/Mooncake test convention.
- P3: CP gate docstring now says all-CP-rank transfer *without DSA cache
layer split* keeps the synchronous path.
- Add real call-chain coverage:
* _send_kv_chunk -> sender.send -> add_transfer_request -> transfer_worker
-> materialize_state_indices -> maybe_send_extra, with DSV4's four
components (DSV4_REQUEST_STATE, REQUEST_ROW, SWA, SWA);
* the worker materializes pinned tensors only after indices_event and
fails loudly when a tensor has no event;
* a second materialization without an event (staging retry) is a no-op;
* real CUDA ordering: the pinned helper returns before a long kernel and
the event-protected values match kv_to_page_indices.
- Verified on b300 (P container, overlay source): 9 passed in
test_async_state_payload.py, 2 passed in the new
TestMooncakeAsyncStateMaterialization class; the old batching suite has
11 pre-existing failures that also fail on the image source.
The test-only batching suite requires the detail envs to be off when run in
the live pod (the inherited SGLANG_TRACE_MOONCAKE_DETAIL=1 exposes an existing
test fixture omission); noted, not changed here.
Implements the v2 plan in handoff/prefill-state-payload-async-review-20261106.md.
- mem_cache/common.py: add kv_to_page_indices_pinned() (pinned CPU buffer,
non-blocking copy; no synchronous .cpu()).
- prefill.py: gate via should_async_state_payload() (last chunk, independent
supports_async_state_indices, kv_shard_size==1, CP guard); only the
_swa_payload request_window-is-None hot branch uses the pinned helper.
Final segment asserts async state has an indices_event. Other state
builders (mamba/full/DSA) stay synchronous.
- mooncake/conn.py: MooncakeKVSender advertises
supports_async_state_indices; the worker materializes pinned state tensors
to numpy only after indices_event.synchronize(), and fails loudly if a
tensor arrives without an event.
- tests: guard/event/materialize/pinned-body unit tests + CUDA ordering test
(skipped unless torch+CUDA+sglang are available).
No scheduler/transfer semantics change; rollback is the helper switch or
reverting this commit. 49 tests pass, 1 skipped locally.
Implements M2 of handoff/prefill-cpu-kv-overlap-design-20261106.md, behind
SGLANG_TRACE_MOONCAKE_DETAIL=1 (default off; reuses the existing trace stack).
- TransferKVChunk gains optional observation fields: chunk_seq, loop_seq,
send_kind, enqueue/last_enqueue monotonic+wall stamps, queue_wait_accum_ns,
queue_id, requeue_count, num_pages, approx_bytes, observe
- chunk_seq and enqueue timestamps are assigned at the real queue.put()
point (after every early return), so skipped paths cannot leave holes
- Mooncake transfer worker measures queue wait, source-event wait, indices
wait, KV/state/aux transfer, worker busy and residual; emits one
mooncake.kv_chunk span per attempt (+ queue/source/indices wait children)
and updates a per-room join summary
- request completion emits kv.request_join with chunk_count,
last_chunk_queue_wait/worker, final_exposed_ms, completion_poll_ms,
request_join_ms and first-enqueue-to-completion. First enqueue is stamped
by the Mooncake manager at the real queue.put(); no ReqTimeStats field is
added and the legacy queue_entry semantics are unchanged.
- nixl/mori add_transfer_request accept the observation kwarg (no-op there)
No scheduling/transfer semantics change; queue wait is excluded from worker
busy, and worker_busy excludes source/indices event waits. Unit tests: test_mooncake_chunk_detail.py (15 cases, including the
abort/worker interleaving and attempt-vs-total queue wait windows).
Implements M1 of handoff/prefill-cpu-kv-overlap-design-20261106.md:
per-loop detail phases behind SGLANG_TRACE_LOOP_DETAIL=1 (requires
SGLANG_BATCH_STAGE_TIMING=1 + SGLANG_TRACE_LOOP_SPAN=1, only sampled loops).
- run_batch: early_send_host, resolve_inputs, model_forward_host,
copy_result_submit (+ run_residual_ms)
- process_result: copy_done_wait, auxiliary_and_logprob, per_request_result,
kv_state_payload/page_indices/sender_send by send_kind, report_stats
(+ process_residual_ms)
- war_barrier is measured as its own loop phase (outside run_batch's span)
- repeated/per-request phases are summed attributes; single phases are child
spans, so span count never grows with n_reqs/n_segments
- DP-skew join fields: pass_end_wall_ns, arrival_est_wall_ns = pass_end -
mlp_sync, previous-loop phase summary (sglang.prev_*, published only for
consecutive loop_seq; prev_detail_missing marks a gap)
No CUDA syncs added; when disabled every hook is a module-level bool check.
Unit tests: test_loop_detail_phases.py (10 cases).
- prefetch_io_aux_func: allow up to SGLANG_HICACHE_PREFETCH_IO_THREADS
transfers in flight (default 1 = legacy behaviour); PrefetchAck is still
emitted strictly FIFO so the per-rank ack stream (and the TP all_reduce in
prefetch_sync_thread_func) is unchanged.
- backup_thread_func: same pipelining for L2->L3 writes via
SGLANG_HICACHE_BACKUP_IO_THREADS.
- SGLANG_HICACHE_TIMING=1: emit [HC_TIMING] prefetch/backup queue, inflight,
p50/p95 per 100 ops (the prefetch/backup lanes were previously single
threaded; this is what caps L3<->L2 at ~1 op at a time).