20897 Commits
Author SHA1 Message Date
minke.yu 5d5870f3ea observability: add scheduler loop span and no-batch accounting
Adds an opt-in scheduler.loop span (SGLANG_TRACE_LOOP_SPAN) with recv /
get_next / run_batch / process_result / kv_send children. The span is only
created for sampled loops that have pending work; fully idle loops stay in
the counters so the 714 pass/s idle spin never floods Jaeger.

No-batch accounting is always on with SGLANG_BATCH_STAGE_TIMING:
- sglang:scheduler_no_batch_{seconds,total}{dp_rank,reason,forward_overlap}
- sglang:scheduler_dp_lockstep_idle_seconds_total{dp_rank,forward_overlap}
with the design's reason priority (result_backpressure, deferred_input,
waiting_bootstrap, waiting_prefetch, admission_blocked, waiting_pd,
fully_idle, no_ready_other) and local_batch_kind/global_outcome from the DP
sync. DP lockstep idle is counted separately from pure no-batch wait.

loop gpu_ms reuses the existing DeviceTimer intervals (no new CUDA events);
the forward.gpu lane is optional and approximate. Pass/recv spans become
children of the loop span when it is active.

Tests: test_loop_span_accounting (5) and the existing stage-timing suite (8)
pass.
2026-11-05 15:00:23 +08:00
minke.yu 309af6e6e1 observability: record prefetch ready latency on the L3 hit success path
_handle_prefetch_result() retires the operation with a direct
del ongoing_prefetch[request] instead of _retire_ongoing_prefetch(), so
the ready-latency hook only ever saw miss/revoke paths: with a cache-hit
workload storage_prefetch_ready_seconds{result="hit"} stayed at 0 even
though storage_prefetch_hit_tokens_total was moving ~312k tok/s.

Observe right after the del, classifying by loaded_from_storage.
2026-11-04 19:14:02 +08:00
minke.yu 1a274df27e observability: fix prefetch ready latency to cover fast lookups
The first version stamped the request on its first in-flight
check_prefetch_progress() call. A fast L3 query can be drained by
_process_hicache_events before that check runs (the op is already gone
from ongoing_prefetch), so those requests produced no sample -- exactly
the common case.

Stamp at issue time in prefetch_from_storage() and observe once at
_retire_ongoing_prefetch(), classifying by operation.storage_hit_count.
Retire is the single exit for hit completion, below-threshold revoke,
anchor loss and host-capacity fallback, so hit and miss are both covered.

No behavior change; the hook is still behind SGLANG_BATCH_STAGE_TIMING.
2026-11-04 18:04:34 +08:00
minke.yu cb7988f849 observability: measure storage prefetch ready latency (hit/miss)
Under --hicache-storage-prefetch-policy wait_complete a request is not
admitted until its L3 lookup resolves, and a miss pays the same query round
trip as a hit. The scheduler only exposed a skip counter, so the actual wait
was unknown.

Add sglang:storage_prefetch_ready_seconds{dp_rank, result=hit|miss}:
  * stamp cache_request_handle on its first in-flight check,
  * observe at the moment check_prefetch_progress() turns usable, split by
    whether prefetch_loaded_tokens_by_reqid ended up > 0,
  * clear the stamp on revoke / abort / accounting discard.

Buckets cover 1 ms .. 30 s. Gated by SGLANG_BATCH_STAGE_TIMING (off by
default). The measurement is ~2 dict ops + one observe per request, not per
pass.

Tests: test/registered/unit/observability/test_batch_telemetry_stage_timing.py
adds the hit/miss split assertion (9 tests, all passing).
2026-11-04 17:26:41 +08:00
minke.yu 0866012b80 observability: split scheduler recv_requests into pull/broadcast/unwrap
Runtime accounting on the prefill service (64 x 14.6k-token requests,
8-way concurrency) showed the non-forward wall time is dominated by the
scheduler receive path, not by radix match:

  forward (extend+idle) 58.2% | get_next 6.9% | recv_requests 25.6%
  process_batch_result 1.8% | run_batch overhead 0.9% | other ~6%

The existing pass span only covers get_next, so it cannot explain that
25.6%.  Split it with the same opt-in gate:

  * sglang:recv_requests_seconds{dp_rank, part} (total/pull/broadcast/
    unwrap/mm), 5 new series per rank
  * sglang:recv_requests_msgs_total, sglang:recv_requests_prompt_tokens_total
  * OTLP span scheduler.recv_requests with the same breakdown, sampled
    deterministically by a per-process counter (SGLANG_RECV_SPAN_SAMPLE,
    default 0.05; SGLANG_RECV_SPAN=0 disables the span only)

Note on scope: tokenization itself does NOT run in the scheduler. The
tokenizer workers (--tokenizer-worker-num) produce TokenizedGenerateReqInput
and the scheduler receives the already-tokenized ids. What this measures is
ZMQ pull + decode, the cross-rank broadcast of the pickled objects, and
unwrap_pickle_fields() of the token-id buffers -- which is why large prompts
make recv_requests expensive.

Tests: test/registered/unit/observability/test_batch_telemetry_stage_timing.py
now also covers the recv histogram parts, counters, zero-message passes and
deterministic recv-span sampling (8 tests total, all passing).
2026-11-04 16:11:13 +08:00
minke.yu 85f4f9576c scheduler: add opt-in per-pass prefill stage timing (SGLANG_BATCH_STAGE_TIMING)
Motivation: on the PD prefill node the wall-clock budget is dominated by
get_next_batch_to_run (~50% of the hour on 2026-11-02 16:00-17:00), and
forward_overlap=partial is only 21-87 s/h, i.e. it does not overlap the GPU
forward. We know the total but not which segment inside it costs: radix
match, PrefillAdder admission, the HiCache event collective, the bootstrap
all-reduce, or the DP metadata gather.

Measurement design (one sample per scheduling pass, including passes that
produce no batch; see handoff/prefill-batch-stage-timing-20261103.md v2/v3):

  L1  7 per-pass histograms:
      sglang:batch_prefill_{get_next,match,adder,hicache_events,
      bootstrap,mlp_sync,other}_seconds
      other = total - (match + adder + hicache + bootstrap + mlp_sync),
      clamped to >= 0. Nested stages are not double counted.

  L3  5 per-pass counters:
      candidates / skipped_prefetch / admitted / resumed_chunk / new_chunk

  L2  a dedicated per-pass span scheduler.prefill.pass (not the batch span),
      emitted from end_pass() on every pass. Its sampling decision is a pure
      function of mlp_sync_seq, so a sampled gather round is sampled on all
      DP ranks. Idle ranks and ranks without a local batch still emit, which
      is exactly what the cross-rank skew analysis needs. No links.

Hooks are installed in both scheduling loops:
  * get_next_disagg_prefill_batch_to_run() for --disaggregation-mode prefill
    (hicache / bootstrap / mlp_sync segments), and
  * get_next_batch_to_run() for standalone prefill (hicache / mlp_sync),
so the instrumentation can be validated on ds41-prefill-dp4 without a decode
peer. match/adder/counters live in the shared get_new_batch_prefill path.

Semantics:
  * match times only init_next_round_input; check_prefetch_progress counts as
    skipped_prefetch, never as match.
  * bootstrap and hicache_events are cross-rank sync points measured at the
    outer call, not attributed to local CPU work.
  * dp_gather_ms comes from the existing MLPSyncBatchInfo.sync_wait_seconds;
    sglang:dp_attention_sync_wait_seconds is reused, no duplicate metric.
    The value is cleared before each outer call and mlp_sync_seq only
    increments when a gather actually ran, so stale values cannot leak.
  * resumed vs new chunked requests are captured separately: had_chunked is
    recorded at pass start (before process_prefill_chunk/admission can
    consume the last chunk) and new_chunk from adder.new_chunked_req. This
    fixes both the first-chunk and last-chunk mislabels.

Opt-in:
  * SGLANG_BATCH_STAGE_TIMING=1 (default 0);
    SGLANG_BATCH_PASS_SPAN=1 / SGLANG_BATCH_PASS_SPAN_SAMPLE=0.05.
  * When off every hook is one module-level bool check; no logging, no CUDA
    calls, no new collectives.

Tests: test/registered/unit/observability/test_batch_telemetry_stage_timing.py
covers first/middle/last chunk labelling, all-prefetch-skip passes without a
batch, stale-gather clearing, rank-deterministic pass-span sampling, and the
disabled no-op path. py_compile passes on all touched files.
2026-11-04 14:53:29 +08:00
minke.yu 2434d09848 observability: commit the in-image per-batch telemetry baseline (btelem4-6)
These three files were already running in the gist production image
(gist-nvlink-ee9170e7bb-pzcfix) but were never committed: the image was
produced by docker-commit with the btelem4/5/6 files overlaid on top of
the built source tree.

Port them so the tree matches the image before layering new instrumentation:

  * observability/batch_telemetry.py: per-batch histograms, per-DP-step
    shape histograms and the sampled OTLP span scheduler.batch.{prefill,decode}.
  * managers/scheduler_components/metrics_reporter.py: observe_prefill/
    observe_decode call sites and the batch-trace log suffix.
  * managers/scheduler_components/dp_attn.py: observe_dp_step().

Content is identical to the image; only line endings were normalised to LF
(the image copy was mixed CRLF/LF). No behavior change when
SGLANG_BATCH_TELEMETRY is unset (default off).
2026-11-04 14:48:38 +08:00
Florian Wunderlich d47a99b2ad test: import SWA eviction helper directly
(cherry picked from commit 5729f61e175bde3ef2b67dc3a95abb1042f4bf1b)
2026-11-04 10:41:52 +08:00
Florian Wunderlich 2ba9b8431a Fix SWA eviction behind the replay-trimmed prefix frontier
(cherry picked from commit f399d1e9a2bf7c33c7244ecf405e756292ddaf2b)
2026-11-04 10:41:52 +08:00
zicai.pu ee9170e7bb fix(mooncake): bind IntraNode batches to the bootstrap CUDA device
Sender and executor threads can enter synchronous IntraNode transfers with
another current CUDA device. Keep the explicit bootstrap ordinal, check it
against the published model device, and scope the native batch call to that
device while restoring the calling thread on success or failure.

Ordinary transports, force-TCP and non-CUDA devices keep their existing
path. Missing or inconsistent CUDA identity is rejected before native IO.

Add 13 CPU regressions importing the production wrapper, including real
constructor calls with a mocked SDK, concurrent thread-local device states,
error restoration and transport/device compatibility. All 13 pass; two
selected tests fail on the unchanged wrapper at behavior assertions. Native
and CUDA boundaries are mocked, so this does not certify physical CUDA IPC
or attribute the separate container-PID collision to this change.
B300261102
2026-10-30 11:42:19 +08:00
zicai.pu 17505e6cf6 hicache: align prefetch and host restore boundaries
Use the request radix-key cap for L3 prefetch so bounded replay cannot
split a queried trailing-state checkpoint at the later rematch. Reject
Host-only SWA-gap candidates before they mask a complete ancestor, while
preserving device matching and the final load-back completeness guard.

Add CPU regressions that import the real scheduler, cache and TreeCore
methods. All 13 test methods pass; the same tests against the unmodified
scheduler/SWA code fail six methods (ten assertions including subtests).
The checks use synthetic cache state and controller boundaries, not GPU
or live Store execution.
2026-10-30 11:42:19 +08:00
zicai.pu 81f7398855 fix(hicache): reject unproven trailing restore endpoints
Do not invent shorter checkpoints when a trailing pool has no restorable-prefix inventory. Reject partial allocation and vote no common endpoint while preserving the collective shape and plain-KV contiguous behavior.

Add negative controls for missing inventory and retain pure-KV coverage. Validation: 33 cache/PP CPU tests passed with an external Mac native-hash import adapter using the repository Python hash oracle; repository pre-commit checks passed.
2026-10-30 11:42:19 +08:00
zicai.pu 0a56a5f6e7 fix(hicache): preserve sparse restore checkpoints under host pressure
Choose a common validated trailing-state endpoint before partial host allocation, retain that endpoint set across query synchronization, and size auxiliary staging from the selected length. Assumed trailing hits prove only their original endpoint.

Add controller/cache CPU regressions and a PP checkpoint-vote control. Validation: 14 targeted CPU tests and repository pre-commit checks passed.
2026-10-30 11:35:29 +08:00
minke.yu 7f6fe22445 mem_cache: fail-soft on SWA load-back gaps instead of asserting
With long prompts and L3 under pressure, a matched prefix can contain a node
whose SWA component has neither a device nor a host value (host tombstone
evicted between match and load-back). build_hicache_transfers asserted on
this and killed the prefill scheduler.

Add SWAComponent.has_load_back_gap() and have build_load_back_spec return an
empty transfer spec when a gap is found, so the caller backs off and the
request is recomputed. Measured on peak15-v6 (1M, cap 64): 0 scheduler
exceptions, hit 59.4%->94.0%, TTFT p95 47.0s->3.4s with the SWA prefix-match
gate disabled.
2026-10-26 13:23:17 +00:00
minke.yu f6fc6ac3e7 mem_cache: allow disabling the DSv4.1 SWA prefix-match gate via env
The unified-cache SWA component gates prefix matching on a live SWA window,
so a long prefix only matches up to the last sliding window and the tail is
recomputed. Add SGLANG_DSV4_SWA_GATES_PREFIX_MATCH (default True) so the gate
can be turned off for measurement; the SWA window is then rebuilt/recovered
downstream as for the req-ring paths. Measured on 4xB300 DP4 at 160k prefix:
c32 hit tokens 40.6% -> ~100% (with HiCache L2/L3 enabled), needle test
(cold vs hit) identical.
2026-10-26 04:06:25 +00:00
minke.yu 0a3c370451 mooncake: probe pointers via cuda.bindings instead of ctypes libcudart
The ctypes-based probing loaded a second libcudart copy into the process,
which corrupted CUDA/runtime state and segfaulted the scheduler processes
shortly after registration (reproducible on prefill ranks, intermittent on
decode). cuda.bindings.runtime.cudaPointerGetAttributes is a properly
typed binding and boots cleanly on both roles.
2026-10-25 14:04:56 +00:00
minke.yu b1b303c9f2 mooncake: support host-memory payloads with intra-node NVLink transport
IntraNodeNvlinkTransport can only register/reach device memory. sglang's
register list includes host regions (aux buffers, some state components);
one host region failed the whole registerLocalMemoryBatch and the engine
rolled back every region, leaving the segment descriptor empty so all KV
transfers failed with 'Requested address ... not found'.

When the intra-node NVLink transport is active (MC_INTRANODE_NVLINK etc.):
- register only device-memory regions (probe via cudaPointerGetAttributes)
- route transfer blocks whose local source is host memory over the ordered
  per-endpoint zmq channel (new STATE_DATA message) instead of the engine;
  the receiver validates and writes the block into its buffer, relying on
  the same ordering guarantee as the aux TCP path
- force aux over the TCP socket path automatically
2026-10-25 14:04:52 +00:00
Liangsheng Yin 7d996300b9 [Perf] dsv41: Use the fused ratio-2 compressor in the prefill graph (#46230) 2026-10-23 21:33:42 -07:00
Liangsheng Yin 54df642fe6 [Perf] dsv41: Translate SWA locations once per prefill graph step (#46229) 2026-10-23 21:33:15 -07:00
Liangsheng Yin 564aec61c8 [Perf] dsv41: Feed wo_b the fused MXFP8 attention output in the prefill graph (#46231) 2026-10-23 21:31:33 -07:00
Yihao WangandClaude Opus 5.5 a7183a8246 [Feat] spec: Let out-of-tree platforms provide EAGLE draft graph runners (#45145)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 21:28:56 -07:00
Yihao WangandClaude Opus 5.5 af3f9e4c8f [Feat] spec: Let out-of-tree platforms provide their own speculative worker (#45144)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 21:24:58 -07:00
MickandMick Qian f09cec65e9 [Fix] diffusion/storage: Join uploads before request cleanup (#46068)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-24 12:19:10 +08:00
MickandMick Qian f4ecaae888 [Fix] diffusion/mesh: Guard asynchronous job publication and defaults (#46081)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-24 12:18:25 +08:00
MickandMick Qian cdab6c1bcc [Fix] diffusion/loader: Preserve checkpoints on read failures (#46095)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-24 12:17:46 +08:00
f2c634961b [Perf] diffusion: Project LTX-2 prompt K/V once per denoising loop (#46105)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 12:17:04 +08:00
Liangsheng Yin 3b95120f00 [Fix] dsv41: Revert MegaGate routing in prefill (#46242) 2026-10-23 21:15:38 -07:00
55f23a8b59 [Fix] [AMD] dsv4.1: Fix decode graph capture with quantized low-ratio activations (#46151)
Co-authored-by: Kevin Mi <kevin.mi@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 20:55:13 -07:00
Liangsheng Yin 3c6a6c8068 [Perf] dsv41: Run MegaGate routing in prefill (#46232) 2026-10-23 20:31:50 -07:00
Liangsheng Yin 17d92ea671 [Feat] openai: Parse DeepSeek text for the native endpoints through deepseek-recipe (#46219) 2026-10-23 20:17:37 -07:00
Richard WangandMick Qian 7cefa60765 [Perf] diffusion: Wait on the socket when the scheduler is idle (#46007)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-24 11:02:24 +08:00
25ba24e9a7 [Perf] diffusion: Build one LTX-2.3 AdaLN row per sample when every token shares the sigma (#46109)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 11:00:14 +08:00
42ac81c655 [Perf] diffusion: Route dense attention with a sub-tile side to FA4 on Blackwell (#46107)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 10:59:25 +08:00
jambow0320andShangming Cai de7055f596 [Refactor] pd: Consolidate sender and receiver implementations (#41140)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-24 10:50:10 +08:00
jacky.cheng 88ac7c02a6 [Fix] [AMD] diffusion: Fix the first-step denoise regression from compiling ENABLE_CK=0 BF16 attention (#45838) 2026-10-23 19:42:54 -07:00
Jackey Huaandhnyls2002 a77998e157 [Feat] openai: parse DeepSeek output through deepseek-recipe (#45907)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-23 19:23:22 -07:00
Liangsheng Yinandkpham-sgl 79701a2549 [Perf] dsv41: Read Engram rows from peer device shards (#46180)
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai>
2026-10-23 19:01:26 -07:00
Yihao WangandClaude Opus 5 327c906764 [Perf] [XPU] diffusion: Avoid QKV copies in flash attention (#41944)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-24 09:33:26 +08:00
Yihao WangandClaude Opus 5 5bef70c0a2 [Perf] flux2: Build the single-block output projection without a cat (#41943)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-24 09:33:26 +08:00
MickandMick Qian 1d4dcc81cc [Perf] diffusion/registry: Reduce model discovery work at startup (#46043)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-24 09:30:04 +08:00
47c0791ad1 [Perf] diffusion: Write uncompressed PNG responses without Pillow's row filters (#46099)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:28:55 +08:00
bbc77e7db4 [Perf] diffusion: Cut LTX-2.3 per-block elementwise work (contiguous AdaLN values, gate-folded masks, vectorized head gate) (#46110)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:27:39 +08:00
4e2d5b17ad [Perf] diffusion: Keep every component resident in auto mode when memory allows (#45978)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:26:13 +08:00
be8600d02d [Fix] diffusion: Step MOVA latents outside the serial-CFG branch (#45993)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:23:54 +08:00
a08bb0ec9e [Perf] diffusion: Run LTX-2 stage 2 on a premerged resident DiT (#46108)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:23:08 +08:00
e66c9e981d [Perf] diffusion: Turn on breakable CUDA graphs by default for LTX-2/LTX-2.3 two-stage serving (#45977)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:22:06 +08:00
2e6d87e2ca [Perf] diffusion: Serve FA4 head sizes on Blackwell instead of falling back to sm80 SDPA (#46102)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:20:32 +08:00
386fa9ecfd [Perf] diffusion: Drop LTX-2's all-valid prompt mask under lossless quality (#46106)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:19:12 +08:00
5b8955e70f [Perf] diffusion: Shard LTX-2 stage 2 over the CFG ranks as Ulysses (#45965)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:17:41 +08:00
738e4535ea [Perf] diffusion/qwen-image-2.1: Keep an image prefix's K/V in the packed buffer across steps (#46101)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:14:10 +08:00