Commit Graph
6313 Commits
Author SHA1 Message Date
minke.yu d6f1e1efbd rebase(hicache): per-op async backup on top of #46884 + #48782
Net async change (from exp-async-backup-perop-20261107) applied onto
b300-pick-46884-20261107. Runtime files applied cleanly; the new tests and
the timeline probe are added as-is.

Interaction to validate on hardware: #46884 publishes prefill read-done before
the replay, so the scheduler-stream fence that the index copy waits on is
shorter; the async worker's wait should shrink and the copy may overlap more of
the in-flight forward. Re-baseline the 4-arm A/B on this branch.
2026-11-07 22:04:33 +08:00
metamergebot 5f7be7fab4 [Perf] hicache: Publish prefill read-done when offload copies are fenced on the forward stream
The prefill shared-read-done fastpath lets the schedule stream run ahead of
the forward that writes a stashed chunk's KV instead of waiting for the whole
forward. It was disabled whenever hierarchical cache or a storage backend was
enabled, because host and storage offload copies used to be ordered only
behind the schedule stream and could have read unfinished KV.

The cache controller now orders its device-to-host copies behind the forward
stream through write_fence_stream, and its load-back H2D copies through
load_fence_stream, so that reason no longer holds when every KV offload copy
carries that fence. The scheduler records on the model runner whether that is
the case: hierarchical cache with a cache controller and no decode offload
manager, which has no such fence. The publish helper keeps the coarse
whole-forward barrier for LMCache and for unfenced offload, and publishes
read-done otherwise.

With the coarse barrier, a synchronous page-index copy issued while preparing
the next batch inherited the wait on the entire in-flight forward, leaving the
GPU idle between batches. With read-done published, that copy only waits for
the shared-table reads to finish; KV ordering is unchanged because the backup
copies still wait on the forward stream.

Tests: the CPU runner test covers the fenced hierarchical-cache and storage
cases publishing, and LMCache keeping the barrier even when hierarchical cache
is fenced. The GPU ordering test checks that a write-through backup of a chunk
stashed before its forward finished still copies the final KV.

Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com>
(cherry picked from commit bb1cb2dded6fb9352d98076280342b10684e061b)
2026-11-07 14:36:38 +08:00
minke.yu aa53670fcd observability: split flush_pending_backups host chain (off by default)
Adds the phase-3/phase-4 instrumentation from
handoff/prefill-flush-backups-validation-20261107.md:

* process_batch_result outer phases: flush_trace, load_snapshot,
  flush_backups, step_counters, metrics_report. process_residual now
  subtracts them, so the 37.5% residual is attributable.
* SGLANG_TRACE_BACKUP_DETAIL=1 (default 0) traces one flush:
  merge_ops / move_write (anchor + per-sidecar indices) / indices_cpu /
  index_sort / index_select / write_fence / build_l2 / submit_d2h
  (start-event wait, prepare, per-pool launch, ack events) / ack_enqueue,
  plus token/op/pool/device dimensions and ack queue depth.
* Per-flush resolution: one scheduler.cache_flush_backup span per
  non-empty flush, child of the loop span. Sub-phases never enter the
  loop aggregate, so the residual identity cannot double count.
* Empty vs non-empty flushes are counted on the loop.

All timings are scheduler host wall, not device transfer time.
2026-11-07 09:23:44 +08:00
ymk 7b3f34cbc0 test: cover async SWA state-payload call chain; clarify CP gate
Review follow-up for the v2 async state payload:

- P2: new tests use CustomTestCase (with a local fallback when sglang is not
  importable), matching the disaggregation/Mooncake test convention.
- P3: CP gate docstring now says all-CP-rank transfer *without DSA cache
  layer split* keeps the synchronous path.
- Add real call-chain coverage:
  * _send_kv_chunk -> sender.send -> add_transfer_request -> transfer_worker
    -> materialize_state_indices -> maybe_send_extra, with DSV4's four
    components (DSV4_REQUEST_STATE, REQUEST_ROW, SWA, SWA);
  * the worker materializes pinned tensors only after indices_event and
    fails loudly when a tensor has no event;
  * a second materialization without an event (staging retry) is a no-op;
  * real CUDA ordering: the pinned helper returns before a long kernel and
    the event-protected values match kv_to_page_indices.
- Verified on b300 (P container, overlay source): 9 passed in
  test_async_state_payload.py, 2 passed in the new
  TestMooncakeAsyncStateMaterialization class; the old batching suite has
  11 pre-existing failures that also fail on the image source.

The test-only batching suite requires the detail envs to be off when run in
the live pod (the inherited SGLANG_TRACE_MOONCAKE_DETAIL=1 exposes an existing
test fixture omission); noted, not changed here.
2026-11-07 00:22:57 +08:00
ymk 3b0baa3455 prefill: async SWA state-payload D2H (first version)
Implements the v2 plan in handoff/prefill-state-payload-async-review-20261106.md.

- mem_cache/common.py: add kv_to_page_indices_pinned() (pinned CPU buffer,
  non-blocking copy; no synchronous .cpu()).
- prefill.py: gate via should_async_state_payload() (last chunk, independent
  supports_async_state_indices, kv_shard_size==1, CP guard); only the
  _swa_payload request_window-is-None hot branch uses the pinned helper.
  Final segment asserts async state has an indices_event. Other state
  builders (mamba/full/DSA) stay synchronous.
- mooncake/conn.py: MooncakeKVSender advertises
  supports_async_state_indices; the worker materializes pinned state tensors
  to numpy only after indices_event.synchronize(), and fails loudly if a
  tensor arrives without an event.
- tests: guard/event/materialize/pinned-body unit tests + CUDA ordering test
  (skipped unless torch+CUDA+sglang are available).

No scheduler/transfer semantics change; rollback is the helper switch or
reverting this commit. 49 tests pass, 1 skipped locally.
2026-11-06 23:00:22 +08:00
ymk fe9c6574fe observability: trace KV transfer chunks end-to-end (M2)
Implements M2 of handoff/prefill-cpu-kv-overlap-design-20261106.md, behind
SGLANG_TRACE_MOONCAKE_DETAIL=1 (default off; reuses the existing trace stack).

- TransferKVChunk gains optional observation fields: chunk_seq, loop_seq,
  send_kind, enqueue/last_enqueue monotonic+wall stamps, queue_wait_accum_ns,
  queue_id, requeue_count, num_pages, approx_bytes, observe
- chunk_seq and enqueue timestamps are assigned at the real queue.put()
  point (after every early return), so skipped paths cannot leave holes
- Mooncake transfer worker measures queue wait, source-event wait, indices
  wait, KV/state/aux transfer, worker busy and residual; emits one
  mooncake.kv_chunk span per attempt (+ queue/source/indices wait children)
  and updates a per-room join summary
- request completion emits kv.request_join with chunk_count,
  last_chunk_queue_wait/worker, final_exposed_ms, completion_poll_ms,
  request_join_ms and first-enqueue-to-completion. First enqueue is stamped
  by the Mooncake manager at the real queue.put(); no ReqTimeStats field is
  added and the legacy queue_entry semantics are unchanged.
- nixl/mori add_transfer_request accept the observation kwarg (no-op there)

No scheduling/transfer semantics change; queue wait is excluded from worker
busy, and worker_busy excludes source/indices event waits. Unit tests: test_mooncake_chunk_detail.py (15 cases, including the
abort/worker interleaving and attempt-vs-total queue wait windows).
2026-11-06 20:01:56 +08:00
ymk 72e556ec81 observability: split prefill run_batch/process_result into host phases (M1)
Implements M1 of handoff/prefill-cpu-kv-overlap-design-20261106.md:
per-loop detail phases behind SGLANG_TRACE_LOOP_DETAIL=1 (requires
SGLANG_BATCH_STAGE_TIMING=1 + SGLANG_TRACE_LOOP_SPAN=1, only sampled loops).

- run_batch: early_send_host, resolve_inputs, model_forward_host,
  copy_result_submit (+ run_residual_ms)
- process_result: copy_done_wait, auxiliary_and_logprob, per_request_result,
  kv_state_payload/page_indices/sender_send by send_kind, report_stats
  (+ process_residual_ms)
- war_barrier is measured as its own loop phase (outside run_batch's span)
- repeated/per-request phases are summed attributes; single phases are child
  spans, so span count never grows with n_reqs/n_segments
- DP-skew join fields: pass_end_wall_ns, arrival_est_wall_ns = pass_end -
  mlp_sync, previous-loop phase summary (sglang.prev_*, published only for
  consecutive loop_seq; prev_detail_missing marks a gap)

No CUDA syncs added; when disabled every hook is a module-level bool check.
Unit tests: test_loop_detail_phases.py (10 cases).
2026-11-06 20:01:56 +08:00
minke.yu 23053730fc observability: count SWA gate rejections per prefill admission
Review follow-up. match_kv_cache also runs on waiting-queue probes
(calc_priority), PD decode _match_prefix_and_lock and HiCache rematches.
Emitting the counter there counted the same gap once per scheduling round
and attributed "re-prefilled" tokens to decode engines.

- match_kv_cache now only records req.full_kv_hit_length (plus the match-time
  gap); it no longer emits.
- the metric is emitted once in PrefillAdder._commit_prefill_admission, where
  the accepted prefix is final after any host load-back.
- rename the request counter to swa_gate_reject_admissions_total and document
  that a retracted and re-admitted request is counted again (it is re-prefilled).
- pre-create zero-valued children so gate-off deployments expose the series
  before the first rejection.
- tests: repeated match does not emit, admission emits once, a retry counts
  again, and host load-back closes the gap.

Not runtime verified (no GPU run on this build).
2026-11-06 13:18:07 +08:00
ymk bf8cf7f1dc hicache: pipeline prefetch/backup IO lanes + stage timing
- prefetch_io_aux_func: allow up to SGLANG_HICACHE_PREFETCH_IO_THREADS
  transfers in flight (default 1 = legacy behaviour); PrefetchAck is still
  emitted strictly FIFO so the per-rank ack stream (and the TP all_reduce in
  prefetch_sync_thread_func) is unchanged.
- backup_thread_func: same pipelining for L2->L3 writes via
  SGLANG_HICACHE_BACKUP_IO_THREADS.
- SGLANG_HICACHE_TIMING=1: emit [HC_TIMING] prefetch/backup queue, inflight,
  p50/p95 per 100 ops (the prefetch/backup lanes were previously single
  threaded; this is what caps L3<->L2 at ~1 op at a time).
2026-11-06 13:18:07 +08:00
minke.yu c911d1b572 observability: expose SWA prefix-match gate rejections
The unified-cache SWA component gates prefix matching on a live sliding
window: when the window is missing, the match stops before an otherwise
available Full-KV prefix. The existing hit counters do not show the tokens
lost to that gate, which makes the gate-on/gate-off trade-off invisible.

Record the gap between the longest Full-KV match and the accepted prefix
(full_kv_hit_length - (device_prefix_len + host_hit_length)) on the Req and
export it as:
  sglang:swa_gate_reject_tokens_total
  sglang:swa_gate_reject_reqs_total

This is the token volume that will be re-prefilled because a component
validator (SWA window; Mamba where present) rejected the match. With the
gate enabled it measures the hit length lost to a missing window; with the
gate disabled it only captures the host-only / load-back-gap fallback.

Includes unit tests for the gap and the host-hit case. Not runtime verified
yet (no GPU run on this build).
2026-11-06 10:33:52 +08:00
minke.yu 145e05c44d observability: fix scheduler no-batch accounting 2026-11-05 16:42:39 +08:00
minke.yu 5d5870f3ea observability: add scheduler loop span and no-batch accounting
Adds an opt-in scheduler.loop span (SGLANG_TRACE_LOOP_SPAN) with recv /
get_next / run_batch / process_result / kv_send children. The span is only
created for sampled loops that have pending work; fully idle loops stay in
the counters so the 714 pass/s idle spin never floods Jaeger.

No-batch accounting is always on with SGLANG_BATCH_STAGE_TIMING:
- sglang:scheduler_no_batch_{seconds,total}{dp_rank,reason,forward_overlap}
- sglang:scheduler_dp_lockstep_idle_seconds_total{dp_rank,forward_overlap}
with the design's reason priority (result_backpressure, deferred_input,
waiting_bootstrap, waiting_prefetch, admission_blocked, waiting_pd,
fully_idle, no_ready_other) and local_batch_kind/global_outcome from the DP
sync. DP lockstep idle is counted separately from pure no-batch wait.

loop gpu_ms reuses the existing DeviceTimer intervals (no new CUDA events);
the forward.gpu lane is optional and approximate. Pass/recv spans become
children of the loop span when it is active.

Tests: test_loop_span_accounting (5) and the existing stage-timing suite (8)
pass.
2026-11-05 15:00:23 +08:00
minke.yu cb7988f849 observability: measure storage prefetch ready latency (hit/miss)
Under --hicache-storage-prefetch-policy wait_complete a request is not
admitted until its L3 lookup resolves, and a miss pays the same query round
trip as a hit. The scheduler only exposed a skip counter, so the actual wait
was unknown.

Add sglang:storage_prefetch_ready_seconds{dp_rank, result=hit|miss}:
  * stamp cache_request_handle on its first in-flight check,
  * observe at the moment check_prefetch_progress() turns usable, split by
    whether prefetch_loaded_tokens_by_reqid ended up > 0,
  * clear the stamp on revoke / abort / accounting discard.

Buckets cover 1 ms .. 30 s. Gated by SGLANG_BATCH_STAGE_TIMING (off by
default). The measurement is ~2 dict ops + one observe per request, not per
pass.

Tests: test/registered/unit/observability/test_batch_telemetry_stage_timing.py
adds the hit/miss split assertion (9 tests, all passing).
2026-11-04 17:26:41 +08:00
minke.yu 0866012b80 observability: split scheduler recv_requests into pull/broadcast/unwrap
Runtime accounting on the prefill service (64 x 14.6k-token requests,
8-way concurrency) showed the non-forward wall time is dominated by the
scheduler receive path, not by radix match:

  forward (extend+idle) 58.2% | get_next 6.9% | recv_requests 25.6%
  process_batch_result 1.8% | run_batch overhead 0.9% | other ~6%

The existing pass span only covers get_next, so it cannot explain that
25.6%.  Split it with the same opt-in gate:

  * sglang:recv_requests_seconds{dp_rank, part} (total/pull/broadcast/
    unwrap/mm), 5 new series per rank
  * sglang:recv_requests_msgs_total, sglang:recv_requests_prompt_tokens_total
  * OTLP span scheduler.recv_requests with the same breakdown, sampled
    deterministically by a per-process counter (SGLANG_RECV_SPAN_SAMPLE,
    default 0.05; SGLANG_RECV_SPAN=0 disables the span only)

Note on scope: tokenization itself does NOT run in the scheduler. The
tokenizer workers (--tokenizer-worker-num) produce TokenizedGenerateReqInput
and the scheduler receives the already-tokenized ids. What this measures is
ZMQ pull + decode, the cross-rank broadcast of the pickled objects, and
unwrap_pickle_fields() of the token-id buffers -- which is why large prompts
make recv_requests expensive.

Tests: test/registered/unit/observability/test_batch_telemetry_stage_timing.py
now also covers the recv histogram parts, counters, zero-message passes and
deterministic recv-span sampling (8 tests total, all passing).
2026-11-04 16:11:13 +08:00
minke.yu 85f4f9576c scheduler: add opt-in per-pass prefill stage timing (SGLANG_BATCH_STAGE_TIMING)
Motivation: on the PD prefill node the wall-clock budget is dominated by
get_next_batch_to_run (~50% of the hour on 2026-11-02 16:00-17:00), and
forward_overlap=partial is only 21-87 s/h, i.e. it does not overlap the GPU
forward. We know the total but not which segment inside it costs: radix
match, PrefillAdder admission, the HiCache event collective, the bootstrap
all-reduce, or the DP metadata gather.

Measurement design (one sample per scheduling pass, including passes that
produce no batch; see handoff/prefill-batch-stage-timing-20261103.md v2/v3):

  L1  7 per-pass histograms:
      sglang:batch_prefill_{get_next,match,adder,hicache_events,
      bootstrap,mlp_sync,other}_seconds
      other = total - (match + adder + hicache + bootstrap + mlp_sync),
      clamped to >= 0. Nested stages are not double counted.

  L3  5 per-pass counters:
      candidates / skipped_prefetch / admitted / resumed_chunk / new_chunk

  L2  a dedicated per-pass span scheduler.prefill.pass (not the batch span),
      emitted from end_pass() on every pass. Its sampling decision is a pure
      function of mlp_sync_seq, so a sampled gather round is sampled on all
      DP ranks. Idle ranks and ranks without a local batch still emit, which
      is exactly what the cross-rank skew analysis needs. No links.

Hooks are installed in both scheduling loops:
  * get_next_disagg_prefill_batch_to_run() for --disaggregation-mode prefill
    (hicache / bootstrap / mlp_sync segments), and
  * get_next_batch_to_run() for standalone prefill (hicache / mlp_sync),
so the instrumentation can be validated on ds41-prefill-dp4 without a decode
peer. match/adder/counters live in the shared get_new_batch_prefill path.

Semantics:
  * match times only init_next_round_input; check_prefetch_progress counts as
    skipped_prefetch, never as match.
  * bootstrap and hicache_events are cross-rank sync points measured at the
    outer call, not attributed to local CPU work.
  * dp_gather_ms comes from the existing MLPSyncBatchInfo.sync_wait_seconds;
    sglang:dp_attention_sync_wait_seconds is reused, no duplicate metric.
    The value is cleared before each outer call and mlp_sync_seq only
    increments when a gather actually ran, so stale values cannot leak.
  * resumed vs new chunked requests are captured separately: had_chunked is
    recorded at pass start (before process_prefill_chunk/admission can
    consume the last chunk) and new_chunk from adder.new_chunked_req. This
    fixes both the first-chunk and last-chunk mislabels.

Opt-in:
  * SGLANG_BATCH_STAGE_TIMING=1 (default 0);
    SGLANG_BATCH_PASS_SPAN=1 / SGLANG_BATCH_PASS_SPAN_SAMPLE=0.05.
  * When off every hook is one module-level bool check; no logging, no CUDA
    calls, no new collectives.

Tests: test/registered/unit/observability/test_batch_telemetry_stage_timing.py
covers first/middle/last chunk labelling, all-prefetch-skip passes without a
batch, stale-gather clearing, rank-deterministic pass-span sampling, and the
disabled no-op path. py_compile passes on all touched files.
2026-11-04 14:53:29 +08:00
Florian Wunderlich d47a99b2ad test: import SWA eviction helper directly
(cherry picked from commit 5729f61e175bde3ef2b67dc3a95abb1042f4bf1b)
2026-11-04 10:41:52 +08:00
Florian Wunderlich 2ba9b8431a Fix SWA eviction behind the replay-trimmed prefix frontier
(cherry picked from commit f399d1e9a2bf7c33c7244ecf405e756292ddaf2b)
2026-11-04 10:41:52 +08:00
zicai.pu ee9170e7bb fix(mooncake): bind IntraNode batches to the bootstrap CUDA device
Sender and executor threads can enter synchronous IntraNode transfers with
another current CUDA device. Keep the explicit bootstrap ordinal, check it
against the published model device, and scope the native batch call to that
device while restoring the calling thread on success or failure.

Ordinary transports, force-TCP and non-CUDA devices keep their existing
path. Missing or inconsistent CUDA identity is rejected before native IO.

Add 13 CPU regressions importing the production wrapper, including real
constructor calls with a mocked SDK, concurrent thread-local device states,
error restoration and transport/device compatibility. All 13 pass; two
selected tests fail on the unchanged wrapper at behavior assertions. Native
and CUDA boundaries are mocked, so this does not certify physical CUDA IPC
or attribute the separate container-PID collision to this change.
2026-10-30 11:42:19 +08:00
zicai.pu 17505e6cf6 hicache: align prefetch and host restore boundaries
Use the request radix-key cap for L3 prefetch so bounded replay cannot
split a queried trailing-state checkpoint at the later rematch. Reject
Host-only SWA-gap candidates before they mask a complete ancestor, while
preserving device matching and the final load-back completeness guard.

Add CPU regressions that import the real scheduler, cache and TreeCore
methods. All 13 test methods pass; the same tests against the unmodified
scheduler/SWA code fail six methods (ten assertions including subtests).
The checks use synthetic cache state and controller boundaries, not GPU
or live Store execution.
2026-10-30 11:42:19 +08:00
zicai.pu 81f7398855 fix(hicache): reject unproven trailing restore endpoints
Do not invent shorter checkpoints when a trailing pool has no restorable-prefix inventory. Reject partial allocation and vote no common endpoint while preserving the collective shape and plain-KV contiguous behavior.

Add negative controls for missing inventory and retain pure-KV coverage. Validation: 33 cache/PP CPU tests passed with an external Mac native-hash import adapter using the repository Python hash oracle; repository pre-commit checks passed.
2026-10-30 11:42:19 +08:00
zicai.pu 0a56a5f6e7 fix(hicache): preserve sparse restore checkpoints under host pressure
Choose a common validated trailing-state endpoint before partial host allocation, retain that endpoint set across query synchronization, and size auxiliary staging from the selected length. Assumed trailing hits prove only their original endpoint.

Add controller/cache CPU regressions and a PP checkpoint-vote control. Validation: 14 targeted CPU tests and repository pre-commit checks passed.
2026-10-30 11:35:29 +08:00
Liangsheng Yin 54df642fe6 [Perf] dsv41: Translate SWA locations once per prefill graph step (#46229) 2026-10-23 21:33:15 -07:00
Yihao WangandClaude Opus 5.5 a7183a8246 [Feat] spec: Let out-of-tree platforms provide EAGLE draft graph runners (#45145)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 21:28:56 -07:00
Yihao WangandClaude Opus 5.5 af3f9e4c8f [Feat] spec: Let out-of-tree platforms provide their own speculative worker (#45144)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 21:24:58 -07:00
Liangsheng Yin 3b95120f00 [Fix] dsv41: Revert MegaGate routing in prefill (#46242) 2026-10-23 21:15:38 -07:00
55f23a8b59 [Fix] [AMD] dsv4.1: Fix decode graph capture with quantized low-ratio activations (#46151)
Co-authored-by: Kevin Mi <kevin.mi@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 20:55:13 -07:00
Liangsheng Yin 3c6a6c8068 [Perf] dsv41: Run MegaGate routing in prefill (#46232) 2026-10-23 20:31:50 -07:00
Liangsheng Yin 17d92ea671 [Feat] openai: Parse DeepSeek text for the native endpoints through deepseek-recipe (#46219) 2026-10-23 20:17:37 -07:00
jambow0320andShangming Cai de7055f596 [Refactor] pd: Consolidate sender and receiver implementations (#41140)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-24 10:50:10 +08:00
Jackey Huaandhnyls2002 a77998e157 [Feat] openai: parse DeepSeek output through deepseek-recipe (#45907)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-23 19:23:22 -07:00
Liangsheng Yinandkpham-sgl 79701a2549 [Perf] dsv41: Read Engram rows from peer device shards (#46180)
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai>
2026-10-23 19:01:26 -07:00
Yihao WangandClaude Opus 5 327c906764 [Perf] [XPU] diffusion: Avoid QKV copies in flash attention (#41944)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-24 09:33:26 +08:00
Yihao WangandClaude Opus 5 5bef70c0a2 [Perf] flux2: Build the single-block output projection without a cat (#41943)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-24 09:33:26 +08:00
bbc77e7db4 [Perf] diffusion: Cut LTX-2.3 per-block elementwise work (contiguous AdaLN values, gate-folded masks, vectorized head gate) (#46110)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:27:39 +08:00
Jackey Huaandhnyls2002 e22a530859 [Feat] openai: Render DeepSeek-V4/V4.1 chat prompts with deepseek-recipe (#45315)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-23 18:05:35 -07:00
Liangsheng Yin 79e5c4bb9b [Perf] dsv41: Trim per-step DSpark decode overhead (#46176) 2026-10-23 17:24:06 -07:00
Baizhou Zhangandsid-rp 6a319e7b34 [Fix] muse: Support Responses namespace tools and defer_loading (#44685)
Co-authored-by: sid-rp <sidrp@meta.com>
2026-10-23 16:48:08 -07:00
0d592c475b [Fix] rust-server: Keep a transfer backend's own bootstrap port on prefill (#46066)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-23 16:47:14 -07:00
Cheng Wan 855c092deb [Fix] inkling: Keep BF16 LM-head operands with FP32 output (#45335) 2026-10-23 15:51:44 -07:00
Baizhou Zhang ea226cf782 [Feat] glm5.3flash: Support padded CP BCG and prefix replay with PD/DFlash2 (#43774) 2026-10-23 14:21:31 -07:00
27ac614f05 [Feat] parser: Add checkpoint-driven response-template adapters (#40479)
Signed-off-by: Yoni Gozlan <yonigozlan@users.noreply.github.com>
Co-authored-by: Yoni Gozlan <yonigozlan@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
2026-10-23 14:05:54 -07:00
em-dreyer 4542fc8f59 [Fix] mistral3: Subclass LlavaForConditionalGeneration instead of wrapping it (#45656) 2026-10-23 14:05:32 -07:00
1d605b7ece [CI] vlm: Harden fixture cleanup after setup and shutdown failures (#45461)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Alison Shao <a.shao@wustl.edu>
2026-10-23 13:20:06 -07:00
005333d029 [Refactor] radix-tree: Make each cache action a typed CacheAction implementation (#45444)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: Jialin <Jialin@users.noreply.github.com>
2026-10-23 12:49:30 -07:00
Cheng Wan d0022e1df3 [Fix] lora: Preserve MoE projection identity and routing lifetimes (#45681) 2026-10-23 12:32:55 -07:00
Cheng Wan ac3db731b3 [Fix] moe: Preserve CUTLASS routing metadata and execution budgets (#45917) 2026-10-23 12:32:05 -07:00
Cheng Wan 8b9b992e41 [Perf] moe: Preserve Marlin workspace and route-count budgets (#45462) 2026-10-23 12:31:17 -07:00
Cheng Wan e6451c1a95 [Fix] gguf: Preserve MoE routing, weighting and allocation budgets (#45916) 2026-10-23 12:29:51 -07:00
Cheng Wan 1340044e97 [Fix] deepseek-v4: Preserve decoder-tail token-count ownership (#45915) 2026-10-23 12:28:43 -07:00
9d7ad308bd [Fix] [AMD] Quark: fix Quark block-FP8 loading in DeepSeek loader (#40776)
Co-authored-by: clintg6 <7388379+clintg6@users.noreply.github.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: Bowen Bao <bowenbao@amd.com>
Co-authored-by: Hubert Lu <Hubert.Lu@amd.com>
2026-10-23 12:11:21 -07:00