Commit Graph
136 Commits
Author SHA1 Message Date
minke.yu d6f1e1efbd rebase(hicache): per-op async backup on top of #46884 + #48782
Net async change (from exp-async-backup-perop-20261107) applied onto
b300-pick-46884-20261107. Runtime files applied cleanly; the new tests and
the timeline probe are added as-is.

Interaction to validate on hardware: #46884 publishes prefill read-done before
the replay, so the scheduler-stream fence that the index copy waits on is
shorter; the async worker's wait should shrink and the copy may overlap more of
the in-flight forward. Re-baseline the 4-arm A/B on this branch.
2026-11-07 22:04:33 +08:00
minke.yu 97c087778b observability: harden flush span attributes
* Normalize the host-pool layout to a string (a None attribute would make
  the OTel SDK drop the whole flush span).
* Drop any remaining None-valued flush attributes for the same reason.
2026-11-07 11:35:03 +08:00
minke.yu aa53670fcd observability: split flush_pending_backups host chain (off by default)
Adds the phase-3/phase-4 instrumentation from
handoff/prefill-flush-backups-validation-20261107.md:

* process_batch_result outer phases: flush_trace, load_snapshot,
  flush_backups, step_counters, metrics_report. process_residual now
  subtracts them, so the 37.5% residual is attributable.
* SGLANG_TRACE_BACKUP_DETAIL=1 (default 0) traces one flush:
  merge_ops / move_write (anchor + per-sidecar indices) / indices_cpu /
  index_sort / index_select / write_fence / build_l2 / submit_d2h
  (start-event wait, prepare, per-pool launch, ack events) / ack_enqueue,
  plus token/op/pool/device dimensions and ack queue depth.
* Per-flush resolution: one scheduler.cache_flush_backup span per
  non-empty flush, child of the loop span. Sub-phases never enter the
  loop aggregate, so the residual identity cannot double count.
* Empty vs non-empty flushes are counted on the loop.

All timings are scheduler host wall, not device transfer time.
2026-11-07 09:23:44 +08:00
ymk bf8cf7f1dc hicache: pipeline prefetch/backup IO lanes + stage timing
- prefetch_io_aux_func: allow up to SGLANG_HICACHE_PREFETCH_IO_THREADS
  transfers in flight (default 1 = legacy behaviour); PrefetchAck is still
  emitted strictly FIFO so the per-rank ack stream (and the TP all_reduce in
  prefetch_sync_thread_func) is unchanged.
- backup_thread_func: same pipelining for L2->L3 writes via
  SGLANG_HICACHE_BACKUP_IO_THREADS.
- SGLANG_HICACHE_TIMING=1: emit [HC_TIMING] prefetch/backup queue, inflight,
  p50/p95 per 100 ops (the prefetch/backup lanes were previously single
  threaded; this is what caps L3<->L2 at ~1 op at a time).
2026-11-06 13:18:07 +08:00
c677171c1d [Feat] hicache: Support MLA DCP with HiCache L3(file/mooncake) (#41187)
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: alphabetc1 <2508695655@qq.com>
Co-authored-by: Venkat Raman <venkat.raman@verda.com>
2026-10-23 23:09:04 +08:00
3b0003dac1 [Fix] HiCache: stale host KV from write-through copies racing the in-flight forward (#42701)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
2026-10-22 14:46:51 -07:00
lpdinkandZhangheng 4a651146de [Feat] hicache: Notify storage backends when the initial host-pool registration burst completes (#44030)
Co-authored-by: Zhangheng <hzh0425@apache.org>
2026-10-20 14:09:13 +08:00
f75232cb42 [Fix] hicache: Order write-back host copies after the forward that produced the KV (#44800)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-17 01:31:40 -07:00
173e0f2e9b [Feat] hicache: Add opt-in tail-only SWA backups in buffer mode (#44173)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-15 12:02:59 -07:00
Shuwen Wang ef3e5c83c5 [HiCache] Keep storage workers alive on backend exceptions (#43550) 2026-10-11 22:26:56 +08:00
Ping QiuandZhiqiang Xie 685d4d453d [HiCache] Add SeaweedFS L3 storage backend (#42399)
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
2026-10-07 15:05:37 -07:00
08aad0c196 [HiCache] Manage buffer-mode backups per storage pool (#42633)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-06 16:45:05 -07:00
937c0a6cfe Fix unified HiCache physical transfers (#39479)
Co-authored-by: yhzhuang <yhzhuang@fb.com>
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
Co-authored-by: Yonghao Zhuang <yhzhuang@users.noreply.github.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
2026-10-04 00:38:56 -07:00
Cheng Wan 10edb05e72 [Refactor] Let the remaining placement consumers read the parallel context (#42348) 2026-10-03 14:59:32 -07:00
Mohammad Miadh AngkadandMohammad Angkad 00bcc25f6c [HiCache] Fix DeepSeek-V4 storage backend crash from missing storage_format_tag (#42215)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-02 22:27:57 -07:00
cdadb1c79a [unified-memory] Token-major dense views for the unified memory pool (3/7) (#38592)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-01 17:09:38 -07:00
78b382b1ad Support unified memory decode host pools (#39478)
Co-authored-by: yhzhuang <yhzhuang@fb.com>
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
Co-authored-by: Yonghao Zhuang <yhzhuang@users.noreply.github.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-09-24 11:02:15 -07:00
Shuwen Wang 2bc435d32e [HiCache] fix: bound the controller reset join so a stalled storage thread cannot hang the scheduler (#40312) 2026-09-23 10:52:03 +08:00
Liangsheng Yin 4ce23542bf [HiCache] Remove the unused HiRadixCache (#40787) 2026-09-22 16:17:12 -07:00
Yuhan Zhou 15eba3b464 Feat: Add TensorCast storage as a new HiCache backend (#27265) 2026-09-22 10:19:04 +08:00
Cheng Wan 506698761d [unified-memory] Hierarchical cache for every unified pool shape (#37507) 2026-09-21 16:50:37 -07:00
Shuwen WangandSeokhoon Kang 9f3d275940 [HiCache] Fix sparse hybrid transfer layer IDs (#37870)
Co-authored-by: Seokhoon Kang <sh.kang@postech.ac.kr>
2026-09-20 16:24:24 +08:00
Cheng Wan d0730a0e8b Give the attention-DP width and rank one home (#40067) 2026-09-18 17:34:33 -07:00
Michele PalazziandTeng Ma 329ffc89b9 [Mooncake] Fix silent SSD offload corruption when TP/PP ranks share ssd_offload_path (#31926)
Signed-off-by: Michele Palazzi <sysdadmin@m1k.cloud>
Co-authored-by: Teng Ma <sima.mt@alibaba-inc.com>
2026-09-17 10:51:03 +08:00
Shuwen WangandZhangheng 6b33338242 [HiCache] Forward prefix metadata to v2 storage calls (#39567)
Co-authored-by: Zhangheng <hzh0425@apache.org>
2026-09-16 14:05:42 +08:00
Yanbin JiangandShuwen Wang 0b415fa573 [HiCache][LoRA] Isolate storage pages by extra key (#38577)
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com>
2026-09-12 18:18:14 +08:00
James 165d8dd177 [NPU][Hicache] Add Ascend Memcache Hicache L3 storage backend (#38827) 2026-09-11 18:54:00 +08:00
James 209654c420 [NPU][Hicache] Optimize HiCache L2 IO with Memfabric acc_offload (#38826) 2026-09-11 00:23:36 +08:00
Zhiqiang Xie a480f388b2 [HiCache] L3 storage prefetch lifecycle metrics and cross-tier attribution fixes (#37503) 2026-09-03 16:00:04 -07:00
Alex NailsandAlison Shao 28262c20df [CI][RFC] Replace black-jupyter with ruff-format (#37210)
Co-authored-by: Alison Shao <a.shao@wustl.edu>
2026-09-02 19:46:08 -07:00
Zhiqiang Xie f611e0c073 [HiCache] Fence load-back behind the forward stream (#36738) 2026-08-28 11:12:51 -07:00
cctry aa718f7343 Refactor HiCache host pool management (#36232) 2026-08-25 16:31:57 -07:00
829138a31e [HiCache] Fix PP inconsistency with HiCache L3 (#22607) (#27010)
Co-authored-by: ybyang <ybyang7@iflytek.com>
Co-authored-by: hzh0425 <hzh0425@apache.org>
Co-authored-by: shangmingc <csmthu@gmail.com>
Co-authored-by: 晟海 <huangtingwei.htw@antgroup.com>
2026-08-25 20:49:57 +08:00
Shuwen Wangandhzh0425 41c018a9ec [UnifiedTree] feat: support runtime attach/detach (#35269)
Co-authored-by: hzh0425 <hzh0425@apache.org>
2026-08-19 22:49:08 +08:00
Zhiqiang Xie 977412ae61 [HiCache] Buffer-only mode for HiCache host memory layer (#34798) 2026-08-18 19:21:24 -07:00
cctry 8922bb98e2 refactor(hicache): flatten L2 transfer execution (#34793)
GB300 test fails unrelated
2026-08-16 00:33:34 -07:00
Zhiqiang Xie f6de147b8d Remove revoke queue after hit-then-alloc refactoring (#33613) 2026-08-06 03:10:10 -07:00
8e11feb68e [HiCache] Support packed and sidecar draft caches for MTP/EAGLE/DSpark (#30393)
Co-authored-by: hjzhang <hjzhang89.gmail.com>
Co-authored-by: Zhangheng <hzh0425@apache.org>
Co-authored-by: shuwenn <47200617+alphabetc1@users.noreply.github.com>
2026-08-06 14:31:11 +08:00
Zhiqiang Xie 106bcc1293 Observability enhancement for HiCache (#32388) 2026-08-05 14:13:06 -07:00
Bob ChenandZhiqiang Xie 7cd55c6818 [HiCache] Optimize L2 mem allocation when cache miss in L3 (#19320)
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
2026-07-16 15:00:53 -07:00
095a817612 [Bugfix][HiCache] measure load-back duration with CUDA events (#26411)
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: vuuihc <vuuihc@users.noreply.github.com>
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
2026-07-16 00:11:04 -07:00
Cheng Wan e703f9e566 [refactor] Adopt get_parallel() everywhere and close out the parallel wrapper surface (#30492) 2026-07-09 02:09:39 -07:00
AMD-yanfeiwangandDuyi-Wang d74619b373 [AMD] Enable JIT staged HiCache write-back and fix CPU-index crash (#28534)
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
2026-07-09 01:22:37 -07:00
Zhiqiang Xie f19246e59a [HiCache] write_back policy refinement (#29817) 2026-07-02 12:16:03 -07:00
inkcherryTianDi101 ditian12@amd.comNiko Ma nima@amd.comWu, Yutong yutong.wu@amd.comfigo fizhang@amd.comAMD-yanfeiwangZhangheng
13dc5f2dc7 [HiCache][AMD] Add UMBP tiered DRAM + SSD L3 storage backend with hugepage host allocator (#25377)
Co-authored-by: TianDi101 ditian12@amd.com
Co-authored-by: Niko Ma nima@amd.com
Co-authored-by: Wu, Yutong yutong.wu@amd.com
Co-authored-by: figo fizhang@amd.com
Co-authored-by: AMD-yanfeiwang <yanfei.wang@amd.com>
Co-authored-by: Zhangheng <hzh0425@apache.org>
2026-07-01 22:21:37 +08:00
huangtingwei 5e1ccd9320 [HiCache] Optimize HiCache hash generation with bulk token byte conversion (#28287) 2026-07-01 15:44:23 +08:00
da0f4f6f92 [HiCache] remove large host mem constraint (#28614)
Co-authored-by: Teng Ma <stmatengss@gmail.com>
Co-authored-by: Vladislav Nosivskoy <vladnosiv@gmail.com>
2026-06-26 16:20:48 -07:00
shuwenn ff1fc1fbdf [mem_cache][5/N] refactor: extract host KV cache base layer into pool_host package (#27273) 2026-06-20 20:44:08 +08:00
shuwenn 941a11ada3 [HiCache] refactor: remove unused transfer buffer (#26003) 2026-06-19 09:46:04 +00:00
huangtingweiandhzh0425 6b7ecca663 [HiCache]Support hybrid pool staged H2D kernel (#28434)
Co-authored-by: hzh0425 <hzh0425@apache.org>
2026-06-19 09:48:03 +08:00