minke.yu
d6f1e1efbd
rebase(hicache): per-op async backup on top of #46884 + #48782
...
Net async change (from exp-async-backup-perop-20261107) applied onto
b300-pick-46884-20261107. Runtime files applied cleanly; the new tests and
the timeline probe are added as-is.
Interaction to validate on hardware: #46884 publishes prefill read-done before
the replay, so the scheduler-stream fence that the index copy waits on is
shorter; the async worker's wait should shrink and the copy may overlap more of
the in-flight forward. Re-baseline the 4-arm A/B on this branch.
2026-11-07 22:04:33 +08:00
minke.yu
97c087778b
observability: harden flush span attributes
...
* Normalize the host-pool layout to a string (a None attribute would make
the OTel SDK drop the whole flush span).
* Drop any remaining None-valued flush attributes for the same reason.
2026-11-07 11:35:03 +08:00
minke.yu
aa53670fcd
observability: split flush_pending_backups host chain (off by default)
...
Adds the phase-3/phase-4 instrumentation from
handoff/prefill-flush-backups-validation-20261107.md:
* process_batch_result outer phases: flush_trace, load_snapshot,
flush_backups, step_counters, metrics_report. process_residual now
subtracts them, so the 37.5% residual is attributable.
* SGLANG_TRACE_BACKUP_DETAIL=1 (default 0) traces one flush:
merge_ops / move_write (anchor + per-sidecar indices) / indices_cpu /
index_sort / index_select / write_fence / build_l2 / submit_d2h
(start-event wait, prepare, per-pool launch, ack events) / ack_enqueue,
plus token/op/pool/device dimensions and ack queue depth.
* Per-flush resolution: one scheduler.cache_flush_backup span per
non-empty flush, child of the loop span. Sub-phases never enter the
loop aggregate, so the residual identity cannot double count.
* Empty vs non-empty flushes are counted on the loop.
All timings are scheduler host wall, not device transfer time.
2026-11-07 09:23:44 +08:00
ymk
bf8cf7f1dc
hicache: pipeline prefetch/backup IO lanes + stage timing
...
- prefetch_io_aux_func: allow up to SGLANG_HICACHE_PREFETCH_IO_THREADS
transfers in flight (default 1 = legacy behaviour); PrefetchAck is still
emitted strictly FIFO so the per-rank ack stream (and the TP all_reduce in
prefetch_sync_thread_func) is unchanged.
- backup_thread_func: same pipelining for L2->L3 writes via
SGLANG_HICACHE_BACKUP_IO_THREADS.
- SGLANG_HICACHE_TIMING=1: emit [HC_TIMING] prefetch/backup queue, inflight,
p50/p95 per 100 ops (the prefetch/backup lanes were previously single
threaded; this is what caps L3<->L2 at ~1 op at a time).
2026-11-06 13:18:07 +08:00
c677171c1d
[Feat] hicache: Support MLA DCP with HiCache L3(file/mooncake) ( #41187 )
...
Co-authored-by: Codex <codex@openai.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: alphabetc1 <2508695655@qq.com >
Co-authored-by: Venkat Raman <venkat.raman@verda.com >
2026-10-23 23:09:04 +08:00
3b0003dac1
[Fix] HiCache: stale host KV from write-through copies racing the in-flight forward ( #42701 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu >
2026-10-22 14:46:51 -07:00
lpdink and Zhangheng
4a651146de
[Feat] hicache: Notify storage backends when the initial host-pool registration burst completes ( #44030 )
...
Co-authored-by: Zhangheng <hzh0425@apache.org >
2026-10-20 14:09:13 +08:00
f75232cb42
[Fix] hicache: Order write-back host copies after the forward that produced the KV ( #44800 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com >
2026-10-17 01:31:40 -07:00
173e0f2e9b
[Feat] hicache: Add opt-in tail-only SWA backups in buffer mode ( #44173 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com >
2026-10-15 12:02:59 -07:00
Shuwen Wang
ef3e5c83c5
[HiCache] Keep storage workers alive on backend exceptions ( #43550 )
2026-10-11 22:26:56 +08:00
Ping Qiu and Zhiqiang Xie
685d4d453d
[HiCache] Add SeaweedFS L3 storage backend ( #42399 )
...
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu >
2026-10-07 15:05:37 -07:00
08aad0c196
[HiCache] Manage buffer-mode backups per storage pool ( #42633 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com >
2026-10-06 16:45:05 -07:00
937c0a6cfe
Fix unified HiCache physical transfers ( #39479 )
...
Co-authored-by: yhzhuang <yhzhuang@fb.com >
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com >
Co-authored-by: Yonghao Zhuang <yhzhuang@users.noreply.github.com >
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
2026-10-04 00:38:56 -07:00
Cheng Wan
10edb05e72
[Refactor] Let the remaining placement consumers read the parallel context ( #42348 )
2026-10-03 14:59:32 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
00bcc25f6c
[HiCache] Fix DeepSeek-V4 storage backend crash from missing storage_format_tag ( #42215 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-02 22:27:57 -07:00
cdadb1c79a
[unified-memory] Token-major dense views for the unified memory pool (3/7) ( #38592 )
...
Co-authored-by: Caihua Li <caihua.li@bytedance.com >
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-01 17:09:38 -07:00
78b382b1ad
Support unified memory decode host pools ( #39478 )
...
Co-authored-by: yhzhuang <yhzhuang@fb.com >
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com >
Co-authored-by: Yonghao Zhuang <yhzhuang@users.noreply.github.com >
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-09-24 11:02:15 -07:00
Shuwen Wang
2bc435d32e
[HiCache] fix: bound the controller reset join so a stalled storage thread cannot hang the scheduler ( #40312 )
2026-09-23 10:52:03 +08:00
Liangsheng Yin
4ce23542bf
[HiCache] Remove the unused HiRadixCache ( #40787 )
2026-09-22 16:17:12 -07:00
Yuhan Zhou
15eba3b464
Feat: Add TensorCast storage as a new HiCache backend ( #27265 )
2026-09-22 10:19:04 +08:00
Cheng Wan
506698761d
[unified-memory] Hierarchical cache for every unified pool shape ( #37507 )
2026-09-21 16:50:37 -07:00
Shuwen Wang and Seokhoon Kang
9f3d275940
[HiCache] Fix sparse hybrid transfer layer IDs ( #37870 )
...
Co-authored-by: Seokhoon Kang <sh.kang@postech.ac.kr >
2026-09-20 16:24:24 +08:00
Cheng Wan
d0730a0e8b
Give the attention-DP width and rank one home ( #40067 )
2026-09-18 17:34:33 -07:00
Michele Palazzi and Teng Ma
329ffc89b9
[Mooncake] Fix silent SSD offload corruption when TP/PP ranks share ssd_offload_path ( #31926 )
...
Signed-off-by: Michele Palazzi <sysdadmin@m1k.cloud >
Co-authored-by: Teng Ma <sima.mt@alibaba-inc.com >
2026-09-17 10:51:03 +08:00
Shuwen Wang and Zhangheng
6b33338242
[HiCache] Forward prefix metadata to v2 storage calls ( #39567 )
...
Co-authored-by: Zhangheng <hzh0425@apache.org >
2026-09-16 14:05:42 +08:00
Yanbin Jiang and Shuwen Wang
0b415fa573
[HiCache][LoRA] Isolate storage pages by extra key ( #38577 )
...
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com >
2026-09-12 18:18:14 +08:00
James
165d8dd177
[NPU][Hicache] Add Ascend Memcache Hicache L3 storage backend ( #38827 )
2026-09-11 18:54:00 +08:00
James
209654c420
[NPU][Hicache] Optimize HiCache L2 IO with Memfabric acc_offload ( #38826 )
2026-09-11 00:23:36 +08:00
Zhiqiang Xie
a480f388b2
[HiCache] L3 storage prefetch lifecycle metrics and cross-tier attribution fixes ( #37503 )
2026-09-03 16:00:04 -07:00
Alex Nails and Alison Shao
28262c20df
[CI][RFC] Replace black-jupyter with ruff-format ( #37210 )
...
Co-authored-by: Alison Shao <a.shao@wustl.edu >
2026-09-02 19:46:08 -07:00
Zhiqiang Xie
f611e0c073
[HiCache] Fence load-back behind the forward stream ( #36738 )
2026-08-28 11:12:51 -07:00
cctry
aa718f7343
Refactor HiCache host pool management ( #36232 )
2026-08-25 16:31:57 -07:00
829138a31e
[HiCache] Fix PP inconsistency with HiCache L3 ( #22607 ) ( #27010 )
...
Co-authored-by: ybyang <ybyang7@iflytek.com >
Co-authored-by: hzh0425 <hzh0425@apache.org >
Co-authored-by: shangmingc <csmthu@gmail.com >
Co-authored-by: 晟海 <huangtingwei.htw@antgroup.com >
2026-08-25 20:49:57 +08:00
Shuwen Wang and hzh0425
41c018a9ec
[UnifiedTree] feat: support runtime attach/detach ( #35269 )
...
Co-authored-by: hzh0425 <hzh0425@apache.org >
2026-08-19 22:49:08 +08:00
Zhiqiang Xie
977412ae61
[HiCache] Buffer-only mode for HiCache host memory layer ( #34798 )
2026-08-18 19:21:24 -07:00
cctry
8922bb98e2
refactor(hicache): flatten L2 transfer execution ( #34793 )
...
GB300 test fails unrelated
2026-08-16 00:33:34 -07:00
Zhiqiang Xie
f6de147b8d
Remove revoke queue after hit-then-alloc refactoring ( #33613 )
2026-08-06 03:10:10 -07:00
8e11feb68e
[HiCache] Support packed and sidecar draft caches for MTP/EAGLE/DSpark ( #30393 )
...
Co-authored-by: hjzhang <hjzhang89.gmail.com>
Co-authored-by: Zhangheng <hzh0425@apache.org >
Co-authored-by: shuwenn <47200617+alphabetc1@users.noreply.github.com >
2026-08-06 14:31:11 +08:00
Zhiqiang Xie
106bcc1293
Observability enhancement for HiCache ( #32388 )
2026-08-05 14:13:06 -07:00
Bob Chen and Zhiqiang Xie
7cd55c6818
[HiCache] Optimize L2 mem allocation when cache miss in L3 ( #19320 )
...
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu >
2026-07-16 15:00:53 -07:00
095a817612
[Bugfix][HiCache] measure load-back duration with CUDA events ( #26411 )
...
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com >
Co-authored-by: vuuihc <vuuihc@users.noreply.github.com >
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu >
2026-07-16 00:11:04 -07:00
Cheng Wan
e703f9e566
[refactor] Adopt get_parallel() everywhere and close out the parallel wrapper surface ( #30492 )
2026-07-09 02:09:39 -07:00
AMD-yanfeiwang and Duyi-Wang
d74619b373
[AMD] Enable JIT staged HiCache write-back and fix CPU-index crash ( #28534 )
...
Co-authored-by: Duyi-Wang <duyi.wang@amd.com >
2026-07-09 01:22:37 -07:00
Zhiqiang Xie
f19246e59a
[HiCache] write_back policy refinement ( #29817 )
2026-07-02 12:16:03 -07:00
13dc5f2dc7
[HiCache][AMD] Add UMBP tiered DRAM + SSD L3 storage backend with hugepage host allocator ( #25377 )
...
Co-authored-by: TianDi101 ditian12@amd.com
Co-authored-by: Niko Ma nima@amd.com
Co-authored-by: Wu, Yutong yutong.wu@amd.com
Co-authored-by: figo fizhang@amd.com
Co-authored-by: AMD-yanfeiwang <yanfei.wang@amd.com >
Co-authored-by: Zhangheng <hzh0425@apache.org >
2026-07-01 22:21:37 +08:00
huangtingwei
5e1ccd9320
[HiCache] Optimize HiCache hash generation with bulk token byte conversion ( #28287 )
2026-07-01 15:44:23 +08:00
da0f4f6f92
[HiCache] remove large host mem constraint ( #28614 )
...
Co-authored-by: Teng Ma <stmatengss@gmail.com >
Co-authored-by: Vladislav Nosivskoy <vladnosiv@gmail.com >
2026-06-26 16:20:48 -07:00
shuwenn
ff1fc1fbdf
[mem_cache][5/N] refactor: extract host KV cache base layer into pool_host package ( #27273 )
2026-06-20 20:44:08 +08:00
shuwenn
941a11ada3
[HiCache] refactor: remove unused transfer buffer ( #26003 )
2026-06-19 09:46:04 +00:00
huangtingwei and hzh0425
6b7ecca663
[HiCache]Support hybrid pool staged H2D kernel ( #28434 )
...
Co-authored-by: hzh0425 <hzh0425@apache.org >
2026-06-19 09:48:03 +08:00