bab4a808d8
[Perf] unified-memory: Allocate free pages from each sub-pool's own end first ( #45222 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com >
2026-10-20 13:24:09 -07:00
Cheng Wan
d318a546da
[Perf] unified-memory: Reduce allocation and SWA retirement overhead ( #44970 )
2026-10-18 05:40:25 -07:00
caihuali95 and Caihua Li
ba50ea3d77
[Perf] unified-memory: Free SWA segments without a host sync ( #43170 )
...
Co-authored-by: Caihua Li <caihua.li@bytedance.com >
2026-10-18 05:36:56 -07:00
SuperSong
ee30dc4d40
[Fix] unified-memory: Bound joint reclaim and preserve cached prefixes ( #39294 )
...
Bound joint FULL/SWA/Mamba reclaim so feasible allocations recover capacity without evicting unnecessary cached prefixes, and impossible demands preserve the cache. Use exact unsharded paged-extend demand and update allocation regression fixtures.
2026-10-17 04:46:48 -07:00
36cf028346
[Perf] unified-memory: Prewarm the compaction move at startup and copy whole page envelopes by page id ( #44592 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com >
2026-10-16 16:47:50 -07:00
caihuali95
8d229d96e0
[Fix] unified-memory: enabling trtllm_mha with decode graphs and speculative decoding ( #42930 )
2026-10-16 02:24:06 -07:00
Liangsheng Yin and Jingshu Chen
3a5d46e946
[Perf] pd: Stage decode-side KV preallocation lengths through pinned memory ( #38955 )
...
Co-authored-by: Jingshu Chen <96519051+elmo2019@users.noreply.github.com >
2026-10-16 00:34:56 -07:00
huangtingwei
18210ed849
[HiSparse] Add speculative decoding support with multi-step KV cache swapping ( #37771 )
2026-10-13 13:38:31 -07:00
01f3ed18a5
[unified memory pool] Prefer SWA reclaim quotas with a HiCache storage backend or on PD decode ( #43919 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com >
2026-10-13 11:58:24 -07:00
f2e9a2bed1
[unified memory pool] Allocate without transfer waits, pin floating SWA reservations, drop the HiCache startup asserts ( #43206 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com >
2026-10-09 20:19:30 -07:00
c892301ff7
[Platform] Thread vendor KV pool classes through the in-tree builders; add PlatformCapabilities ( #35594 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: Yihao Wang <42559837+AgainstEntropy@users.noreply.github.com >
2026-10-07 19:16:10 -07:00
SuperSong and Cheng Wan
6cec8f98ce
[Bugfix] Fix unified-memory compaction gates and pending page reuse ( #39982 )
...
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-04 02:48:08 -07:00
937c0a6cfe
Fix unified HiCache physical transfers ( #39479 )
...
Co-authored-by: yhzhuang <yhzhuang@fb.com >
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com >
Co-authored-by: Yonghao Zhuang <yhzhuang@users.noreply.github.com >
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
2026-10-04 00:38:56 -07:00
b016ca406f
Support post-capture KV sizing for the unified hybrid-SWA pool ( #41961 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com >
2026-10-03 09:54:20 -07:00
Liangsheng Yin
122b59152c
[mem_cache] Replace is_chunk_cache / is_tree_cache with supports_prefix_sharing ( #42362 )
2026-10-03 02:07:55 -07:00
caihuali95 and Cheng Wan
2a3655fea1
[unified-memory] Name the fused KV translate for what it computes (7/7) ( #40331 )
...
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-01 17:14:03 -07:00
df530479c6
[unified-memory] Stride the KV translate kernel and route every translate through it (6/7) ( #40330 )
...
Co-authored-by: Caihua Li <caihua.li@bytedance.com >
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-01 17:12:14 -07:00
9855e9c6f8
[unified-memory] Remove the kernel-page multiplier plumbing (5/7) ( #40329 )
...
Co-authored-by: Caihua Li <caihua.li@bytedance.com >
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-01 17:11:25 -07:00
cdadb1c79a
[unified-memory] Token-major dense views for the unified memory pool (3/7) ( #38592 )
...
Co-authored-by: Caihua Li <caihua.li@bytedance.com >
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-01 17:09:38 -07:00
Shuwen Wang
bb62f125aa
[HiSparse] fix: return HiSparse slots on PD decode waiting abort and DeepSeek V4 free ( #41346 )
2026-10-01 10:24:51 +08:00
metamergebot and Yonghao Zhuang
671000aee6
[unified-memory] Honor move gates in float relocation and size auto HiCache from host capacity ( #41248 )
...
Co-authored-by: Yonghao Zhuang <yhzhuang@meta.com >
2026-09-25 12:17:28 -07:00
Liangsheng Yin
621136e094
[mem_cache] Remove unused helpers in mem_cache, storage backends, and metrics ( #40963 )
2026-09-23 18:16:44 -07:00
Guangda Liu and Guangda Liu
04c0913434
[HiSparse] Add MHA hisparse support for MiniMax M3 ( #31446 )
...
Co-authored-by: Guangda Liu <bingps@users.noreply.github.com >
2026-09-22 13:28:03 +08:00
Cheng Wan
506698761d
[unified-memory] Hierarchical cache for every unified pool shape ( #37507 )
2026-09-21 16:50:37 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
df0dc44931
[Fix] Forward SWA prealloc reclaim through the DSV4 HiSparse allocator ( #40354 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-09-19 20:27:51 -07:00
5931fd60ee
Support unified memory page-envelope transfers in PD ( #39477 )
...
Co-authored-by: yhzhuang <yhzhuang@fb.com >
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com >
Co-authored-by: Yonghao Zhuang <yhzhuang@users.noreply.github.com >
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-09-18 17:39:50 -07:00
Jialin Ouyang
f3851486cb
[Perf] Fuse SWA page lookup and mapping clear ( #38948 )
2026-09-18 15:51:07 -07:00
2929a39927
Use a shared byte budget for unified hybrid-SWA memory ( #36729 )
...
Co-authored-by: yhzhuang <yhzhuang@fb.com >
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-09-15 15:27:00 -07:00
Cheng Xin
ca8ecc6a6f
[NPU] Support DSV4 host memory cache management ( #37382 )
2026-09-14 10:38:50 +08:00
Cheng Wan
2ec4bbcbd4
[unified-memory] PD disaggregation for every unified pool shape ( #37506 )
2026-09-13 19:16:43 -07:00
822e73ccdd
[Unified Cache][AMD] Support DeepSeek-V4 unified KV in direct external linkers ( #38269 )
...
Co-authored-by: amd-danli103 <danli103@amd.com >
Co-authored-by: Duyi-Wang <duyi.wang@amd.com >
Co-authored-by: TianDi101 <tiandi920722@gmail.com >
2026-09-11 01:49:29 -07:00
Shunkangz
40a84d6dfc
[kv-shard 1/4] Logical-page placement with UnifiedRadixCache ( #38356 )
2026-09-10 22:06:42 -07:00
bede776c2a
fix(unified-memory): evict Full KV for Mamba byte shortfalls ( #36713 )
...
Co-authored-by: Yangmin Li <yangminl@nvidia.com >
Co-authored-by: YAMY <74099316+YAMY1234@users.noreply.github.com >
2026-09-09 13:46:51 -07:00
Ankith Averineni and HaiShaw
eb42598bdd
[AMD][DSV4] Skip the paged SWA page return under the per-request ring ( #38571 )
...
Co-authored-by: HaiShaw <hixiao@gmail.com >
2026-09-08 21:15:30 -07:00
Liangsheng Yin
28ebede865
[mem_cache] Free hybrid SWA pages by one representative per page on page_size > 1 ( #38159 )
2026-09-07 20:15:54 -07:00
570087ceda
[AMD][DSV4] Reland unified-KV pool sizing and SWA ring accounting, fully gated ( #38192 )
...
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com >
2026-09-07 13:13:04 -07:00
Cheng Wan
b5766336d4
[Perf] Unified memory: close the DCP decode gap on Blackwell ( #37926 )
2026-09-07 01:10:44 -07:00
+2
97c6978369
GLM-5.3-Flash support ( #36507 )
...
Co-authored-by: zRzRzRzRzRzRzR <Yuxuan.Zhang2@liverpool.ac.uk >
Co-authored-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com >
Co-authored-by: zanes-ops <zanes@nvidia.com >
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com >
Co-authored-by: Jian Chen <jianchen0311@gmail.com >
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com >
Co-authored-by: andyluo7 <43718156+andyluo7@users.noreply.github.com >
Co-authored-by: Ehsan Akhgari <ehsan.akhgari@gmail.com >
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai >
Co-authored-by: BBuf <1182563586@qq.com >
Co-authored-by: Raiden Makoto <81530826+Raiden-Makoto@users.noreply.github.com >
2026-09-06 02:27:59 -07:00
Liangsheng Yin
f5819b09bf
Revert "[AMD][DSV4] Fix unified-KV pool sizing and SWA ring accounting" ( #38163 )
2026-09-05 17:28:46 -07:00
yuttian1
514b45fd34
[AMD][DSV4] Fix unified-KV pool sizing and SWA ring accounting ( #30315 )
2026-09-05 16:39:40 -07:00
Liangsheng Yin
0454c074b4
[mem_cache] Clean up unified allocator leftovers ( #38103 )
2026-09-04 22:02:50 -07:00
Liangsheng Yin
0645398a32
[mem_cache] Move the unified-memory allocators into allocator/ and split the composites out ( #38072 )
2026-09-04 19:53:15 -07:00
Alex Nails
a44bb397a9
[Perf] Vectorize alloc_extend_naive to remove the per-request Python loop ( #37938 )
2026-09-04 17:26:25 -07:00
67248e04b4
[mem_cache] Route hybrid SWA full-side kv-row frees through free_segment ( #37876 )
...
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com >
Co-authored-by: Sam Shleifer <sshleifer@gmail.com >
2026-09-04 01:48:33 -07:00
Liangsheng Yin
2a980cbf10
[mem_cache] Require page-aligned starts in free_segment and drop the boundary trim ( #37729 )
2026-09-03 13:28:33 -07:00
Alex Nails and Alison Shao
28262c20df
[CI][RFC] Replace black-jupyter with ruff-format ( #37210 )
...
Co-authored-by: Alison Shao <a.shao@wustl.edu >
2026-09-02 19:46:08 -07:00
c05f8ae830
[PD] Optimize paged allocator free-list release ( #37146 )
...
Co-authored-by: wangwenming.41 <wangwenming.41@jd.com >
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com >
2026-09-02 16:51:37 -07:00
Liangsheng Yin
19c7679e9e
[mem_cache] Make free_swa sync-free on page_size == 1 ( #36723 )
2026-09-02 14:18:22 -07:00
Liangsheng Yin
01c3a5f54f
[misc] Resolve SWA ownership at enqueue time for grouped free() ( #36646 )
2026-09-01 23:06:55 -07:00
Liangsheng Yin
83a9b5dd88
[mem_cache] Drop the torch.unique sync from the SWA page expansion ( #37463 )
2026-09-01 14:16:47 -07:00