Commit Graph
111 Commits
Author SHA1 Message Date
Liangsheng Yin 37556c1b91 [mem_cache] Replace cache_finished_req with insert_req; release_kv_cache frees and unpins (#41281) 2026-09-27 02:53:16 -07:00
Liangsheng Yin 8ca82118e0 [mem_cache] Drop is_insert from cache_finished_req; release rows from release_kv_cache (#40988) 2026-09-24 17:23:26 -07:00
Liangsheng Yin 954458567e [mem_cache] Remove unreachable RadixCache paths in KV canary and HiCache accessors (#40807) 2026-09-23 13:22:58 -07:00
Liangsheng Yin 4ce23542bf [HiCache] Remove the unused HiRadixCache (#40787) 2026-09-22 16:17:12 -07:00
Liangsheng Yin 6cc9090d1f [mem_cache] Release up to owned_kv_len on radix cache insert (#40075) 2026-09-18 15:37:52 -07:00
Nan Jiang 20518d8518 [Scheduler] Align RadixCache no-insert cleanup with kv_len_to_handle (#35204) 2026-09-17 17:00:59 -07:00
Yanbin JiangandShuwen Wang 0b415fa573 [HiCache][LoRA] Isolate storage pages by extra key (#38577)
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com>
2026-09-12 18:18:14 +08:00
Xiaoyu ZhangandCursor 0a57403468 [Perf] Optimize Qwen3-VL unique-image serving on H100 (#36411)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-12 15:16:59 +08:00
Shuwen Wang 6e312af8c2 fix: collect prefix hash values iteratively (#38204) 2026-09-07 11:10:32 +08:00
Shuwen WangandClaude Opus 5 4b44a1cde2 [Refactor] Let eviction policies take construction parameters (#37795)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-05 13:48:30 +08:00
Khoa PhamandClaude Opus 5 cf3173aeb9 [Perf] Walk the radix tree by offset instead of re-slicing token storage (ported from #36507) (#37324)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-03 11:13:03 +08:00
Alex NailsandAlison Shao 28262c20df [CI][RFC] Replace black-jupyter with ruff-format (#37210)
Co-authored-by: Alison Shao <a.shao@wustl.edu>
2026-09-02 19:46:08 -07:00
Liangsheng Yin 007ef5e23a [mem_cache] Move req_pool_idx into ReqKvInfo (#37094) 2026-08-30 14:46:21 -07:00
Liangsheng Yin 6be767c2d2 [mem_cache] Move cache_protected_len and swa_evict_floor into ReqKvInfo (#36982) 2026-08-29 19:37:53 -07:00
Ke Bao 480033def0 Refactor kv cache event mixin into a recorder (#35164) 2026-08-19 01:20:17 +08:00
jthomson04 385903b0ac feat: add cache salt support to KV cache events (#30827)
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
2026-08-12 16:14:04 -07:00
Leon Gao cfb354bcfc [Bugfix] Fix batched KV free aliasing (#34067) 2026-08-08 03:34:44 -07:00
Leon Gao 4d4f8023c4 [srt] Batch scheduler cache frees (#33475) 2026-08-06 21:49:59 -07:00
056474cdb0 feat: Session-reference-aware Unified Radix Cache for agentic multi-turn workloads (#29173)
Co-authored-by: Ishan Dhanani <ishandhanani@gmail.com>
Co-authored-by: hzh0425 <hzh0425@apache.org>
Co-authored-by: ispobock <ispobaoke@gmail.com>
2026-08-02 13:43:55 +08:00
Liangsheng Yin c4fc241fd3 [Perf] Free KV pages by segment in the paged allocator without a device sync (#32701) 2026-07-28 22:24:16 -07:00
Liangsheng Yin 10908a6793 [Fix] Respect cache_protected_len in ChunkCache and disabled-radix release paths (#31662) 2026-07-18 12:04:25 -07:00
fzyzcjy 27256aee5b Let cache backend do not couple with owned committed kv details and avoid kv_committed_freed/kv_overallocated_freed fields (#29428) 2026-07-15 14:43:10 +08:00
huangtingwei 5e1ccd9320 [HiCache] Optimize HiCache hash generation with bulk token byte conversion (#28287) 2026-07-01 15:44:23 +08:00
ishandhanani aaa31eb0a1 feat: first-class session identity in SGLang (#29436) 2026-06-28 07:16:38 -07:00
ishandhanani e67b228d4c feat: session radix cache (#27058)
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
2026-06-23 01:51:41 -07:00
Qiaolin Yu f293ddf3ce [perf] reduce overhead of fill_ids list reconstruction and decref (#27965) 2026-06-14 00:41:11 -07:00
Cheng Wan 97a0031799 [lint] Enable Ruff UP037 to drop redundant quoted annotations (#27984) 2026-06-11 17:38:05 -07:00
fzyzcjy 259a2da3e0 Refactor Req.fill_ids into full_untruncated_fill_ids + fill_len with equivalence (#26637) 2026-06-08 14:52:18 +08:00
Qiaolin Yu 4b0f629082 [perf] reduce radix cache match overhead by changing the match algorithm (#27364) 2026-06-06 15:40:28 -07:00
ishandhanani 978fb6ed1a hicache kv events: publish split write-through fragments (#27072) 2026-06-03 14:52:25 -07:00
Zhangheng 7662210406 [UnifiedTree]: Support eviction priority (#26549) 2026-05-30 15:19:43 +08:00
Ethan ZHUandZhangheng e86fdf3a3c [Bug Fix][HiCache] TreeNode.get_prefix_hash_values @lru_cache can return mutated list (#26177)
Co-authored-by: Zhangheng <hzh0425@apache.org>
2026-05-25 11:15:10 +08:00
Jialin Ouyangandjialino 06c23d55b5 perf: migrate Req token-id storage to array.array('q') in Scheduler (#25098)
Co-authored-by: jialino <jialino@fb.com>
2026-05-22 10:51:07 -07:00
Ke Bao e4378ff37f [UnifiedRadixCache] Fix HiCache load back start node (#25088) 2026-05-14 13:18:12 +08:00
zhongdaor-nvandzhongdaor-nv 2cf1a4ab38 feat: Add KV events for Mamba radix cache (#23678)
Signed-off-by: zhongdaor-nv <220807034+zhongdaor-nv@users.noreply.github.com>
Co-authored-by: zhongdaor-nv <220807034+zhongdaor-nv@users.noreply.github.com>
2026-05-08 11:53:36 -07:00
Lianmin Zheng 3c3f0bd55e Cache empty MatchResult in RadixCache (#24470) 2026-05-07 17:13:20 -07:00
ishandhanani 5b7ce417d0 [P/D disagg] - support decode side radix cache (#19746) 2026-05-01 21:55:34 +08:00
Liangsheng Yin 2b2cad70d6 [Refactor] Move radix-cache utils onto RadixKey as methods (#23209) 2026-04-20 23:11:58 -07:00
ishandhananiandjthomson04 3c007ee5d4 fix(hicache): emit KV events for L2 host cache insertions (#22894)
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Co-authored-by: jthomson04 <jwillthomson19@gmail.com>
2026-04-20 19:07:03 -07:00
Liangsheng Yin c7a4ebf3c8 [Refactor] Replace page_align_keys helper with RadixKey.page_aligned method (#23107) 2026-04-20 18:10:42 -07:00
Liangsheng Yin 8cb957ccff [Perf] Make EAGLE bigram key an O(1) view on RadixKey (#23106) 2026-04-20 12:01:11 -07:00
Liangsheng Yin 49cb7d546e Move hash utils out of hicache_storage to break CUDA import chain (#22214) 2026-04-06 18:16:40 -07:00
ishandhanani 1081a25983 revert: remove TTL-based hard pin from HiRadixCache (#21884) 2026-04-01 16:51:15 -07:00
pansicheng 97b2a89334 [RadixTree][8/N Refactor]: unify lock interface (#20330) 2026-03-16 11:49:51 +08:00
liubiyonggeandzhangheng cc73355a1f [Feature] Add SLRU eviction policy & fix RadixCache hit_count bug (#18843)
Co-authored-by: zhangheng <hzh0425@apache.org>
2026-03-08 21:30:55 +08:00
ishandhananiandClaude f7da379b61 feat: TTL-based prefix pinning with refresh-on-hit for HiRadixCache (#18941)
Co-authored-by: Claude <noreply@anthropic.com>
2026-03-01 23:27:22 -08:00
zhangheng a0d8a7ae6d [RadixTree][6/N Refactor]: Refactor SWARadixTree to simplify the computation and alignment of bigram keys. (#19427) 2026-02-28 20:01:39 +08:00
SoluMilken 07a24f1a38 update pre-commit config (#18860) 2026-02-16 00:18:31 +08:00
Yi-Chia Chenandcopilot-swe-agent[bot] 2bfab1bb67 Fix radix cache key to include generated tokens in multi-turn (regression) (#16521)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
2026-02-10 14:08:34 -08:00
ishandhanani 01e3f4682e feat(kv-events): Add medium field to KV event types for storage tier tracking (#18205) 2026-02-09 12:39:15 -08:00