 cctry
|
7a6191c4b9
|
Preallocate HiCache MHA staging before post-capture KV sizing (#40256)
|
2026-09-21 10:44:29 -07:00 |
|
|
cctry
|
7ad55e4386
|
[HiCache] TMA-staged host<->device KV transfer kernel (sm_90+) (#40278)
|
2026-09-21 10:38:23 -07:00 |
|
|
cctry
|
1b78083b42
|
[PD] Add optional KV transfer checksums (#39500)
|
2026-09-16 23:53:30 +08:00 |
|
|
cctry
|
dad8c074e7
|
Scope prefetch cache state to the request attempt (#39318)
|
2026-09-14 15:03:03 -07:00 |
|
|
cctry
|
24b6c1c7f5
|
Fix multimodal embedding cache retaining full batches through views (#39120)
|
2026-09-12 21:37:14 -07:00 |
|
|
cctryandcctry
|
7e3d18bbcc
|
Add a provider hook for prefill-buffer ceilings (#39182)
Co-authored-by: cctry <17473714+cctry@users.noreply.github.com>
|
2026-09-12 21:37:10 -07:00 |
|
|
cctryandcctry
|
206034e520
|
Keep graph-pool borrows on their allocation stream (#39180)
Co-authored-by: cctry <17473714+cctry@users.noreply.github.com>
|
2026-09-12 19:26:09 -07:00 |
|
|
cctryandcctry
|
18cc55dc0b
|
Expose a capacity check for graph-pool borrows (#39178)
Co-authored-by: cctry <17473714+cctry@users.noreply.github.com>
|
2026-09-12 19:24:57 -07:00 |
|
|
cctryandcctry
|
7ae4af8187
|
Scope graph-pool borrowing to the runtime and reduce fragmentation (#39177)
Co-authored-by: cctry <17473714+cctry@users.noreply.github.com>
|
2026-09-12 11:45:17 -07:00 |
|
|
cctryandcctry
|
2784a86062
|
Reuse live CUDA graph executables during dedup registration (#39176)
Co-authored-by: cctry <17473714+cctry@users.noreply.github.com>
|
2026-09-12 11:44:17 -07:00 |
|
|
cctry
|
203d7e812c
|
Fix KV-canary workspace accounting after graph capture (#38596)
|
2026-09-10 16:10:12 -07:00 |
|
|
cctry
|
42bbaac259
|
[metrics] Report logical prefill token counts (#38566)
|
2026-09-10 16:09:21 -07:00 |
|
|
cctry
|
354ee46c26
|
Keep VMM capability votes on CPU (#38722)
|
2026-09-09 11:56:30 -07:00 |
|
|
cctry
|
0eff0f7460
|
Add num_prealloc_ready_tokens to decode load snapshot (#38065)
|
2026-09-04 15:05:57 -07:00 |
|
|
cctry
|
33a22b1b08
|
[Cache] Forward fast prefix matching capability (#37844)
|
2026-09-03 10:33:30 -07:00 |
|
|
cctry
|
3a855b050a
|
fix(disagg): poll receivers during decode preallocation (#37483)
|
2026-09-02 14:26:40 -07:00 |
|
|
cctry
|
ad6e830858
|
[Bugfix] Key CUDA graph dedup signatures on kernel function identity (#37657)
|
2026-09-02 14:24:31 -07:00 |
|
|
cctryandcctry
|
9a05b470fa
|
[Memory] Size the CUDA graph pool from warmup measurements and fix graph-pool borrowing (#36911)
Co-authored-by: cctry <cctry@fb.com>
|
2026-09-01 09:32:38 -07:00 |
|
|
cctry
|
aa718f7343
|
Refactor HiCache host pool management (#36232)
|
2026-08-25 16:31:57 -07:00 |
|
|
cctryandcctry
|
32d98aad13
|
[HiCache] Allow a retraction host pool smaller than the device pool (#35543)
Co-authored-by: cctry <cctry@fb.com>
|
2026-08-19 22:59:57 -07:00 |
|
|
cctry
|
1c82955861
|
[HiCache] Split the host-memory budget across co-located ranks (#35540)
|
2026-08-19 13:01:58 -07:00 |
|
|
cctryandcctry
|
37c09ff3d8
|
[Memory] Borrow CUDA graph pool storage for EAGLE sampling (#35375)
Co-authored-by: cctry <cctry@fb.com>
|
2026-08-18 16:19:41 -07:00 |
|
|
cctryandcctry
|
2e7c85da68
|
[PD] Preserve decode KV across retraction in HiCache (#34801)
Co-authored-by: cctry <cctry@fb.com>
|
2026-08-17 08:49:11 -07:00 |
|
|
cctry
|
8922bb98e2
|
refactor(hicache): flatten L2 transfer execution (#34793)
GB300 test fails unrelated
|
2026-08-16 00:33:34 -07:00 |
|
|
cctryandYilong Zhao
|
e5b3a48751
|
Add --http2-max-concurrent-streams server arg (#34796)
Co-authored-by: Yilong Zhao <74357408+happierpig@users.noreply.github.com>
|
2026-08-15 10:34:49 -07:00 |
|
|
cctry
|
df986c4d5e
|
Consolidate CUDA VMM allocation helpers (#34199)
|
2026-08-10 18:11:11 -07:00 |
|
|
cctryandkangwangamd
|
df3aa20d89
|
Reland serving-time Triton load diagnostics (#33908)
Co-authored-by: kangwangamd <100359556+kangwangamd@users.noreply.github.com>
|
2026-08-07 14:51:43 -07:00 |
|
|
cctry
|
a3a1ebc7b7
|
Warn on risky serving-time Triton work (#33120)
|
2026-08-05 14:55:03 -07:00 |
|
|
cctry
|
96c89863a3
|
Measure prefill busy time between launches (#33595)
|
2026-08-05 11:26:27 -07:00 |
|
|
cctry
|
c8822fd990
|
Clarify post-capture KV reservation logs (#33598)
|
2026-08-04 15:06:11 -07:00 |
|
|
cctry
|
db0fe370b7
|
[PD] Fix false health-503 during decode retraction re-admission (#33118)
|
2026-08-03 09:44:25 -07:00 |
|
|
cctry
|
6cf661117d
|
[PD] Add a queues.prealloc_ready counter to the load snapshot (#33133)
|
2026-08-03 09:44:11 -07:00 |
|
|
cctry
|
3c1717d9b6
|
Follow up on #30157 post-merge review (#32672)
|
2026-07-29 15:03:59 -07:00 |
|
|
|
2c63a2f12b
|
Fix --hicache-size allocating ~2x host memory on hybrid SWA (#32373)
Co-authored-by: cctry <cctry@fb.com>
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
|
2026-07-25 17:19:44 -07:00 |
|
|
cctryandJialin Ouyang
|
a690e5e0b3
|
Add stream label to TTFT metrics (#32363)
Co-authored-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
|
2026-07-24 17:44:04 -07:00 |
|
|
cctryandYinghai Lu
|
ce705bb6dc
|
Report accelerator type in /v1/loads (#32348)
Co-authored-by: Yinghai Lu <yinghai@meta.com>
|
2026-07-24 17:31:28 -07:00 |
|
|
cctry
|
8727d105db
|
Add prefill and decode load counters to LoadSnapshot (#32245)
|
2026-07-24 14:04:54 -07:00 |
|
|
cctryandZhiqiang Xie
|
d28e35b1a1
|
Fix num_running_reqs gauge on disagg prefill servers (#31495)
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
|
2026-07-16 15:43:06 -07:00 |
|
|
cctryandcctry
|
cb47a68717
|
[PD] Stride KV token->page indices on device before D2H copy (#31173)
Co-authored-by: cctry <cctry@fb.com>
|
2026-07-14 10:08:30 -07:00 |
|
|
cctry
|
1b4176cc46
|
[PD] Fix optimistic prefill inflight-queue hangs on parked/aborted reqs (#31075)
|
2026-07-13 20:34:44 -07:00 |
|
|
cctry
|
c616d5a55e
|
[PD] Improve optimistic prefill (#30951)
|
2026-07-12 15:31:53 -07:00 |
|
|
cctry
|
b86466d54b
|
Make KvVmmArena JIT stub unique per process (#30702)
|
2026-07-09 15:56:26 -07:00 |
|
|
cctryandcctry
|
1959335997
|
refactor(load-snapshot): build LoadSnapshot directly, drop legacy get_loads IPC (#30525)
Co-authored-by: cctry <cctry@fb.com>
|
2026-07-09 05:58:10 -07:00 |
|
|
cctry
|
096551eed6
|
Make CUDA graph disabling PD-role-aware (prefill/decode) (#30409)
|
2026-07-08 15:08:22 -07:00 |
|
|
cctry
|
2ad9a243f5
|
Size KV pool after CUDA graph capture (opt-in) (#30157)
|
2026-07-07 12:05:01 -07:00 |
|
|
cctry
|
fbe3110866
|
Make mem_fraction_static reserve disaggregation-mode aware (#29615)
|
2026-07-04 18:44:06 -07:00 |
|
|
cctry
|
697b400d70
|
Share one logits output buffer across prefill/decode/draft cuda-graph runners (#29779)
|
2026-07-01 23:30:56 -07:00 |
|
|
cctry
|
80ac11eda3
|
Fix capture-mode detection during breakable CUDA graph capture (#29866)
|
2026-07-01 22:12:56 -07:00 |
|
|
cctryandcctry
|
4b4b4af583
|
Budget EAGLE/STANDALONE draft KV pool in SWA pool configurators (#29622)
Co-authored-by: cctry <cctry@fb.com>
|
2026-06-30 00:58:33 -07:00 |
|
|
cctryandcctry
|
a5c45a12bb
|
CUDA graph executable dedup via cudaGraphExecUpdate (#29625)
Co-authored-by: cctry <cctry@fb.com>
|
2026-06-29 13:56:33 -07:00 |
|
|
cctry
|
dbe9e3b706
|
[PD] Early-send cached-prefix KV overlapping uncached prefill forward (#29316)
|
2026-06-25 13:31:53 -07:00 |
|
|
cctryandcctry
|
76db6c9d9e
|
[Refactor] Share CUDA graph memory pool across prefill and decode (#28973)
Co-authored-by: cctry <cctry@fb.com>
|
2026-06-24 10:18:20 -07:00 |
|
|
cctryandcctry
|
743ce88bc5
|
Fix flaky optimistic prefill retry test (#28995)
Co-authored-by: cctry <cctry@fb.com>
|
2026-06-23 00:18:01 -07:00 |
|
|
cctryandcctry
|
0c065671c9
|
[Spec] Redo: split init_backends; account draft weights in --mem-fraction-static (#28855)
Co-authored-by: cctry <cctry@fb.com>
|
2026-06-21 20:45:16 -07:00 |
|
|
cctry
|
9691a29fe0
|
[Spec] Split init_backends; account draft weights in --mem-fraction-static (#28683)
|
2026-06-21 01:22:26 -07:00 |
|
|
cctry
|
6d4ca9bc54
|
Cap SWA pool sizing with chunk cache (#28755)
|
2026-06-21 01:06:59 -07:00 |
|
|
cctry
|
fa71064147
|
fix: speculative draft worker clobbering target attention backend (#28559)
|
2026-06-18 01:21:14 -07:00 |
|
|
cctry
|
7976928c57
|
Abort during chunked prefill + PD peer-liveness abort (#28086)
|
2026-06-17 23:13:28 -07:00 |
|
|
cctry
|
cfa4aa988f
|
Revert "revert the head_dim assignment from PR 23862" (#28583)
|
2026-06-17 17:11:05 -07:00 |
|
|
cctry
|
5d6b35eabb
|
revert the head_dim assignment from PR 23862 (#28571)
|
2026-06-17 15:46:28 -07:00 |
|
|
|
33719cfb31
|
[PD] Optimize SWA allocation (#28085)
Co-authored-by: cctry <cctry@fb.com>
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
|
2026-06-15 11:01:55 -07:00 |
|
|
cctry
|
75998d0421
|
Fix --mem-fraction-static not accounting for EAGLE draft model KV cache (#23862)
|
2026-06-12 10:35:48 -07:00 |
|
|
cctryandcctry
|
10219bd9d6
|
[PD] Fix negative prefill kv_transfer_alloc_ms under optimistic prefill (#27885)
Co-authored-by: cctry <cctry@fb.com>
|
2026-06-11 13:12:25 -07:00 |
|
|
cctry
|
b3e4c204fd
|
Don't write crash dump on graceful exit (#27405)
|
2026-06-05 17:06:39 -07:00 |
|
|
|
f6cd1a9822
|
Add num_waiting_uncached_tokens load metric (#27174)
Co-authored-by: cctry <cctry@fb.com>
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
|
2026-06-03 18:29:49 -07:00 |
|
|
cctryandgemini-code-assist[bot]
|
b55570d38e
|
[PD] Optimistic prefill (#26780)
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
|
2026-06-02 01:16:14 -07:00 |
|
|
|
163bf1ba71
|
[PD] Fix KV transfer metrics (#24416)
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
|
2026-05-06 03:44:48 -07:00 |
|
|
cctry
|
660a77f221
|
Silence noisy health-check race log in TokenizerManager (#24466)
|
2026-05-05 21:06:43 -07:00 |
|
|
cctry
|
5c31b35db2
|
[hicache] Optimization for DMA copy (#8245)
|
2025-08-09 17:16:07 -07:00 |
|