Commit Graph
14603 Commits
Author SHA1 Message Date
b2786a670e [MemCache] Count reclaimable tokens in the unified tree ledgers (#43185)
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-08 17:48:11 -07:00
Liangsheng Yin b4fe282836 [MemCache] Delete ChunkCache (#43201) 2026-10-08 17:31:16 -07:00
Liangsheng Yin 508ba79688 [MemCache] Serve disabled hybrid-SWA models with UnifiedRadixCache (#43200) 2026-10-08 17:30:01 -07:00
Liangsheng Yin 265efe5e01 [MemCache] Serve disabled full-attention models with UnifiedRadixCache (#43199) 2026-10-08 17:27:24 -07:00
Liangsheng Yin 977e2719c7 [MemCache] Serve disabled pure-SWA models with PureSWARadixCache (#43198) 2026-10-08 17:25:54 -07:00
Liangsheng Yin 57ce47862d [MemCache] Report no prefix sharing from a disabled RadixCache (#43197) 2026-10-08 17:23:11 -07:00
Khoa Pham fc3385d530 feat(kda): integrate FlashInfer prefill checkpoints (#41400) 2026-10-08 17:16:33 -07:00
Khoa Pham 1e77db3215 [Perf] Avoid gate copies in gated RMSNorm (#43227) 2026-10-08 17:11:58 -07:00
20d1f3f56a [Scheduler] Refresh waiting requests' cached prefixes under FCFS-like policies with LRU eviction (#41593)
Signed-off-by: Pei Li <peili@nvidia.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-08 17:05:04 -07:00
bb43927404 [PD] Charge decode admission the page-rounded allocation when the radix cache is disabled (#43173)
Co-authored-by: hanming-lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
2026-10-08 16:48:31 -07:00
Liangsheng Yin 2e64740442 [CI] Avoid Hugging Face Hub API calls when the local model cache is complete (#43229) 2026-10-08 16:27:31 -07:00
Liangsheng Yin ae4da8cd78 [MemCache] Skip SWA request-cap pool sizing when chunked prefill is off (#43203) 2026-10-08 16:19:35 -07:00
2abb761639 [metrics] Count per-rank DP attention pairs and their imbalance (#43180)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: hanming-lu <hanming-lu@users.noreply.github.com>
2026-10-08 15:01:37 -07:00
metamergebotandHanming Lu 0b658c12ff [Bench][MoE] Deal the simulated round-robin experts evenly across EP ranks (#43188)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
2026-10-08 15:01:16 -07:00
Siyuan Chen b0e06359ac [DSV4.1][PD][5/N] Decode node support DP attention (#40177) 2026-10-08 14:42:16 -07:00
Xuanteng Huang d20bdc1443 Add per-token NVFP4 MoE support for ReLU2 activation (#39199)
Signed-off-by: Xuanteng Huang <xuanteng.huang@outlook.com>
2026-10-08 14:28:22 -07:00
ShunkangzandClaude Opus 5 3d8d44a4a9 [Qwen3.8 CP 3/4] Collocated prefill CP integration (#39723)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-08 14:20:27 -07:00
Sage 5a5ea81036 [rust-server] warm up each rust listener in-process (#43004)
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
2026-10-08 14:09:46 -07:00
e902207a4e [Speculative] Support block verification for the DFlash family (#43012)
Co-authored-by: maocheng <mao.cheng@MacBook-Pro-MD663G60VV.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 13:59:52 -07:00
d1ba943016 [Model] Add support for EmbeddingGemma 2 (#42782)
Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 13:54:45 -07:00
metamergebotandhanming-lu 93f2acf614 [Metrics] Use widely supported bucket boundaries for the DP attention imbalance ratio (#43187)
Co-authored-by: hanming-lu <hanming-lu@users.noreply.github.com>
2026-10-08 12:58:58 -07:00
Ming Yang 3fef129ff8 [PD] Allow state-only KV checksum inputs (#42904) 2026-10-08 12:54:03 -07:00
075fd8fa0b [Mamba] Describe the track snapshot dtype by its kernel instead of fp32 (#43192)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: jiayisuse <jiayisuse@users.noreply.github.com>
2026-10-08 12:47:46 -07:00
e3c5719713 [DFlash] Build the attention QKV projection through an overridable hook (#43179)
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: 842974287 <842974287@users.noreply.github.com>
2026-10-08 12:46:28 -07:00
kunchengit 00b3a48753 [dLLM] Add JointThresholdInDel algorithm for insertion & deletion decoding (#31773) 2026-10-08 12:44:14 -07:00
Aurick QiaoandQiaolin-Yu b2cb24995d [Debug] Add opt-in NaN detection before sampling (#42690)
Co-authored-by: Qiaolin-Yu <liin1211@outlook.com>
2026-10-08 12:20:57 -07:00
Shenxiu LiuandQiaolin-Yu 393eaed509 [Fix] overlap scheduler: record_stream mix_running_indices for the forward-stream relay gather (#34076)
Co-authored-by: Qiaolin-Yu <liin1211@outlook.com>
2026-10-08 12:19:35 -07:00
shodoco 1638c5d699 [http_server] Let custom /generate routes skip the second contract check (#43114) 2026-10-08 11:56:06 -07:00
214347891a [Bugfix] Skip post-experts EP all-reduce on the FlashInfer cutlass FP4 all-gather MoE path (#41072)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-08 18:01:27 +00:00
shodoco e2fbc6a550 Stream a chunk once stream_interval tokens are unsent (#42643) 2026-10-08 10:12:37 -07:00
Eric Zhang 344b8a863b [Perf] Fuse short-convolution checkpoint tracking metadata (#43013) 2026-10-09 00:26:15 +08:00
Ke Bao 642e81fbb8 Fix DSA index host buffer cleanup (#43100) 2026-10-09 00:21:38 +08:00
Ke Bao 46ad9eb439 Fix K-only host buffer cleanup (#43101) 2026-10-09 00:21:27 +08:00
billishyahao 4ab4d90d78 [AMD][DCP 3/N] add aiter asm for kimi k3 target verify (#34537) 2026-10-08 09:13:03 -07:00
MickandMick Qian 3030e31ff8 [diffusion] refactor: deduplicate request extraction and ComfyUI test helpers (#42903)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 23:44:35 +08:00
11b1fb21de [MiniMax-M3] Decode: paged K/V tile loads and a tiny GEMM for the router projection (#38841)
Co-authored-by: Hao Phan <htphan@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 19:39:31 +08:00
6e6ccb26f2 [MiniMax-M3] Instantiate the decode block top-k in register buckets (#38615)
Co-authored-by: Hao Phan <htphan@nvidia.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-08 19:33:17 +08:00
MickandMick Qian 7dd512b0e8 [diffusion] refactor: deduplicate FLUX positions and RoPE application (#43002)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-08 18:18:39 +08:00
Horace He 8ad24fb560 [Perf] Cache compiled ModelOpt exclusion patterns (#42927) 2026-10-08 17:10:25 +08:00
Liangsheng Yin 4b384df0c9 [CI] Keep the KL tests' LongBench cache outside the checkout and bound its download (#43068) 2026-10-08 01:17:31 -07:00
Liangsheng Yin 7b9c42cf49 [CI] Remove the unregistered chunked-prefill tests under test/manual (#43081) 2026-10-08 01:16:39 -07:00
Liangsheng Yin 89b5935793 [MemCache] Return only the loaded length from init_load_back; drop empty_device_indices (#43023) 2026-10-08 01:14:33 -07:00
Liangsheng Yin 5a4d01a3ae [MemCache] Return only the matched length from match_prefix; read KV indices off the node path (#42923) 2026-10-08 01:10:14 -07:00
Spandan Tiwari 2d26823f94 [AMD] Drop static input_scale on aiter per-token FP8 path (use dynamic per-token) (#42152) 2026-10-08 01:06:34 -07:00
Cheng Wan b3e6c87be3 [Refactor] Translate each KV loc once per iteration (#42753) 2026-10-08 00:58:28 -07:00
508ddc859a [unified-memory] Admit DFLASH and DSPARK with fused draft KV (#40492)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-08 00:56:21 -07:00
c2d491e38b [unified-memory] Admit EAGLE/EAGLE3 onto fused draft KV, on mamba-MHA and MLA hosts (#40491)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-08 00:55:54 -07:00
8b80808366 [unified-memory] Fuse the draft model's KV into the target's page envelope (mechanism) (#37628)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-08 00:54:19 -07:00
08d30f2300 [unified-memory] Speculative decoding on the unified memory pool: translate every read and write path (#37627)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-08 00:52:21 -07:00
33883a5fa3 [AMD] Small-M W8A8 FP8 projection GEMM for Qwen3.5 AttnFP8 on gfx950 (#41134)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-10-08 00:50:16 -07:00