Commit Graph
323 Commits
Author SHA1 Message Date
51ace48387 [Metrics] Add request-level TPOT histogram (sglang:request_time_per_output_token_seconds) (#40275)
Co-authored-by: Gilford Ting <gilford@modal.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-29 12:20:22 -07:00
8f2a9638f7 [metrics] Fix non-streaming TTFT by flushing the first output through detokenization (#38600)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-09-29 11:24:52 -07:00
79cafec013 [Model] Add IQuest Q1 support and MTP draft (#41590)
Co-authored-by: zelong huang <yzhu@ubiquant.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: zelong518 <zelonghuang05@gmail.com>
Co-authored-by: chenwenxiaolive <cocoshirolive@gmail.com>
2026-09-28 17:21:31 -07:00
7889fbab9a [PD] Keep EAGLE DP graph and token metadata consistent (#32196)
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Khoa Pham <khoa.pham@radixark.ai>
2026-09-28 17:15:51 -07:00
Denji-kk 6caf0ff4ae perf(multimodal): offload CPU feature hashing with bounded admission (#39539) 2026-09-28 17:51:05 +08:00
Rain Jiang 6c4f36e47a Rust server unify datapath for mm and generate requests (#39679) 2026-09-27 23:36:07 -07:00
e75e3b8a98 [Bugfix] fix(hicache): wait for decode offload before retraction (#30899)
Co-authored-by: agent <agent@local>
Co-authored-by: Kevin Flansburg <6134007+kflansburg@users.noreply.github.com>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-27 23:24:19 -07:00
ShunkangzandZhangheng df1fc6ab64 [kv-shard 3/4] Enable Control Plane B (#39964)
Co-authored-by: Zhangheng <hzh0425@apache.org>
2026-09-27 14:19:08 +08:00
Sundara Raman Ramachandran 3ed56a341b [Score API] Setwise scoring: CausalLM support (batched + --enable-mis) (#41188) 2026-09-25 23:02:44 -07:00
Lianmin ZhengandLu Fang 3ecb6cd729 [MemCache] Unify component eviction cursors and lock receipts (#41276)
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
2026-09-25 20:53:35 -07:00
Liangsheng Yin f35ec18d8b [Test] Remove more unit tests that mirror implementation or never run in CI (#41297) 2026-09-25 19:22:14 -07:00
50e50e13ab [DCP] Use logical token capacity for PD admission and load reporting (#39731)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Khoa Pham <khoa.pham@radixark.ai>
2026-09-25 17:39:47 -07:00
Liangsheng Yin 1f6ce4b068 [Test] Remove unit tests that only mirror implementation or never run in CI (#41286) 2026-09-25 16:15:02 -07:00
Yuwei AnandClaude Opus 5.5 95047d3464 [Fix] Fix cpu CI fail introduced by pr #38652 (#41287)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 15:55:21 -07:00
Erik WijmansandEthan Su c421d16563 [sglang][lora] Support DP attention in LoRA backends (#36389)
Co-authored-by: Ethan (Yusheng) Su <yushengsu@radixark.ai>
2026-09-25 12:03:47 -10:00
cf5df82680 [KVCache] Support lmcache unified radix cache (#38652)
Signed-off-by: chunxiaozheng <1179548172@qq.com>
Co-authored-by: Yuwei An <ayw.sirius19@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 14:08:45 -07:00
Nan Jiang 24b6930076 [Sampling] Stream sampling masks as per-request arrays (#40986) 2026-09-25 12:20:25 -07:00
Richard Wang 16d1c93b3d [feat] add a system one compatible /v1/systemone route (#41208) 2026-09-25 12:59:11 +08:00
7325b38412 [PD] Honor gracefully_exit in disaggregation event loops and keep non-zero-rank launchers alive on SIGTERM (#40793)
Co-authored-by: metamergebot <324680979+metamergebot@users.noreply.github.com>
Co-authored-by: Shiyan Deng <dsy842974287@meta.com>
2026-09-24 20:29:07 -07:00
jacky.cheng 9dc4c5d891 [AMD] Restore non-DCP Mamba checkpoint donation to fix agent-mode cache hit at high conc with HiCache (#40907) 2026-09-24 18:08:10 -07:00
Sundara Raman Ramachandran 2f2f9d12f8 [Score API] Setwise Scoring Support (#38965) 2026-09-24 16:50:58 -07:00
cctryandcctry efff836911 [PD] Add decode host receive for custom transfer backends (#40238)
Co-authored-by: cctry <cctry@fb.com>
2026-09-23 21:57:33 -07:00
Nan Jiang c3685dff42 [Sampling] Add selected/support sampling logprob modes (#40932) 2026-09-23 21:38:19 -07:00
f4d9d2e670 [RL] Add RL weight-update sessions and support updating spec draft runners (#40777)
Co-authored-by: Jiajun Li <guapisolo@gmail.com>
Co-authored-by: Nan Jiang <59716405+nanjiangwill@users.noreply.github.com>
Co-authored-by: maocheng23 <maocheng@berkeley.edu>
Co-authored-by: Byron Hsu <byron+per@periodiclabs.ai>
Co-authored-by: JD-ETH <jaedon.guo@gmail.com>
Co-authored-by: Letian Ruan <ruanlt420@gmail.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-09-23 20:04:57 -07:00
6fe4b66f5c [RL] Keep pause_generation and weight updates from deadlocking each other (#40779)
Co-authored-by: maocheng23 <maocheng@berkeley.edu>
Co-authored-by: Zhichenzzz <zczeng@uw.edu>
2026-09-23 15:13:42 -07:00
Cheng Wan 79fec592a8 [Refactor] Read parallel placement in consumers (#40638) 2026-09-23 14:57:25 -07:00
7fef014d51 feat(kv-hints): add kv hint envelope to request transport (#38891)
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
Co-authored-by: Ishan Dhanani <ishandhanani@gmail.com>
2026-09-23 10:31:13 -07:00
inkcherryandDuyi-Wang 58f622ebc4 Reduce decode bootstrap latency with request-owned speculative KV (#38978)
Signed-off-by: inkcherry <mingzhi.liu@amd.com>
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
2026-09-23 02:24:53 -07:00
Richard Wang 40048f6e51 [dLLM] feat: support DiffusionGemma serving (#34061) 2026-09-23 09:59:31 +08:00
Shunkangz ddebc52f23 [kv-shard 3/4] Enable Control plane (#38468) 2026-09-22 19:57:44 +08:00
jthomson04 15ba54bd5d perf(engine): avoid timed waits for Engine responses (#39486)
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
2026-09-21 20:42:03 -07:00
YAMY 0229025127 [Spec][PP] Launch extend microbatches before the spec output exchange (#40499) 2026-09-21 15:47:03 -07:00
Cheng Wan acac4dd9d9 [Refactor] Clean up parallel runtime comments (#40632) 2026-09-21 14:32:22 -07:00
Cheng Wan 970e946e4f Retire the per-runner parallel record (#40343) 2026-09-21 12:26:40 -07:00
Cheng Wan 73f071db52 Deprecate the parallel getters the context answers, and ratchet them shut (#40342) 2026-09-21 12:25:32 -07:00
Cheng Wan 0db1a93adb State the draft's whole topology in its scope, and read the rest from the context (#40339) 2026-09-21 12:19:38 -07:00
AMRUTHA MandMa Mingfei d20cd9d77f [XPU]Enable HiSparse hierarchical sparse KV cache on Intel XPU (#32792)
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
2026-09-21 14:07:49 +08:00
Liangsheng Yin 0024efa0de [CI] Derive registered-test kind from the registry call instead of the path (#40294) 2026-09-20 02:01:22 -07:00
Qiaolin Yu f4c256354c [kimi k3][pd disagg] support pp prefill + dcp decode with dspark (#40045) 2026-09-20 00:15:28 -07:00
Mohammad Miadh AngkadandMohammad Angkad 22f02cc339 [Test] Fix scheduler fixtures after prefill burst counting (#40411)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-09-20 00:08:04 -07:00
huangtingweiandChao Shi 020703923d [PP + HiCache] Add PP Prefetch Tickets for eager cross-stage storage prefetch (#36700)
Co-authored-by: Chao Shi <stepinto@live.com>
2026-09-20 11:19:31 +08:00
3a64faa1f2 Fix disagg PP MTP for GLM-5.2 (#39378)
Co-authored-by: Julien Lin <jullin@nvidia.com>
Co-authored-by: YAMY1234 <74099316+YAMY1234@users.noreply.github.com>
Co-authored-by: Yangmin Li <yangminl@nvidia.com>
2026-09-19 13:55:07 -07:00
Xiaoyu Zhang 0b0d2c257a [Fix] Repair CI fixtures and ROCm speculative tree device checks (#40325) 2026-09-19 18:16:05 +08:00
Cheng Wan 3a5f52e144 Record a process's placement at publish, not at group build (#40071) 2026-09-18 23:52:37 -07:00
677c1cbdc9 [Metrics] Propagate idle gaps across all scheduler loops (#40004)
Co-authored-by: metamergebot <324680979+metamergebot@users.noreply.github.com>
Co-authored-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
2026-09-18 22:42:00 -07:00
8189e3896b [PD] Skip singleton transfer-status all-reduces (#40003)
Co-authored-by: Pranjal Shankhdhar <pranjalssh@users.noreply.github.com>
Co-authored-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
2026-09-18 22:41:50 -07:00
Vincent Liu 090263eff6 [VLM] avoid CUDA placement on non-CUDA platforms (#38750) 2026-09-19 11:13:22 +08:00
Xiaoyu Zhang 986959e3c4 [Refactor] Deduplicate kernel helpers and remove unused code (#40197) 2026-09-19 09:27:30 +08:00
5931fd60ee Support unified memory page-envelope transfers in PD (#39477)
Co-authored-by: yhzhuang <yhzhuang@fb.com>
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
Co-authored-by: Yonghao Zhuang <yhzhuang@users.noreply.github.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-09-18 17:39:50 -07:00
a6cf05817f dsv4.1: remaining model and runtime integration (#38798)
Co-authored-by: BBuf <1182563586@qq.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Xiaoyu Zhang <xiaoyu.zhang@radixark.ai>
Co-authored-by: Yuwei An <ayw.sirius19@gmail.com>
Co-authored-by: Khoa Pham <khoa.pham@radixark.ai>
Co-authored-by: Yuhao Yang <47235274+yhyang201@users.noreply.github.com>
Co-authored-by: Zhichen Zeng <zczeng@uw.edu>
Co-authored-by: Ke Bao <ispobaoke@gmail.com>
2026-09-18 02:55:30 -07:00