Commit Graph
87 Commits
Author SHA1 Message Date
4a4e1ce014 [Router] Hold a booting rank's batches and graft a snapshot on the pump (7/13) (#40693)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-29 20:07:37 +08:00
8d1643b21e [Router] Vet a peer's snapshot before it may touch the tree (6/13) (#40692)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-29 19:58:11 +08:00
8b3a4cad8b [Router] Track a booting rank's bootstrap and fetch a peer's snapshot (5/13) (#40691)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-29 19:06:13 +08:00
7bcfcf5aa0 [Router] Watch EndpointSlices for sibling router replicas (4/13) (#40690)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-29 18:56:32 +08:00
c5a38d3998 [sgl-router] Add --tokenizer-backend fast and --tokenizer-l1-cache-mb (#41268)
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 18:55:58 +08:00
6aacca2d91 [Router] Name a replica's siblings with --kv-peer-selector (3/13) (#40689)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-28 18:49:09 +08:00
2f88c65289 [Router] Serve the cache-aware tree at /internal/kv_snapshot (2/13) (#40688)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-28 18:38:52 +08:00
47ad1928a2 [sgl-router] Scope input_ids forwarding by renderer: all text chats for DeepSeek-V4 (#41226)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-09-28 18:37:40 +08:00
df378bca42 [Router] Give the cache-aware tree a snapshot surface (1/13) (#40687)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-28 16:59:59 +08:00
b4584ab838 [sgl-router] Take SGLang's render defaults: --default-chat-template-kwargs and thinking/effort envs (#41221)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-27 11:01:17 +08:00
Shangming CaiandClaude Fable 5 710a41ba57 [sgl-router] Book the input_ids forwarding outcome only for built bodies (#41342)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-26 19:39:01 +08:00
7152aabdcb [sgl-router] Track input_ids forwarding outcomes per chat request (#41185)
Co-authored-by: Kan Wu <kan.wu@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 13:09:46 +08:00
Kan WuandClaude Fable 5.1 2e3568567b [sgl-router] Launch reorg routing with existing policy options (#40766)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-23 10:18:11 +08:00
5c18780c31 [Router] Keep e2e workers inside the job's CUDA_VISIBLE_DEVICES allotment (#40686)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 11:32:12 -07:00
2032f3a071 [Router] Abort the engine when a client disconnects mid-request (#39461)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Kan Wu <wukanustc@gmail.com>
2026-09-22 12:35:56 +08:00
Kan Wu 59a723ef1e [sgl-router] refactor - SLO ordering for bucket selection (#40292) 2026-09-22 11:16:10 +08:00
Kan Wu 27f796ca6c [sgl-router] Fix readiness, IPv6 discovery, logging, and model validation (#40604) 2026-09-22 11:13:06 +08:00
Kan Wu d47b8c454c [sgl-router] Release cancelled circuit-breaker probes (#40603) 2026-09-21 15:39:50 -07:00
Kan WuandShangming Cai 008470abd8 [sgl-router] Bound streaming lifetimes and release guards on idle disconnect (#40391)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-09-21 10:07:07 -07:00
Kan Wu 0f6761b54f [sgl-router] Add SGLang-compatible DeepSeek V4.1 Flash rendering (#40532) 2026-09-21 18:35:09 +08:00
0abb251a20 [sgl-router] Match DeepSeek V4 rendering to SGLang (#40530)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-09-21 17:56:41 +08:00
Kan WuandClaude Fable 5.1 2016f5e7a1 [sgl-router] Add Kimi-K3 rendering with SGLang parity (#40390)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-21 17:38:49 +08:00
630b1ef322 [sgl-router] Fix reorg admission proxy test build after BucketResolver::new (#40537)
Co-authored-by: Kan Wu <wukanustc@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-21 16:42:29 +08:00
Kan WuandClaude Fable 5.1 a9871012ac [sgl-router] refactor - generalized admission policy definitions (#40271)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-21 16:26:30 +08:00
Kan WuandClaude Fable 5.1 fcb080bd40 [sgl-router] refactor - cache-aware policy (#40366)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-21 13:53:57 +08:00
Kan WuandClaude Fable 5.1 4027740569 [sgl-router] refactor - session-aware policy (#40379)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 17:22:36 -07:00
Kan WuandClaude Fable 5.1 aedda8377e [sgl-router] refactor - layout BucketResolver, Bucket, EngineGroup and implement PowerOfTwo (#40241)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 16:57:07 -07:00
Kan WuandClaude Fable 5.1 4a9dc5c4af [sgl-router] refactor - move policy-required states under src/state (#40272)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 16:49:23 -07:00
Kan Wu c610c40399 [sgl-router] refactor - config and organize CLI options (#39867) 2026-09-20 19:40:04 +08:00
Kan WuandCursor 671630abf1 [sgl-router] refactor - main startup logic (#39861)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-20 16:55:34 +08:00
Kan Wu 99d53fe0c2 [sgl-router] refactor - chat_completions() into modules (#39848) 2026-09-20 13:51:25 +08:00
aed3fb1cdd [Router] Log every request at one site; derive its outcome from the final status (3/3) (#39465)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-09-19 03:11:00 +08:00
4e0b56c811 [Router] Treat an upstream 503/429 as backpressure, not a breaker fault (2/3) (#39464)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-19 00:46:29 +08:00
6de4666e43 [Router] Derive error status from a failure class; preserve the worker's status (1/3) (#39463)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 16:32:45 +08:00
3bf243d6a9 [Router] Shard the cache-aware KV tree by chain root (#39167)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-09-18 15:00:17 +08:00
72d9419bef [Router] Sample k random candidates for the min-load fallback (--min-load-choices) (#39170)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-09-18 03:09:09 +08:00
e4cbb28ea1 [router] Improve SGLang chat render parity (#39133)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-09-17 23:53:40 +08:00
Kan WuandClaude Fable 5.1 3ce7e2a29f [sgl-router] Render chat prompts with dynamo-render (#38983)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-17 15:57:20 +08:00
b02e16a895 [Router] Fleet-wide sampling contract 3/3: splice injection without re-serializing (#39002)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 09:59:48 -07:00
3e03879f68 [Router] Drain readiness before SIGTERM shutdown so k8s deregisters the pod first (#39016)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 09:59:26 -07:00
Kan Wu 279339f113 [sgl-router] Share model-file discovery for chat formatters (#39485) 2026-09-16 09:36:11 -07:00
Kan WuandClaude Fable 5.1 ad94978adf [sgl-router] Rename chat encoder to chat formatter (#39459)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 08:46:11 -07:00
Kan WuandClaude Fable 5.1 e41026f434 [sgl-router] Forward input_ids only for string content; count tokenize errors only when forwardable (#39458)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-16 22:49:41 +08:00
8baeded6f3 [Router] Pin to the prefix owner when the whole fleet is queueing (--saturation-queue-floor) (#39169)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:16:24 +08:00
54f0d72adf [Router] Fleet-wide sampling contract 2/3: enforce and inject per request (#39001)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 00:29:33 -07:00
7835f1de9a [Router] Add the shutdown-drain configuration surface (#39015)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 00:26:50 -07:00
e2d56bbbfc [Router] Bound the e2e worker memory budget so prefill graph capture stops OOMing (#39713)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 00:13:09 -07:00
d634320e48 [Router] Count open HTTP exchanges until their response body finishes (#39014)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 00:04:38 -07:00
afde31a2f5 [router] Speak cleartext h2c on both edges: serve it inbound, forward it outbound (#39006)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-15 22:30:31 -07:00
c1f5b4736a [Router] Add --worker-queue-limit: stop sending cache-affinity traffic to a queueing worker (#39168)
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 13:24:48 +08:00