 Simo Lin
|
458c0219a6
|
[router] simplify tokenizer dev doc (#10895)
|
2025-09-24 22:15:56 -07:00 |
|
|
Simo Lin
|
e738703547
|
[router] consolidate worker get loads (#10880)
|
2025-09-24 22:13:31 -04:00 |
|
|
Simo Lin
|
7a06ef984d
|
[router] consolidate health endpoints and flush cache (#10876)
|
2025-09-24 15:23:21 -07:00 |
|
|
Simo Lin
|
b24b2e7ed7
|
[router] use dashmap for radix tree instead of hash for multi model (#10814)
|
2025-09-23 11:25:53 -07:00 |
|
|
Simo Linandkey4ng
|
98c3b04ff2
|
[router] responses api POST and GET with local storage (#10581)
Co-authored-by: key4ng <rukeyang@gmail.com>
|
2025-09-23 09:12:02 -07:00 |
|
|
Simo Lin
|
ddab4fc7c7
|
[router] fix cache aware routing strategy and lock contention (#10773)
|
2025-09-23 08:53:49 -07:00 |
|
|
Simo Lin
|
c3a1d7759f
|
[router] remove pd router draining channel (#10767)
|
2025-09-22 20:49:33 -07:00 |
|
|
Simo LinandChang Su
|
89971c4c3c
|
[router] refactor router and worker management 4/n (#10756)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
|
2025-09-22 18:35:10 -07:00 |
|
|
Simo Lin
|
7ca1bea63d
|
[router] update ci so only execute benchmarks when labels are added (#10757)
|
2025-09-22 13:23:07 -07:00 |
|
|
Simo Lin
|
97c3823931
|
[router] refactor router and worker management 3/n (#10727)
|
2025-09-22 12:17:50 -07:00 |
|
|
Simo Lin
|
1d1ce62495
|
[router] refactor router and worker management 2.5/n (#10677)
|
2025-09-19 20:54:40 -07:00 |
|
|
Simo Lin
|
00eb5eb721
|
[router] refactor router and worker management 2/n (#10666)
|
2025-09-19 12:37:57 -07:00 |
|
|
Simo Lin
|
36efd5be8a
|
[router] refactor router and worker management 1/n (#10664)
|
2025-09-19 06:19:57 -07:00 |
|
|
Simo Lin
|
873d858b28
|
[router] refactor worker to builder pattern 5/n (#10653)
|
2025-09-19 05:43:23 -04:00 |
|
|
Simo Lin
|
4f2055ad56
|
[router] refactor worker to builder pattern 4/n (#10650)
|
2025-09-18 23:49:10 -07:00 |
|
|
Simo Lin
|
ac2a723bb3
|
[router] refactor worker to builder pattern 3/n (#10647)
|
2025-09-18 22:52:57 -07:00 |
|
|
Simo Lin
|
780d6a22cd
|
[router] refactor worker to builder pattern 2/n (#10633)
|
2025-09-18 21:47:56 -07:00 |
|
|
Simo Lin
|
5291f32d75
|
[router] refactor worker to builder pattern 1/n (#10628)
|
2025-09-18 13:25:40 -07:00 |
|
|
Simo Lin
|
16e9335998
|
[router] add router db connector for responses api (#10487)
|
2025-09-15 22:04:56 -07:00 |
|
|
Simo Lin
|
7eccbe992d
|
[router] fix service discovery and mcp ut (#10449)
|
2025-09-14 21:07:23 -07:00 |
|
|
Simo Lin
|
7c5a0a1b77
|
[router] add not implemented functions for multi model trait (#10394)
|
2025-09-12 16:44:18 -07:00 |
|
|
Simo Lin
|
2f173ea074
|
[router] allow one router to support different model families and serving mode (#10244)
|
2025-09-12 16:18:27 -07:00 |
|
|
Simo Lin
|
07bcad7fb7
|
[bug] fix router ci syntax error (#10390)
|
2025-09-12 14:39:15 -07:00 |
|
|
Simo Lin
|
8c86595c93
|
[router] enable sccache in ci and local build (#10099)
|
2025-09-12 09:43:48 -07:00 |
|
|
Simo Lin
|
db37422c92
|
[router] move to mcp sdk instead (#10057)
|
2025-09-05 18:03:46 -07:00 |
|
|
Simo Lin
|
bde73ee43f
|
[router] add rust cache in benchmark ci (#10080)
|
2025-09-05 09:59:36 -07:00 |
|
|
Simo Lin
|
bbf261ae4a
|
[router] fix grpc connection mode detection (#9999)
|
2025-09-03 21:36:16 -07:00 |
|
|
Simo Lin
|
4f8a982d52
|
[router] clean up dependency injector to use ctx (#10000)
|
2025-09-03 21:35:51 -07:00 |
|
|
Simo Lin
|
d966b902af
|
[router] move tokenizer, reasoning, tool initialization to server (#9996)
|
2025-09-03 19:35:13 -07:00 |
|
|
Simo Lin
|
9491d6e554
|
[router] include rust benchamrks (#9932)
|
2025-09-02 09:32:09 -07:00 |
|
|
Simo Lin
|
92d79646e5
|
[router] add reasoning parser readme (#9837)
|
2025-08-30 21:06:23 -07:00 |
|
|
Simo Lin
|
5343058875
|
[router] grpc router bootstraps (#9759)
|
2025-08-28 12:07:06 -07:00 |
|
|
Simo Lin
|
07c9d8fba2
|
[router] add llama3.2 multi json streaming parser (#9735)
|
2025-08-28 05:57:13 -07:00 |
|
|
Simo Lin
|
e1f7cf57dc
|
[router] additional llama32 parser unit test and multi json support (#9732)
|
2025-08-27 20:34:11 -07:00 |
|
|
Simo Lin
|
2bb9d454b5
|
[router] additional pythonic parser unit test (#9730)
|
2025-08-27 19:55:59 -07:00 |
|
|
Simo LinandChang Su
|
07ee0ab750
|
[router] add gpt-oss and glm4 tool parser (#9703)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
|
2025-08-27 11:26:00 -07:00 |
|
|
Simo LinandChang Su
|
5c06dcb75a
|
[router] add kimi-k2 tool parser (#9702)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
|
2025-08-27 11:04:55 -07:00 |
|
|
Simo LinandChang Su
|
6f6beca49d
|
[router] add step3 tool parser (#9695)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
|
2025-08-27 10:44:52 -07:00 |
|
|
Simo LinandChang Su
|
6e4e1c8cdc
|
[router] add deepseek tool parser (#9694)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
|
2025-08-27 06:18:24 -07:00 |
|
|
Simo Lin
|
9768c50d90
|
[router] restructure tool parser module folder (#9693)
|
2025-08-27 06:05:53 -07:00 |
|
|
Simo Linandfzyzcjy
|
3578eb1e9b
|
[router] address worker load tracking consistency (#9523)
Co-authored-by: fzyzcjy <5236035+fzyzcjy@users.noreply.github.com>
|
2025-08-26 06:40:51 -07:00 |
|
|
Simo LinandChang Su
|
e2e378caba
|
[router] add ut for mistral, llama, pythonic, and streaming tool parser (#9632)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
|
2025-08-25 22:02:15 -07:00 |
|
|
Simo LinandChang Su
|
dc1decc6af
|
[router] add llama tool parser (#9629)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
|
2025-08-25 20:43:36 -07:00 |
|
|
Simo LinandChang Su
|
03680f33be
|
[router] add pythonic parser (#9628)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
|
2025-08-25 20:40:06 -07:00 |
|
|
Simo LinandChang Su
|
d4c5e53401
|
[router] add qwen tool parser (#9623)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
|
2025-08-25 20:32:05 -07:00 |
|
|
Simo LinandChang Su
|
817c62a077
|
[router] add mistral tool parser (#9622)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
|
2025-08-25 20:09:51 -07:00 |
|
|
Simo Lin
|
f556ac8bd8
|
[router] add json tool parser (#9516)
|
2025-08-22 12:13:04 -07:00 |
|
|
Simo Lin
|
49f9d02538
|
[router] tokenizer arch doc (#9513)
|
2025-08-22 09:52:33 -07:00 |
|
|
Simo Lin
|
0f8cee8cd3
|
[router] fix router load guard tracking for streaming (#9491)
|
2025-08-21 22:48:29 -07:00 |
|
|
Simo Lin
|
78ae175866
|
[router] add tokenizer benchmark (#9427)
|
2025-08-21 11:09:39 -07:00 |
|
|
Simo LinandChang Su
|
5fbad308cd
|
[router] add tokenizer chat template support (#9370)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
|
2025-08-19 20:14:02 -07:00 |
|
|
Simo Lin
|
b45f753cba
|
[router] adds reasoning parser pooling and thread-safe (#9360)
|
2025-08-19 13:35:39 -07:00 |
|
|
Simo LinandChang Su
|
0b95a01a8f
|
[router] add tiktokenizer and sequence in router (#9354)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
|
2025-08-19 10:46:28 -07:00 |
|
|
Simo Lin
|
94959237bf
|
[router] add dsr1, kimi, and qwen reasoning parser (#9353)
|
2025-08-19 10:15:24 -07:00 |
|
|
Simo LinandChang Su
|
6e316588f8
|
[router] add reasoning parser base structure (#9310)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
|
2025-08-18 09:26:09 -07:00 |
|
|
Simo LinandChang Su
|
24247b4168
|
[router] add tokenizer metrics (#9307)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
|
2025-08-18 09:25:51 -07:00 |
|
|
Simo LinandChang Su
|
d08663eec1
|
[router] tokenizer factory, hf tokenizer, and stop sequence detector (#9293)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
|
2025-08-17 22:38:38 -07:00 |
|
|
Simo Lin
|
ff0cf51c8e
|
[router] introducing tokenizer trait (#9287)
|
2025-08-17 16:30:01 -07:00 |
|
|
Simo Lin
|
b341b7dbce
|
[router] introduce prefill response draining for http compliance (#9281)
|
2025-08-17 14:23:04 -07:00 |
|
|
Simo Lin
|
bb10e3a1c3
|
[router] fix pd prefill http request complinace issue (#9237)
|
2025-08-16 22:36:45 -07:00 |
|
|
Simo Lin
|
da53e13cbb
|
[router] preserve original worker response header in router (#9236)
|
2025-08-15 11:01:47 -07:00 |
|
|
Simo Lin
|
21b8846066
|
[router] allow more health check configuration (#9198)
|
2025-08-15 08:07:45 -07:00 |
|
|
Simo Lin
|
6337d9057c
|
[router] optimize Rust compilation and development workflow (#9133)
|
2025-08-13 05:14:25 -07:00 |
|
|
Simo Lin
|
1ce30dd13e
|
[router] update router documentation (#9121)
|
2025-08-12 13:16:34 -07:00 |
|
|
Simo Lin
|
9d68bdb240
|
[router] Add Rust Binary Entrypoint for SGLang Router (#9089)
|
2025-08-11 21:37:36 -07:00 |
|
|
Simo Lin
|
38907fe639
|
refactor(pd-router): extract common patterns to reduce code duplication (#9081)
|
2025-08-11 13:32:31 -07:00 |
|
|
Simo Lin
|
6f81a710f7
|
[pd-router] add retry and circuit breakfor for pd router (#9051)
|
2025-08-11 05:53:26 -07:00 |
|
|
Simo Lin
|
067068f271
|
[router] regular router circuit breaker (#8997)
|
2025-08-10 21:19:30 -07:00 |
|
|
Simo Lin
|
ce9cf35327
|
[router] update pyo3 version to 0.25.1 (#9022)
|
2025-08-10 06:45:51 -07:00 |
|
|
Simo Lin
|
473400e452
|
[router] upgrade kube version to latest (#9018)
|
2025-08-09 22:49:45 -07:00 |
|
|
Simo Lin
|
dd665f967f
|
[router] upgrade rand to latest version (#9017)
|
2025-08-09 22:49:30 -07:00 |
|
|
Simo Lin
|
3817a37d87
|
[router] upgrade to latest sgl kernel for router ci (#9019)
|
2025-08-09 21:49:18 -07:00 |
|
|
Simo Lin
|
7b7e56150e
|
[router] fix radix tree integration issues in PD router (#8982)
|
2025-08-08 14:47:51 -07:00 |
|
|
Simo Lin
|
1a8706c8b9
|
[router] reduce contention, fix double-count race (#8978)
|
2025-08-08 14:29:14 -07:00 |
|
|
Simo Lin
|
a59cbea92d
|
[router] harden retries + metrics; fix streaming load; header filtering (#8972)
|
2025-08-08 13:10:14 -07:00 |
|
|
Simo Lin
|
61a4680494
|
[router] router circuit breaker core (#8941)
|
2025-08-08 09:20:22 -07:00 |
|
|
Simo Lin
|
2e901e892f
|
[router] dedicated prefill HTTP client and request-path optimizations (#8923)
|
2025-08-07 17:31:45 -07:00 |
|
|
Simo Lin
|
500b15c960
|
[router] upgrade router version to 0.1.9 (#8844)
|
2025-08-07 09:29:12 -07:00 |
|
|
Simo Lin
|
16a4c66d25
|
[router] update pd router ci summary step with new threshold (#8916)
|
2025-08-07 07:15:38 -07:00 |
|
|
Simo Lin
|
89e6521c61
|
[router] re-enable pd router benchmark CI (#8912)
|
2025-08-07 06:29:36 -07:00 |
|
|
Simo Lin
|
a69b637014
|
[router] fix req handling order, improve serialization, remove retry (#8888)
|
2025-08-06 23:24:39 -07:00 |
|
|
Simo Lin
|
8c7bb39dfb
|
[router] PD Router Simplification and Reorganization (#8838)
|
2025-08-05 21:20:38 -07:00 |
|
|
Simo Lin
|
5d62b56f7e
|
[router] complete router oai spec (#8828)
|
2025-08-05 18:30:19 -07:00 |
|
|
Simo Lin
|
354ac43555
|
[pd-router] Add Configurable Retry Logic for reduce backend pressure (#8744)
|
2025-08-04 20:42:07 -07:00 |
|
|
Simo Lin
|
2fa0462c39
|
[router] introduce dp worker abstraction (#8639)
|
2025-08-04 06:42:20 -07:00 |
|
|
Simo Lin
|
828a4fe944
|
[router] Implement HTTP Dependency Injection Pattern for Router System (#8714)
|
2025-08-02 19:16:47 -07:00 |
|
|
Simo Lin
|
6d4fd8826e
|
[router] minor code clean up and and refactoring (#8711)
|
2025-08-02 13:46:31 -07:00 |
|
|
Simo Lin
|
ab9b893e61
|
[bug] limit bootstrap room to to [0, 2^63 - 1] (#8684)
|
2025-08-01 14:41:01 -07:00 |
|
|
Simo Lin
|
39decec10b
|
[router] upgrade router version to 0.1.8 (#8645)
|
2025-07-31 19:00:23 -07:00 |
|
|
Simo Lin
|
f6f46f4629
|
[router] add basic usage doc (#8640)
|
2025-07-31 18:11:48 -07:00 |
|
|
Simo Lin
|
2886e23dbd
|
[bugfix] fix router python parser for pd urls (#8644)
|
2025-07-31 18:09:31 -07:00 |
|
|
Simo Lin
|
5c14515fec
|
[bug] remove pdlb from minilb since its no longer available (#8634)
|
2025-07-31 13:54:02 -07:00 |
|
|
Simo Lin
|
aee0ef52f5
|
[router] update router pypi version (#8628)
|
2025-07-31 11:24:12 -07:00 |
|
|
Simo Lin
|
ae807774f5
|
[ci] fix genai-bench execution cmd (#8629)
|
2025-07-31 10:40:54 -07:00 |
|
|
Simo Lin
|
66a398f49d
|
[router] migrate router from actix to axum (#8479)
|
2025-07-30 17:47:19 -07:00 |
|
|
Simo Lin
|
a9fd80336d
|
[router] allow longer time out for router e2e (#8560)
|
2025-07-29 23:43:37 -07:00 |
|
|
Simo Lin
|
fe6a445d1e
|
[router] improve router logs and request id header (#8415)
|
2025-07-27 19:30:19 -07:00 |
|
|
Simo Lin
|
2ab97023e3
|
[router] add different policies for p node and d node (#8395)
|
2025-07-27 00:39:20 -07:00 |
|
|
Simo Lin
|
39fe1e880d
|
[router] add request format unit test (#8300)
|
2025-07-24 14:30:37 -07:00 |
|
|
Simo Lin
|
33c4b4d04e
|
[router] add streaming unit test (#8299)
|
2025-07-24 14:30:27 -07:00 |
|