 fzyzcjy
|
9c2530642c
|
Change routing policy API to be async to support more policies (#17048)
|
2026-01-17 17:12:51 +08:00 |
|
|
fzyzcjy
|
a7b5f75d88
|
Support integration tests with Redis binary (#17045)
|
2026-01-17 11:59:04 +08:00 |
|
|
fzyzcjy
|
305c1a575a
|
Tiny remove unused code (#17047)
|
2026-01-17 11:58:43 +08:00 |
|
|
fzyzcjy
|
c824ddd581
|
Fix imbalancedness for manual policy min group mode for requests without routing id (#17044)
|
2026-01-17 11:58:35 +08:00 |
|
|
fzyzcjy
|
ff3ddb9d9b
|
Support min num routing keys in key-based load balancing policy (#16564)
|
2026-01-12 21:38:03 -08:00 |
|
|
fzyzcjy
|
9d3018f484
|
Support min load besides random routing key assignment policy in ManualPolicy (#16767)
|
2026-01-12 18:31:18 -08:00 |
|
|
fzyzcjy
|
1f0e3d7fd8
|
Support tracking worker routing key loads in gateway (#16765)
|
2026-01-13 08:07:17 +08:00 |
|
|
fzyzcjyandSimo Lin
|
38b30c7b56
|
Tiny refactor age computation in router (#16850)
Co-authored-by: Simo Lin <linsimo.mark@gmail.com>
|
2026-01-11 21:42:43 -08:00 |
|
|
fzyzcjy
|
a1c48943d7
|
Tiny fix NoAvailableWorkers being a RetryError (#16896)
|
2026-01-11 18:53:17 -08:00 |
|
|
fzyzcjy
|
dae6a4092a
|
Tiny add scheduler status logging (#16872)
|
2026-01-10 20:12:24 +08:00 |
|
|
fzyzcjy
|
a0899bdbd8
|
Fix log_decode_stats_every_iteration when having TP in attention (#16871)
|
2026-01-10 20:10:23 +08:00 |
|
|
fzyzcjy
|
641830c1c2
|
Tiny extract file logging utils (#16870)
|
2026-01-10 20:02:37 +08:00 |
|
|
fzyzcjy
|
c89949bbaf
|
Tiny let soft watchdog cover initialization phase (#16853)
|
2026-01-10 13:20:08 +08:00 |
|
|
fzyzcjy
|
32a569fb77
|
Tiny add CPU resource monitoring for overload diagnosis (#16852)
|
2026-01-10 12:48:18 +08:00 |
|
|
fzyzcjy
|
3ed3b7ef7c
|
Tiny add routing key distribution metrics (#16847)
|
2026-01-10 12:06:40 +08:00 |
|
|
fzyzcjy
|
1f9d4795a9
|
Tiny add gauge histogram abstraction for engine and router (#16848)
|
2026-01-10 11:45:25 +08:00 |
|
|
fzyzcjy
|
1f0ea4f958
|
Add routing key based schedule policy (#16840)
|
2026-01-10 10:16:23 +08:00 |
|
|
fzyzcjy
|
15da3061bc
|
Tiny pass routing key to scheduler processes (#16839)
|
2026-01-10 10:13:00 +08:00 |
|
|
fzyzcjy
|
6406a5969b
|
Tiny support customizing prometheus buckets for prefill delayer (#16831)
|
2026-01-10 08:17:37 +08:00 |
|
|
fzyzcjy
|
cec19b56c6
|
Tiny add command line args for prefill delayer and unify names (#16830)
|
2026-01-10 07:54:25 +08:00 |
|
|
fzyzcjy
|
8eeffbe9aa
|
Support token low usage watermark in prefill delayer (#16814)
|
2026-01-09 23:07:18 +08:00 |
|
|
fzyzcjy
|
71e9c31cb8
|
Tiny add and enhance tests for prefill delayer (#16813)
|
2026-01-09 22:58:38 +08:00 |
|
|
fzyzcjy
|
7656d26737
|
Tiny enhance prefill delayer observability (#16812)
|
2026-01-09 22:51:16 +08:00 |
|
|
fzyzcjy
|
dd24ba9084
|
Refactor prefill delayer for clarity and extensibility (#16811)
|
2026-01-09 22:46:41 +08:00 |
|
|
fzyzcjy
|
9e3a032ad6
|
[smg] cleanup router RAII guards (#16560)
|
2026-01-08 16:39:53 -08:00 |
|
|
fzyzcjy
|
b6e8a0d851
|
Add file size hints in bitwise model file verifier (#16735)
|
2026-01-08 22:28:52 +08:00 |
|
|
fzyzcjy
|
83abecd0c2
|
Support pre-generating and using expected checksums (#16730)
|
2026-01-08 20:26:13 +08:00 |
|
|
fzyzcjy
|
d54f0a10b4
|
Support bitwise weight checksum verifier (#16729)
|
2026-01-08 20:19:25 +08:00 |
|
|
fzyzcjy
|
3be1e734ee
|
[model-gateway] extract header extraction in policy and add (#16566)
|
2026-01-06 20:18:44 -08:00 |
|
|
fzyzcjy
|
d874c8bba4
|
Tiny support http headers in bench serving (#16606)
|
2026-01-07 10:15:17 +08:00 |
|
|
fzyzcjy
|
9a21d89c5b
|
Tiny add metrics for prefill delayer (#16603)
|
2026-01-07 09:53:52 +08:00 |
|
|
fzyzcjy
|
ed307a40bf
|
Tiny add --gsp-ordered (#16575)
|
2026-01-06 22:43:52 +08:00 |
|
|
fzyzcjy
|
f959250f76
|
Tiny support sglang_routing_keys_active in engine (#16570)
|
2026-01-06 22:10:10 +08:00 |
|
|
fzyzcjy
|
c105a3124b
|
Support multi-round conversations in bench_serving (#6135)
|
2026-01-06 11:59:39 +08:00 |
|
|
fzyzcjy
|
1e41069ad1
|
Fix age bucket rendering issue (#16492)
|
2026-01-05 09:29:14 -08:00 |
|
|
fzyzcjy
|
7f35c46efb
|
Tiny add sglang:http_requests_active metric (#16479)
|
2026-01-05 17:20:31 +08:00 |
|
|
fzyzcjy
|
4308c25b47
|
Super tiny move test folder (#16447)
|
2026-01-05 16:29:26 +08:00 |
|
|
fzyzcjy
|
4d737db857
|
Tiny fix prefill delayer not support non-fcfs schedule policy (#16471)
|
2026-01-05 16:28:54 +08:00 |
|
|
fzyzcjy
|
9d6029fb92
|
Fix TokenizerManager bottleneck for offline generation (#16456)
|
2026-01-05 16:28:25 +08:00 |
|
|
fzyzcjy
|
c63e9cb29e
|
Super tiny fix CI (#16437)
|
2026-01-04 20:34:49 -08:00 |
|
|
fzyzcjy
|
1e7b326482
|
Super tiny fix main code (#16432)
|
2026-01-05 10:34:25 +08:00 |
|
|
fzyzcjy
|
249c356331
|
Super tiny update tokenizer benchmark (#16429)
|
2026-01-05 09:14:52 +08:00 |
|
|
fzyzcjy
|
12df16607b
|
Tiny speed up kimi detokenizer by 10x (#16427)
|
2026-01-05 09:12:05 +08:00 |
|
|
fzyzcjy
|
f16606d6f6
|
Super tiny code cleanup (#16401)
|
2026-01-04 23:00:09 +08:00 |
|
|
fzyzcjy
|
387fad2f74
|
Tiny add detokenization benchmarks (#16400)
|
2026-01-04 22:53:38 +08:00 |
|
|
fzyzcjy
|
27a0830593
|
Tiny add manual policy benchmark to CI (#16392)
|
2026-01-04 19:10:35 +08:00 |
|
|
fzyzcjy
|
53479e226b
|
Tiny refactor router benchmark github actions (#16374)
|
2026-01-04 18:00:12 +08:00 |
|
|
fzyzcjy
|
1b2b95d8d1
|
Super tiny cleanup unused function (#16376)
|
2026-01-04 15:08:12 +08:00 |
|
|
fzyzcjy
|
d7aa0ce72f
|
Support multiple engines for router simulation in schedule simulator (#16368)
|
2026-01-04 11:46:05 +08:00 |
|
|
fzyzcjy
|
e797f0c570
|
Support offline generation scenario for prefill delayer (#16363)
|
2026-01-04 11:05:10 +08:00 |
|
|
fzyzcjy
|
5d4b7c78bf
|
Fix memory leak in prefill delayer (#16358)
|
2026-01-04 10:11:18 +08:00 |
|
|
fzyzcjy
|
24e116ef6a
|
Enhance schedule simulator and add several e2e scenarios (#16357)
|
2026-01-04 10:02:38 +08:00 |
|
|
fzyzcjy
|
8ca9597057
|
Tiny support sticky routing algorithm in schedule simulator (#16355)
|
2026-01-04 08:36:52 +08:00 |
|
|
fzyzcjy
|
216ea910ad
|
Tiny support forwarding SMG routing key to engine for dumping and avoid string allocation (#16352)
|
2026-01-04 08:33:35 +08:00 |
|
|
fzyzcjy
|
ef5ac6f01e
|
Support shared prefix (gsp) in schedule simulator (#16353)
|
2026-01-04 08:03:16 +08:00 |
|
|
fzyzcjy
|
c88aaf22c7
|
Support in-flight request age metrics for router (#16341)
|
2026-01-04 07:38:53 +08:00 |
|
|
fzyzcjy
|
e139d2aa76
|
Super tiny fix CI (#16351)
|
2026-01-04 07:24:46 +08:00 |
|
|
fzyzcjy
|
2337b1bbb0
|
Refactor and fix prefill delayer (scheduler enhancer) (#16269)
|
2026-01-04 07:15:06 +08:00 |
|
|
fzyzcjy
|
877c8e3a96
|
Tiny refactor router test contexts (#16340)
|
2026-01-03 09:44:10 -08:00 |
|
|
fzyzcjyandgemini-code-assist[bot]
|
66dfb8c156
|
Tiny fix non-PD router http header missing whitelist (#16339)
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
|
2026-01-03 09:43:02 -08:00 |
|
|
fzyzcjy
|
87ef05e2e1
|
Support simple schedule simulator (#16344)
|
2026-01-03 22:05:40 +08:00 |
|
|
fzyzcjy
|
b65c9889a0
|
Fix incorrect running batch size in prefill stats (#15941)
|
2026-01-03 20:49:54 +08:00 |
|
|
fzyzcjy
|
c7c0d97fc6
|
Tiny support whitelisted headers in request logging (#16342)
|
2026-01-03 20:10:58 +08:00 |
|
|
fzyzcjy
|
7f2fa2167b
|
Tiny add --log-requests-target (#16338)
|
2026-01-03 17:28:27 +08:00 |
|
|
fzyzcjy
|
9821fae5c6
|
Tiny add explanations to realtime token metric labels (#16009)
|
2026-01-03 12:36:55 +08:00 |
|
|
fzyzcjy
|
749736ba95
|
Tiny fix single-gpu dumper and add tests for dumper (#16285)
|
2026-01-02 13:30:09 +08:00 |
|
|
fzyzcjy
|
00562ee14a
|
Support cache eviction for Manual Policy (#16263)
|
2026-01-02 08:52:01 +08:00 |
|
|
fzyzcjy
|
db499e1889
|
Tiny add filter, support duplications, add visualizations, fix error and robustness for dump comparator (#16262)
|
2026-01-01 17:47:00 +08:00 |
|
|
fzyzcjy
|
6cf3a6dd69
|
Support HTTP control for dumper (#16261)
|
2026-01-01 17:40:17 +08:00 |
|
|
fzyzcjy
|
90e24f5c31
|
Tiny add filter, dump dict, failable save to dumper (#16260)
|
2026-01-01 17:35:38 +08:00 |
|
|
fzyzcjy
|
d90f9bfc4e
|
Temporarily disable temp_prefill_info assertion to unblock CI (#16008)
|
2025-12-28 20:13:06 +08:00 |
|
|
fzyzcjy
|
8fab48952e
|
Tiny fix WASM test errors on machines with many cores (#15992)
|
2025-12-28 17:12:04 +08:00 |
|
|
fzyzcjy
|
7ccaec6467
|
Add micro benchmarks for manual policy (#15991)
|
2025-12-28 17:11:56 +08:00 |
|
|
fzyzcjy
|
26c5091217
|
Tiny extract PeriodicTask in router (#15988)
|
2025-12-28 14:07:13 +08:00 |
|
|
fzyzcjy
|
325a4c1945
|
Tiny add smg_manual_policy_cache_entries metric (#15987)
|
2025-12-28 14:07:07 +08:00 |
|
|
fzyzcjy
|
f55d608c89
|
Tiny fix cannot launch nvfp4 checkpoint with bf16 kv cache (#15986)
|
2025-12-28 11:22:20 +08:00 |
|
|
fzyzcjy
|
5d421db883
|
Tiny log warn users when tracing is automatically disabled (#15889)
|
2025-12-26 19:00:37 +08:00 |
|
|
fzyzcjy
|
0271fc3456
|
Fix prefill num tokens metrics (#15858)
|
2025-12-26 10:26:42 +08:00 |
|
|
fzyzcjy
|
68bece8caa
|
Super tiny move last_prefill_tokens to metrics mixin (#15857)
|
2025-12-26 10:26:10 +08:00 |
|
|
fzyzcjy
|
6ef543f99d
|
Use X-SMG-Routing-Key header instead of json body and add tests (#15826)
|
2025-12-25 09:58:14 -08:00 |
|
|
fzyzcjy
|
c12567274a
|
Tiny change bench-serving to use routing key header (#15827)
|
2025-12-25 21:29:49 +08:00 |
|
|
fzyzcjy
|
ccf2330b28
|
Tiny fix missing record_router_upstream_response (#15811)
|
2025-12-25 15:20:43 +08:00 |
|
|
fzyzcjy
|
45adad37d0
|
Add manual routing policy for router (#15586)
|
2025-12-25 11:27:36 +08:00 |
|
|
fzyzcjy
|
1ba897f330
|
Tiny refactor select_workers API for future passing more information (#15596)
|
2025-12-25 10:31:10 +08:00 |
|
|
fzyzcjy
|
9338f63f7d
|
Add request counter in addition to existing response counter (#15768)
|
2025-12-24 22:22:38 +08:00 |
|
|
fzyzcjy
|
8196998a03
|
Tiny add num retracted tokens metric (#15653)
|
2025-12-24 22:15:23 +08:00 |
|
|
fzyzcjy
|
aa21c6e3f1
|
Super tiny code cleanup (#15652)
|
2025-12-24 22:15:00 +08:00 |
|
|
fzyzcjy
|
fd4a558e71
|
Add metrics for having prefill and decode in different ranks (#15752)
|
2025-12-24 21:35:35 +08:00 |
|
|
fzyzcjy
|
b3b818fd86
|
Tiny unify realtime_tokens_total metric (#15747)
|
2025-12-24 21:29:43 +08:00 |
|
|
fzyzcjy
|
e245cac0c5
|
Support JSON format request logging for easier parsing (#15743)
|
2025-12-24 19:52:43 +08:00 |
|
|
fzyzcjy
|
c6a6ba4368
|
Tiny refactor request logger (#15740)
|
2025-12-24 19:51:26 +08:00 |
|
|
fzyzcjy
|
d6108166bc
|
Fix smg_http_requests_total semantics (#15655)
|
2025-12-24 17:41:01 +08:00 |
|
|
fzyzcjy
|
99d3bcdfed
|
[model-gateway] add back router worker health metric and fix init state (#15622)
|
2025-12-23 18:50:38 -08:00 |
|
|
fzyzcjy
|
4d64f15086
|
[mode;-gateway] add back fixes of incorrect metrics after worker removal (#15624)
|
2025-12-23 18:50:03 -08:00 |
|
|
fzyzcjy
|
6a5764a719
|
Super tiny add test_soft_watchdog to nightly (#15692)
|
2025-12-23 22:46:15 +08:00 |
|
|
fzyzcjy
|
b736a1525a
|
Tiny add back missing router per attempt response metric (#15621)
|
2025-12-22 09:59:10 -08:00 |
|
|
fzyzcjy
|
6984837d82
|
Fix router gRPC mode launch error caused by async loading (#15368)
|
2025-12-22 09:39:01 -08:00 |
|
|
fzyzcjy
|
e62e17442e
|
Tiny enable soft watchdog in CI for stuck without logs (#15616)
|
2025-12-22 17:01:04 +08:00 |
|
|
fzyzcjy
|
d5431ff894
|
Tiny add stuck simulation (#15613)
|
2025-12-22 17:00:18 +08:00 |
|
|
fzyzcjy
|
454a2544f2
|
Support soft watchdog for tokenizer/detokenizer/dp-controller processes (#15607)
|
2025-12-22 16:59:16 +08:00 |
|