 Lianmin Zhengandkavioyu
|
ad20b7957e
|
Eagle speculative decoding part 3: small modifications to the general scheduler (#2709)
Co-authored-by: kavioyu <kavioyu@tencent.com>
|
2025-01-02 02:09:08 -08:00 |
|
|
Lianmin Zhengandyukavio
|
b0524c3789
|
Eagle speculative decoding part 2: Fix cuda graph + DP attention hanging (#2684)
Co-authored-by: yukavio <kavioyu@gmail.com>
|
2024-12-31 02:25:05 -08:00 |
|
|
Lianmin Zheng
|
6c42fa229d
|
Update README.md (#2683)
|
2024-12-31 00:13:10 -08:00 |
|
|
Lianmin Zhengandyukavio
|
f44d143949
|
Support target model verification in the attention backend (#2678)
Co-authored-by: yukavio <kavioyu@gmail.com>
|
2024-12-30 22:58:55 -08:00 |
|
|
Lianmin Zheng
|
339c69a243
|
Improve the computation for time_per_output_token Prometheus metrics (#2674)
|
2024-12-30 21:40:14 -08:00 |
|
|
Lianmin Zheng
|
21ec66e59e
|
Minor follow-up fixes for the logprob refactor (#2670)
|
2024-12-30 05:42:08 -08:00 |
|
|
Lianmin Zheng
|
9c6ba2484f
|
Refactor logprob computation to return the real logprob used in sampling (#2664)
|
2024-12-30 04:51:38 -08:00 |
|
|
Lianmin Zheng
|
bdd2827a80
|
Update structured_outputs.ipynb (#2666)
|
2024-12-30 00:46:41 -08:00 |
|
|
Lianmin Zheng
|
8c3b420eec
|
[Docs] clean up structured outputs docs (#2654)
|
2024-12-29 23:57:16 -08:00 |
|
|
Lianmin Zheng
|
3231817861
|
Revert "[feat] Add math eval to CI" (#2656)
|
2024-12-30 15:05:50 +08:00 |
|
|
Lianmin Zheng
|
b08c308ebc
|
Update the timeout in nightly-test.yml (#2649)
|
2024-12-29 14:51:07 -08:00 |
|
|
Lianmin Zheng
|
03d5fbfd44
|
Release 0.4.1.post3 - upload the config.json to PyPI (#2647)
|
2024-12-29 14:25:53 -08:00 |
|
|
Lianmin Zhengandkavioyu
|
9c05c6898e
|
Add llama_eagle.py (#2640)
Co-authored-by: kavioyu <kavioyu@tencent.com>
|
2024-12-29 01:45:35 -08:00 |
|
|
Lianmin Zheng
|
3815b23ccb
|
Clean up wrapper in flashinfer backend (#2638)
|
2024-12-29 00:45:57 -08:00 |
|
|
Lianmin ZhengandSangbin
|
855d0ba381
|
[CI] Fix nightly test and raise better error message (#2626)
Co-authored-by: Sangbin <rkooo567@gmail.com>
|
2024-12-27 22:16:39 -08:00 |
|
|
Lianmin Zheng
|
5276a675f5
|
Add more supporting organizations (#2623)
|
2024-12-27 13:41:41 -08:00 |
|
|
Lianmin Zheng
|
751e5ca273
|
[minor] clean up docs and eos id (#2622)
|
2024-12-27 11:23:46 -08:00 |
|
|
Lianmin Zheng
|
c75adfec59
|
Update CODEOWNERS (#2608)
|
2024-12-26 20:58:08 -08:00 |
|
|
Lianmin Zheng
|
f46f394f4d
|
Update README.md (#2605)
|
2024-12-26 10:58:49 -08:00 |
|
|
Lianmin Zheng
|
2125898af5
|
Update contributor_guide.md (#2603)
|
2024-12-26 08:36:13 -08:00 |
|
|
Lianmin Zheng
|
dc3bee4815
|
Fix test and benchmark scripts (#2598)
|
2024-12-26 07:56:26 -08:00 |
|
|
|
773951548d
|
Fix logprob_start_len for multi modal models (#2597)
Co-authored-by: libra <lihu723@gmail.com>
Co-authored-by: fzyzcjy <ch271828n@outlook.com>
Co-authored-by: Wang, Haoyu <haoyu.wang@intel.com>
|
2024-12-26 06:27:45 -08:00 |
|
|
Lianmin Zheng
|
60bd32723a
|
Update README.md (#2594)
|
2024-12-26 03:31:50 -08:00 |
|
|
Lianmin Zheng
|
23e5e50fd5
|
Fix gemlite import (#2553)
|
2024-12-22 20:21:17 -08:00 |
|
|
Lianmin Zheng
|
41b1db69b8
|
A better aio rwlock that guarantees the order (#2547)
|
2024-12-22 15:44:32 -08:00 |
|
|
Lianmin Zheng
|
8496701934
|
[Misc] Fix metrics, weight update lock, request logging (#2543)
|
2024-12-22 06:27:22 -08:00 |
|
|
Lianmin Zheng
|
21e9e63ad5
|
Print progress bar during cuda graph capture (#2502)
|
2024-12-17 06:33:46 -08:00 |
|
|
|
1fc84cf60b
|
Update readme (#2500)
Co-authored-by: Ravi Theja <ravi03071991@gmail.com>
Co-authored-by: “yixin-huang1” <yixinhuang1@berkeley.edu>
|
2024-12-17 04:33:36 -08:00 |
|
|
Lianmin Zheng
|
361ea8d912
|
Fix openai protocols and pass top_k, min_p (#2499)
|
2024-12-17 04:14:14 -08:00 |
|
|
Lianmin Zheng
|
bd6196163e
|
Small fix for the order of apply_torchao_config (#2495)
|
2024-12-16 19:21:11 -08:00 |
|
|
Lianmin Zheng
|
56198b45d9
|
Add a benchmark script for in-batch prefix caching (#2494)
|
2024-12-16 18:49:02 -08:00 |
|
|
Lianmin Zheng
|
ba36b5520a
|
Revert "Small fixes for torchao quant" (#2493)
|
2024-12-16 15:04:16 -08:00 |
|
|
Lianmin Zheng
|
9cd9dc83b3
|
Temporarily disable unit test of torch native attention backend (#2492)
|
2024-12-16 14:17:27 -08:00 |
|
|
Lianmin Zheng
|
7a1aecb938
|
Simplify pytorch sampling kernel and logit processor (#2491)
|
2024-12-16 14:11:09 -08:00 |
|
|
Lianmin Zheng
|
5282a4735f
|
[Minor] Fix grok model loader (#2473)
|
2024-12-12 14:34:47 -08:00 |
|
|
Lianmin Zheng
|
f8548295d6
|
Fix warmup in bench_offline_throughput.py (#2449)
|
2024-12-11 06:16:01 -08:00 |
|
|
Lianmin Zheng
|
959735fc9e
|
Fix model loader for more quantization formats (#2448)
|
2024-12-11 05:21:23 -08:00 |
|
|
Lianmin Zheng
|
641b7d0ae0
|
[Minor] Improve code style (#2422)
|
2024-12-09 06:30:35 -08:00 |
|
|
Lianmin Zheng
|
0ce091a82d
|
[Minor] Improve code style (#2419)
|
2024-12-09 03:05:59 -08:00 |
|
|
Lianmin Zheng
|
835f8afc77
|
Migrate llama_classification to use the /classify interface (#2417)
|
2024-12-08 23:30:51 -08:00 |
|
|
Lianmin Zheng
|
a6ca736c8e
|
Simplify stream_output (#2398)
|
2024-12-08 12:27:13 -08:00 |
|
|
Lianmin Zheng
|
cc858953a0
|
Fix recv_requests (#2405)
|
2024-12-08 04:08:04 -08:00 |
|
|
Lianmin Zheng
|
a2486eb58f
|
Fix a bug with logprob streaming + chunked prefill (#2403)
|
2024-12-08 03:55:27 -08:00 |
|
|
Lianmin Zheng
|
96db0f666d
|
Update killall_sglang.sh (#2397)
|
2024-12-08 01:56:26 -08:00 |
|
|
Lianmin Zheng
|
e5f227c0ee
|
Release v0.4.0.post1 (#2375)
|
2024-12-06 06:08:19 -08:00 |
|
|
Lianmin Zheng
|
0e7409adb6
|
Fix the overlap for xgrammar (#2377)
|
2024-12-06 05:49:29 -08:00 |
|
|
Lianmin Zheng
|
f5b2a3aa67
|
Use proc.join instead of busy waiting (#2374)
|
2024-12-06 02:01:23 -08:00 |
|
|
Lianmin Zheng
|
71e2a27753
|
Fix the cuda graph capture range for small #max-running-requests (#2359)
|
2024-12-06 14:13:57 +08:00 |
|
|
Lianmin Zheng
|
2b0fc5941d
|
[Minor] Code style improvements (#2355)
|
2024-12-04 19:02:08 -08:00 |
|
|
Lianmin Zhengandroot
|
1228f7ca69
|
Fix gptq for moe layers (#2300)
Co-authored-by: root <me@zhyncs.com>
|
2024-12-03 23:12:33 +08:00 |
|
|
Lianmin Zheng
|
07ec07ad1f
|
Improve torch compile for fused moe (#2327)
|
2024-12-03 01:58:25 -08:00 |
|
|
Lianmin Zheng
|
3ddb1c4679
|
[Minor] Fix logger and style (#2325)
|
2024-12-02 20:45:53 -08:00 |
|
|
Lianmin Zheng
|
33deca81b5
|
Add more fused moe benchmark utilities (#2314)
|
2024-12-02 04:26:55 -08:00 |
|
|
Lianmin Zheng
|
18108abe5d
|
[Minor] Fix code style (#2311)
|
2024-12-02 02:27:36 -08:00 |
|
|
Lianmin Zheng
|
3c79ad35ca
|
[Fix] Fix the padded hash value for image tokens (#2309)
|
2024-12-01 23:36:28 -08:00 |
|
|
Lianmin Zheng
|
5c18a03733
|
Fix logprob for completions (#2301)
|
2024-12-01 05:17:05 -08:00 |
|
|
Lianmin Zheng
|
9449a95431
|
[CI] Balance CI tests (#2293)
|
2024-12-01 01:47:30 -08:00 |
|
|
Lianmin Zheng
|
0303ca918f
|
[CI] Fix missing files in run_suite.py (#2288)
|
2024-11-30 23:53:34 -08:00 |
|
|
Lianmin Zheng
|
4936be8acc
|
Revert "Revert "[FEAT] Support GGUF format"" (#2287)
|
2024-11-30 22:14:48 -08:00 |
|
|
Lianmin Zheng
|
1bfa511b95
|
[CI] Fix ci tests (#2284)
|
2024-11-30 21:12:03 -08:00 |
|
|
Lianmin Zheng
|
f5b5f2bff9
|
Revert "[Fix] fix assertion error for chunked prefill when disabling cache" (#2286)
|
2024-11-30 19:03:42 -08:00 |
|
|
Lianmin Zheng
|
7e4c6dd8da
|
Revert "[FEAT] Support GGUF format" (#2285)
|
2024-11-30 19:03:26 -08:00 |
|
|
Lianmin Zheng
|
0d6a49bd7d
|
[CI] Kill zombie processes (#2280)
|
2024-11-30 00:24:30 -08:00 |
|
|
Lianmin Zheng
|
ccaf1f997c
|
[CI] Print summary on github actions (#2274)
|
2024-11-29 23:48:54 -08:00 |
|
|
|
b53d6cbda3
|
Add new contributors so they can trigger CI automatically (#2269)
Co-authored-by: Qun Yang <qun.yang@intel.com>
Co-authored-by: zhengy001 <zhengy.gator@gmail.com>
Co-authored-by: HandH1998 <1335248067@qq.com>
Co-authored-by: xiaobo <xiaob.chen@outlook.com>
|
2024-11-29 16:37:52 -08:00 |
|
|
Lianmin Zheng
|
94e167ea5a
|
Fix the default chunked prefill size (#2268)
|
2024-11-29 16:03:32 -08:00 |
|
|
Lianmin Zheng
|
afe1e46586
|
[Minor] fix the style for multimodal models (#2257)
|
2024-11-29 04:24:20 -08:00 |
|
|
Lianmin Zheng
|
f50a6cf443
|
Fix hash collision for multi modal models (#2256)
|
2024-11-29 03:15:58 -08:00 |
|
|
Lianmin Zheng
|
fe97a2d40f
|
Simplify tokenizer manager (#2254)
|
2024-11-29 02:18:51 -08:00 |
|
|
Lianmin Zheng
|
4f2ee48ed1
|
Update backend.md (#2251)
|
2024-11-28 23:18:07 -08:00 |
|
|
Lianmin Zheng
|
71ff2728a1
|
Update backend.md (#2250)
|
2024-11-28 23:14:36 -08:00 |
|
|
Lianmin Zheng
|
b2ccf36d4d
|
Fix memory leak during abort (#2238)
|
2024-11-28 02:22:15 -08:00 |
|
|
Lianmin Zheng
|
d4fc1a70e3
|
Crash the server correctly during error (#2231)
|
2024-11-28 00:22:39 -08:00 |
|
|
Lianmin Zheng
|
fb915bd1a2
|
Disable overlap scheduler for multimodal models (#2235)
|
2024-11-27 23:44:33 -08:00 |
|
|
Lianmin Zheng
|
09798b36cd
|
Fix chunked prefill size for bench_offline_throughput (#2234)
|
2024-11-27 23:37:20 -08:00 |
|
|
Lianmin Zheng
|
dd5eba4c88
|
Remove fused_moe_grok (#2223)
|
2024-11-27 14:28:55 -08:00 |
|
|
Lianmin Zheng
|
2a02185c5f
|
Rename DP_RANK to SGLANG_DP_RANK (#2218)
|
2024-11-27 09:36:36 -08:00 |
|
|
Lianmin ZhengandYineng Zhang
|
fed4c6946a
|
Release v0.3.6.post2 (#2214)
Co-authored-by: Yineng Zhang <me@zhyncs.com>
|
2024-11-27 03:35:30 -08:00 |
|
|
Lianmin Zheng
|
fb6e04a0c2
|
Use an env var SGLANG_SET_CPU_AFFINITY to set cpu affinity; turn it off by default (#2222)
|
2024-11-27 02:52:46 -08:00 |
|
|
Lianmin Zheng
|
6997e28f6e
|
Revert "Use an env var SGLANG_SET_CPU_AFFINITY to set cpu affinity; turn it off by default" (#2221)
|
2024-11-27 02:02:01 -08:00 |
|
|
Lianmin Zheng
|
a0e58740a8
|
Use an env var SGLANG_SET_CPU_AFFINITY to set cpu affinity; turn it off by default (#2217)
|
2024-11-27 01:13:41 -08:00 |
|
|
Lianmin Zheng
|
c754652fcd
|
Fix flasky tests (#2212)
|
2024-11-26 23:06:20 -08:00 |
|
|
Lianmin Zheng
|
ba4ee37fa4
|
Update sampler.py to skip the success check (#2197)
|
2024-11-26 00:58:57 -08:00 |
|
|
Lianmin Zheng
|
ac5a0f0488
|
Release v0.3.6.post1 (#2189)
|
2024-11-25 17:31:37 -08:00 |
|
|
Lianmin Zheng
|
ea34350d88
|
Rename double sparsity config file (#2188)
|
2024-11-25 17:12:08 -08:00 |
|
|
Lianmin Zheng
|
1605ae121e
|
[CI] Minor fix for CI (#2187)
|
2024-11-25 16:38:43 -08:00 |
|
|
Lianmin Zheng
|
3c5538f781
|
Update CI threshold (#2186)
|
2024-11-25 15:24:17 -08:00 |
|
|
Lianmin Zheng
|
c4336b2b60
|
Use custom allreduce w/ torch.compile (#2185)
|
2024-11-25 14:55:01 -08:00 |
|
|
Lianmin Zheng
|
254fd130e2
|
[CI] Split test cases in CI for better load balancing (#2180)
|
2024-11-25 04:58:16 -08:00 |
|
|
Lianmin Zheng
|
8e1adb8441
|
Allow overwrite flashinfer use_tensorcore (#2169)
|
2024-11-24 20:58:17 -08:00 |
|
|
Lianmin Zheng
|
8912b7637f
|
Fix docs (#2164)
|
2024-11-24 08:25:56 -08:00 |
|
|
Lianmin Zheng
|
be0124bda0
|
Rename triton_fused_moe -> fused_moe_triton (#2163)
|
2024-11-24 08:12:35 -08:00 |
|
|
Lianmin Zheng
|
fe5d3e818f
|
Balance CI tests (#2162)
|
2024-11-24 07:38:52 -08:00 |
|
|
Lianmin Zheng
|
731146f6cb
|
Fix mixed chunked prefill in overlap mode (#2158)
|
2024-11-24 07:17:37 -08:00 |
|
|
Lianmin Zheng
|
5652c56535
|
Update CI threshold & Improve code style (#2159)
|
2024-11-24 06:29:38 -08:00 |
|
|
Lianmin Zheng
|
c211e7b669
|
Simplify batch update (#2154)
|
2024-11-24 04:47:10 -08:00 |
|
|
Lianmin Zheng
|
751c3a037c
|
Fix dp print message (#2138)
|
2024-11-23 01:22:26 -08:00 |
|
|
Lianmin Zheng
|
a78d8f8db3
|
[CI] Fix test cases (#2137)
|
2024-11-23 01:00:07 -08:00 |
|
|
Lianmin Zheng
|
66d4859acf
|
Revert "Only stream output on tp rank 0" (#2130)
|
2024-11-22 15:46:16 -08:00 |
|
|
Lianmin Zheng
|
e1b63624d7
|
Only stream output on tp rank 0 (#2124)
|
2024-11-22 15:13:44 -08:00 |
|