 Ying Sheng
|
21ba3a88a1
|
Remove useless variables in infer_batch.py (#651)
|
2024-07-18 05:31:44 -07:00 |
|
|
Ying Sheng
|
476584cb6e
|
Increase the capacity of the memory pool (#643)
|
2024-07-17 15:44:41 -07:00 |
|
|
Ying Sheng
|
a470e60c97
|
clean up step function (#635)
|
2024-07-16 20:15:24 -07:00 |
|
|
Ying Sheng
|
5f90e0769c
|
Update README.md
|
2024-07-16 19:18:54 -07:00 |
|
|
Ying Sheng
|
0aa189f150
|
Disable NCCL_NVLS by default (#631)
|
2024-07-16 09:05:10 -07:00 |
|
|
Ying Sheng
|
f6b29f6920
|
Update docker file (#629)
|
2024-07-16 01:12:37 -07:00 |
|
|
Ying Sheng
|
56f5fc4ab5
|
Bump version to 0.1.21 (#626)
|
2024-07-15 13:10:53 -07:00 |
|
|
Ying ShengandMingyi
|
6a2941f4d0
|
Improve tensor parallel performance (#625)
Co-authored-by: Mingyi <wisclmy0611@gmail.com>
|
2024-07-15 07:10:51 -07:00 |
|
|
Ying Sheng
|
bae9541e4c
|
Update benchmark script (#621)
|
2024-07-14 21:38:53 +00:00 |
|
|
Ying Sheng
|
5949b1ca0e
|
Fix memory pool index error (#616)
|
2024-07-13 16:45:11 -07:00 |
|
|
Ying Sheng
|
dc1b8bcfaa
|
Format (#593)
|
2024-07-05 10:06:17 -07:00 |
|
|
Ying Sheng
|
5a57b8addd
|
Add Gemma2 (#592)
|
2024-07-05 09:48:54 -07:00 |
|
|
Ying ShengandYing Sheng
|
ac11388756
|
Add docker file (#588)
Co-authored-by: Ying Sheng <ying.sheng@databricks.com>
|
2024-07-04 00:53:49 -07:00 |
|
|
Ying Sheng
|
2f11936f95
|
bump version to 0.1.18
|
2024-07-04 06:27:29 +00:00 |
|
|
Ying Sheng
|
2a754e57b0
|
2x performance improvement for large prefill & Fix workspace conflicts (#579)
|
2024-07-03 16:14:57 -07:00 |
|
|
Ying Sheng
|
9380f50ff9
|
Turn on flashinfer by default (#578)
|
2024-07-02 02:25:07 -07:00 |
|
|
Ying Sheng
|
75b31a2a88
|
Update run_batch interface and max_prefill_tokens (#574)
|
2024-06-30 18:26:04 -07:00 |
|
|
Ying Sheng
|
9ce89bc14b
|
Update benchmark script (#571)
|
2024-06-28 00:44:22 -07:00 |
|
|
Ying ShengandLianmin Zheng
|
09593e9bc9
|
Multi-node Tensor Parallelism (#550)
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
|
2024-06-17 20:41:24 -07:00 |
|
|
Ying Sheng
|
fb9296f0ed
|
Higher priority for user input of max_prefill_tokens & format (#540)
|
2024-06-12 21:48:40 -07:00 |
|
|
Ying Sheng
|
1374334d38
|
Fix dependency & crash issues (#539)
|
2024-06-12 21:23:19 -07:00 |
|
|
Ying Sheng
|
83525a1df2
|
Revert "Make the server random by default" (#492)
|
2024-05-31 12:00:21 -07:00 |
|
|
|
0463f7fb52
|
Support data parallelism (static) (#480)
Co-authored-by: Ying Sheng <ying.sheng@databricks.com>
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
|
2024-05-27 21:24:10 -07:00 |
|
|
Ying Sheng
|
947bda73fe
|
Add benchmark scripts (#476)
|
2024-05-26 12:09:03 -07:00 |
|
|
Ying Sheng
|
3e684be7a3
|
Fix openai speculative execution (#456)
|
2024-05-20 17:01:13 -07:00 |
|
|
Ying Sheng
|
5be9eb8a8c
|
Add PUT for generate api (#448)
|
2024-05-17 02:35:15 -07:00 |
|
|
Ying Sheng
|
ca4f1ab89c
|
Update model support in readme (#370)
|
2024-04-17 00:16:32 -07:00 |
|
|
Ying Sheng
|
b0890631a0
|
fix gemma import error
|
2024-04-01 07:36:52 +00:00 |
|
|
Ying Sheng
|
a6aa46dd3f
|
minor
|
2024-02-08 04:35:25 +00:00 |
|
|
Ying Sheng
|
79e6b84bec
|
Update README.md
|
2024-02-06 23:14:59 -08:00 |
|
|
Ying Sheng
|
cb8e1982f8
|
Update README.md
|
2024-02-06 18:45:38 -08:00 |
|
|
Ying Sheng
|
45d6592d40
|
Fix no-cache mode (#136)
|
2024-02-03 04:59:06 -08:00 |
|
|
Ying Sheng
|
f6bfe3aaff
|
Release 0.1.11 (#134)
|
2024-02-03 02:50:13 -08:00 |
|
|
Ying Sheng
|
e095b16236
|
Add max_prefill_num_token into server arguments (#133)
|
2024-02-03 02:35:54 -08:00 |
|
|
Ying Sheng
|
67be11c790
|
fix bug of race condition in copy()
|
2024-02-03 01:38:00 -08:00 |
|
|
Ying Sheng
|
489796c7ea
|
minor performance fix
|
2024-01-24 10:45:44 +00:00 |
|
|
Ying Sheng
|
3f5c2f4c4a
|
Add an async example (#37)
|
2024-01-21 15:17:30 -08:00 |
|
|
Ying Sheng
|
e8f2b155fe
|
Update README.md
|
2024-01-21 02:45:58 -08:00 |
|
|
Ying Sheng
|
71d30d6ddc
|
Update README.md
|
2024-01-17 09:49:53 -08:00 |
|
|
Ying ShengandLianmin Zheng
|
ffe4aaee1d
|
Fix for T4 GPUs (#16)
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
|
2024-01-16 15:49:03 -08:00 |
|
|
Ying Sheng
|
f6d40df0ee
|
Initial commit
|
2023-10-09 15:41:15 -07:00 |
|