 Lianmin Zheng
|
68be2f6d3b
|
[CI] Include triton backend and online serving benchmark into CI (#1408)
|
2024-09-12 21:36:41 -07:00 |
|
|
Lianmin Zheng
|
b912de11b0
|
Make stop reason a dict instead of str (#1407)
|
2024-09-12 20:47:31 -07:00 |
|
|
Lianmin Zheng
|
3efa798116
|
Support cuda graph in the triton attention backend (#1401)
|
2024-09-12 00:36:55 -07:00 |
|
|
Lianmin Zheng
|
fec185ce0c
|
Refactor attention backend (#1381)
|
2024-09-11 11:44:26 -07:00 |
|
|
Lianmin Zheng
|
c03cece42f
|
Improve error reporting during server launch (#1390)
|
2024-09-11 04:50:04 -07:00 |
|
|
Lianmin Zheng
|
15c75e4146
|
[Fix] Fix --disable-flashinfer (#1389)
|
2024-09-11 04:36:21 -07:00 |
|
|
Lianmin Zheng
|
46094e0c1b
|
Deprecate --disable-flashinfer and introduce --attention-backend (#1380)
|
2024-09-10 17:11:16 -07:00 |
|
|
Lianmin Zheng
|
3a6e8b6d78
|
[Minor] move triton attention kernels into a separate folder (#1379)
|
2024-09-10 15:15:08 -07:00 |
|
|
Lianmin Zheng
|
6c7cb90365
|
[Minor] improve kill scripts and torchao import (#1375)
|
2024-09-11 04:27:03 +10:00 |
|
|
Lianmin Zheng
|
8d1095dbf0
|
[Docs] Improve documentations (#1368)
|
2024-09-09 20:48:28 -07:00 |
|
|
Lianmin Zheng
|
e4d68afcf0
|
[Minor] Many cleanup (#1357)
|
2024-09-09 04:14:11 -07:00 |
|
|
Lianmin Zheng
|
eda7c09048
|
Remove useless fields in global_config.py (#1328)
|
2024-09-04 05:37:32 -07:00 |
|
|
Lianmin Zheng
|
843e63d809
|
Fix the flaky test test_moe_eval_accuracy_large.py (#1326)
|
2024-09-04 04:15:11 -07:00 |
|
|
Lianmin Zheng
|
1e495e0847
|
[Fix] Fix select by ensuring each request has at least one token (#1318)
|
2024-09-03 06:31:45 -07:00 |
|
|
Lianmin Zheng
|
12cb115d38
|
Fix llama2 weight loader (#1317)
|
2024-09-03 05:32:14 -07:00 |
|
|
Lianmin Zheng
|
c500f96bb1
|
Update README.md for llava-onevision instructions (#1313)
|
2024-09-03 01:43:08 -07:00 |
|
|
Lianmin Zheng
|
f64eae3a29
|
[Fix] Reduce memory usage for loading llava model & Remove EntryClassRemapping (#1308)
|
2024-09-02 21:44:45 -07:00 |
|
|
Lianmin Zheng
|
9999442756
|
Release v0.2.15 (#1295)
|
2024-09-01 22:22:38 -07:00 |
|
|
Lianmin Zheng
|
58fa607622
|
Fix the flaky tests in test_moe_eval_accuracy_large.py (#1293)
|
2024-09-01 12:20:46 -07:00 |
|
|
Lianmin Zheng
|
761b2cebd6
|
[CI] merge all ci tests into one file (#1289)
|
2024-09-01 02:36:56 -07:00 |
|
|
Lianmin Zheng
|
1b5d56f7f8
|
[CI] Add more multi-gpu tests (#1280)
|
2024-09-01 00:27:25 -07:00 |
|
|
Lianmin Zheng
|
79ece2c51f
|
Report median instead of mean in bench_latency.py (#1269)
|
2024-08-30 06:05:01 -07:00 |
|
|
Lianmin Zheng
|
0a97d7962d
|
[Fix] Fix OOM in llava base class (#1249)
|
2024-08-28 08:45:49 -07:00 |
|
|
Lianmin Zheng
|
bf53bf5142
|
[Fix] Fix llava on multi images (#1247)
|
2024-08-28 06:33:05 -07:00 |
|
|
Lianmin Zheng
|
6c49831394
|
Add sglang.bench_latency to CI (#1243)
|
2024-08-28 21:20:54 +10:00 |
|
|
Lianmin Zheng
|
184ae1c683
|
Update README.md (#1239)
|
2024-08-28 02:15:52 -07:00 |
|
|
Lianmin Zheng
|
6cc38b2bf3
|
[Minor] Add more type annotations (#1237)
|
2024-08-28 00:54:26 -07:00 |
|
|
Lianmin Zheng
|
d3efcb3930
|
Update workflow files (#1214)
|
2024-08-25 17:45:35 -07:00 |
|
|
Lianmin Zheng
|
61bb223e0f
|
Update CI runner docs (#1213)
|
2024-08-25 17:31:52 -07:00 |
|
|
Lianmin Zheng
|
15f1a49d2d
|
Update CI workflows (#1210)
|
2024-08-25 16:43:07 -07:00 |
|
|
Lianmin Zheng
|
902278008a
|
[Minor] Improve the function organization in TokenizerManager & improve loggers (#1208)
|
2024-08-25 14:46:34 -07:00 |
|
|
Lianmin Zheng
|
bc4c7a3545
|
Relax the assert in moe throughput test to fix the flaky CI (#1207)
|
2024-08-25 10:27:02 -07:00 |
|
|
Lianmin Zheng
|
b20daf982a
|
Update README.md (#1198)
|
2024-08-24 14:50:05 -07:00 |
|
|
Lianmin Zheng
|
f6af3a6561
|
Cleanup readme, llava examples, usage examples and nccl init (#1194)
|
2024-08-24 08:02:23 -07:00 |
|
|
Lianmin Zheng
|
5623826f73
|
[Minor] Improve logging and rename the health check endpoint name (#1180)
|
2024-08-21 19:24:36 -07:00 |
|
|
Lianmin Zheng
|
bea2bb9eea
|
Improve multi-node stability (#1171)
|
2024-08-20 22:35:05 -07:00 |
|
|
Lianmin Zheng
|
a8ae640328
|
Improve docs and warnings (#1164)
|
2024-08-20 08:31:29 -07:00 |
|
|
Lianmin Zheng
|
3c1f5a9220
|
Fix duplicated imports in hf_transformers_utils.py (#1141)
|
2024-08-17 18:03:00 -07:00 |
|
|
Lianmin Zheng
|
57d0bd91ec
|
Improve benchmark (#1140)
|
2024-08-17 17:43:23 -07:00 |
|
|
Lianmin Zheng
|
cdc8d60752
|
Improve the code style: more comments and remove useless packages (#1139)
|
2024-08-17 14:37:52 -07:00 |
|
|
Lianmin Zheng
|
5a261bd055
|
Fix the deadlock in multi-node tp (#1122)
|
2024-08-16 01:39:24 -07:00 |
|
|
Lianmin Zheng
|
87a0db82b8
|
update hyperparameter guide (#1114)
|
2024-08-15 10:54:24 -07:00 |
|
|
Lianmin Zheng
|
0cb099e20a
|
set CUDA_DEVICE_MAX_CONNECTIONS=1 (#1113)
|
2024-08-16 03:47:39 +10:00 |
|
|
Lianmin Zheng
|
e86b1ccbf0
|
Enable chunked prefill by default (#1040)
|
2024-08-14 21:56:20 -07:00 |
|
|
Lianmin Zheng
|
326df4bab2
|
Use a single workspace for flashinfer (#1077)
|
2024-08-14 19:25:37 -07:00 |
|
|
Lianmin Zheng
|
a59636bb5e
|
Update grok 1 model (#1095)
|
2024-08-14 04:40:44 -07:00 |
|
|
Lianmin Zheng
|
8f790ac100
|
Fix a bug in cuda graph runner (#1094)
|
2024-08-14 03:25:38 -07:00 |
|
|
Lianmin Zheng
|
ad3e4f1619
|
Update the mixtral to use the better FusedMoE layer (#1081)
|
2024-08-13 15:44:25 -07:00 |
|
|
Lianmin Zheng
|
c877292cc1
|
Re-organize CI tests (#1052)
|
2024-08-12 03:39:01 -07:00 |
|
|
Lianmin Zheng
|
0c1c72a0b4
|
Fix accuracy test (#1051)
|
2024-08-12 19:48:40 +10:00 |
|
|
Lianmin Zheng
|
41598e0d8e
|
Add longer accuracy test on CI (#1049)
|
2024-08-12 09:21:38 +00:00 |
|
|
Lianmin Zheng
|
fb1f28cbbb
|
Clean up the comments and names under python/sglang/srt/layers (#1047)
|
2024-08-12 05:54:37 +00:00 |
|
|
Lianmin Zheng
|
14b6493087
|
Delete the useless test/srt/test_throughput.py (#1045)
|
2024-08-11 21:31:52 -07:00 |
|
|
Lianmin Zheng
|
8207637029
|
Improve end-to-end throughput test and its coverage (#1039)
|
2024-08-11 18:27:33 -07:00 |
|
|
Lianmin Zheng
|
d84c5e70f7
|
Test the case when max_new_tokens is very large (#1038)
|
2024-08-11 16:41:03 -07:00 |
|
|
Lianmin Zheng
|
d785412077
|
Fix the case when max_new_tokens is too large (#1025)
|
2024-08-11 15:20:18 -07:00 |
|
|
Lianmin Zheng
|
4080e82244
|
Fix the case where r.prefix_indices is None (#1031)
|
2024-08-11 04:53:51 -07:00 |
|
|
Lianmin Zheng
|
9dae407812
|
Improve type annotation (#1029)
|
2024-08-11 02:44:59 -07:00 |
|
|
Lianmin Zheng
|
a97df79124
|
Clean up readme and arguments of chunked prefill (#1022)
|
2024-08-11 01:18:52 -07:00 |
|
|
Lianmin Zheng
|
54fb1c80c0
|
Clean up unit tests (#1020)
|
2024-08-10 15:09:03 -07:00 |
|
|
Lianmin Zheng
|
bc1154c399
|
Bump version to 0.2.6 (#779)
|
2024-07-27 20:29:33 -07:00 |
|
|
Lianmin Zheng
|
752e643007
|
Allow disabling flashinfer sampling kernel (#778)
|
2024-07-27 20:18:56 -07:00 |
|
|
Lianmin Zheng
|
30db99b3d9
|
Rename prefill_token_logprobs -> input_token_logprobs; decode_token_logprobs -> output_token_logprobs (#776)
|
2024-07-27 19:50:34 -07:00 |
|
|
Lianmin Zheng
|
0a409bd438
|
Fix return_log_probs with cuda graph (#775)
|
2024-07-27 19:15:09 -07:00 |
|
|
Lianmin Zheng
|
bbc07c4197
|
Move sampling logits to float32 (#773)
|
2024-07-27 17:30:12 -07:00 |
|
|
Lianmin Zheng
|
a036d41980
|
Fix max new tokens (#772)
|
2024-07-27 17:22:18 -07:00 |
|
|
Lianmin Zheng
|
f95e661757
|
Fix max_tokens for OpenAI chat completion API (#766)
|
2024-07-27 15:44:27 -07:00 |
|
|
Lianmin Zheng
|
f64b2a9bc0
|
Add slack invitation link.
|
2024-07-27 06:29:15 -07:00 |
|
|
Lianmin Zheng
|
0736b27020
|
[Minor] Improve the code style in TokenizerManager (#767)
|
2024-07-27 05:05:15 -07:00 |
|
|
Lianmin Zheng
|
00e4baa728
|
Update schedule_heuristic.py
|
2024-07-24 01:22:30 -07:00 |
|
|
Lianmin Zheng
|
01d66ae2e8
|
Fix multi-node deadlock (#709)
|
2024-07-23 21:53:36 -07:00 |
|
|
Lianmin Zheng
|
5a4ef2b5c8
|
update readme
|
2024-07-21 02:58:57 -07:00 |
|
|
Lianmin Zheng
|
33ee97b0bf
|
Allow disabling streaming in bench (#687)
|
2024-07-21 01:12:34 -07:00 |
|
|
Lianmin Zheng
|
77e592e8e0
|
support non-streaming benchmark (#682)
|
2024-07-20 18:36:42 -07:00 |
|
|
Lianmin Zheng
|
490a1f39dd
|
Fix cuda graph with flashinfer (#675)
|
2024-07-20 02:43:55 -07:00 |
|
|
Lianmin Zheng
|
9592a1f3bd
|
Fix random dataset (#671)
|
2024-07-20 01:57:43 -07:00 |
|
|
Lianmin Zheng
|
35759efa91
|
Support random dataset in bench_serving.py (#669)
|
2024-07-20 01:06:43 -07:00 |
|
|
Lianmin Zheng
|
e1792cca24
|
Remove cached triton launcher (#656)
|
2024-07-18 23:28:40 -07:00 |
|
|
Lianmin Zheng
|
41d1f67704
|
Fix flush cache (#627)
|
2024-07-15 20:44:04 -07:00 |
|
|
Lianmin Zheng
|
5d264a90ac
|
Bump version to 0.1.20 (#618)
|
2024-07-13 17:27:55 -07:00 |
|
|
Lianmin Zheng
|
0feca02dd9
|
Improve benchmark scripts (#615)
|
2024-07-13 15:59:04 -07:00 |
|
|
Lianmin Zheng
|
65c6577696
|
Improve benchmark scripts & fix llava (#613)
|
2024-07-13 15:00:26 -07:00 |
|
|
Lianmin Zheng
|
665815969a
|
Enable cuda graph by default (#612)
|
2024-07-13 05:29:46 -07:00 |
|
|
Lianmin Zheng
|
396a69240f
|
Cleanup attention backend: flashinfer and triton (#611)
|
2024-07-12 18:21:11 -07:00 |
|
|
Lianmin Zheng
|
af4e7910e7
|
Clean up the usage of flashinfer (#610)
|
2024-07-12 13:00:03 -07:00 |
|
|
Lianmin Zheng
|
519e20cfda
|
Code clean up: Remove deprecated prefill move InputMetadata to infer_batch.py (#609)
|
2024-07-12 12:28:09 -07:00 |
|
|
Lianmin Zheng
|
d9a6902986
|
Fix bench latency (#607)
|
2024-07-11 14:37:01 -07:00 |
|
|
Lianmin Zheng
|
ad872feb14
|
bump version to 0.1.19
|
2024-07-09 02:23:14 -07:00 |
|
|
Lianmin Zheng
|
da2e5d6546
|
Fix the default argument of OpenAI Chat completion (#605)
|
2024-07-09 02:04:43 -07:00 |
|
|
Lianmin Zheng
|
ce62dc73f0
|
Update model_support.md
|
2024-07-09 01:32:46 -07:00 |
|
|
Lianmin Zheng
|
d737da5f17
|
Update README.md
|
2024-07-04 00:56:58 -07:00 |
|
|
Lianmin Zheng
|
dc8cef1d0c
|
Update README.md
|
2024-07-04 00:05:40 -07:00 |
|
|
Lianmin Zheng
|
63fbef9876
|
fix flashinfer & http log level
|
2024-07-03 23:19:33 -07:00 |
|
|
Lianmin Zheng
|
c7709d3abe
|
Update install commands (#583)
|
2024-07-03 02:10:59 -07:00 |
|
|
Lianmin Zheng
|
26294b2f3d
|
Update README.md
|
2024-07-01 09:54:08 -07:00 |
|
|
Lianmin Zheng
|
badf3fa020
|
Expose dtype argument (#569)
|
2024-06-27 23:30:39 -07:00 |
|
|
Lianmin Zheng
|
945aa9beb2
|
Update readme (#568)
|
2024-06-27 11:37:49 -07:00 |
|
|
Lianmin Zheng
|
2e6e62e156
|
Increase the number of thread limitation for tp worker managers. (#567)
|
2024-06-26 09:33:45 -07:00 |
|
|
Lianmin Zheng
|
a385ee27bd
|
Warmup cublas (#566)
|
2024-06-25 12:46:00 -07:00 |
|
|
Lianmin Zheng
|
eb1ae6ae0c
|
Add sglang.bench_latency for offline benchmark (#564)
|
2024-06-25 03:38:04 -07:00 |
|