sglang

Author	SHA1	Message	Date
Lianmin Zheng	bc1154c399	Bump version to 0.2.6 (#779 )	2024-07-27 20:29:33 -07:00
Lianmin Zheng	752e643007	Allow disabling flashinfer sampling kernel (#778 )	2024-07-27 20:18:56 -07:00
Lianmin Zheng	30db99b3d9	Rename prefill_token_logprobs -> input_token_logprobs; decode_token_logprobs -> output_token_logprobs (#776 )	2024-07-27 19:50:34 -07:00
Lianmin Zheng	0a409bd438	Fix return_log_probs with cuda graph (#775 )	2024-07-27 19:15:09 -07:00
Lianmin Zheng	bbc07c4197	Move sampling logits to float32 (#773 )	2024-07-27 17:30:12 -07:00
Lianmin Zheng	a036d41980	Fix max new tokens (#772 )	2024-07-27 17:22:18 -07:00
Lianmin Zheng	f95e661757	Fix max_tokens for OpenAI chat completion API (#766 )	2024-07-27 15:44:27 -07:00
Lianmin Zheng	f64b2a9bc0	Add slack invitation link.	2024-07-27 06:29:15 -07:00
Lianmin Zheng	0736b27020	[Minor] Improve the code style in TokenizerManager (#767 )	2024-07-27 05:05:15 -07:00
Lianmin Zheng	00e4baa728	Update schedule_heuristic.py	2024-07-24 01:22:30 -07:00
Lianmin Zheng	01d66ae2e8	Fix multi-node deadlock (#709 )	2024-07-23 21:53:36 -07:00
Lianmin Zheng	5a4ef2b5c8	update readme	2024-07-21 02:58:57 -07:00
Lianmin Zheng	33ee97b0bf	Allow disabling streaming in bench (#687 )	2024-07-21 01:12:34 -07:00
Lianmin Zheng	77e592e8e0	support non-streaming benchmark (#682 )	2024-07-20 18:36:42 -07:00
Lianmin Zheng	490a1f39dd	Fix cuda graph with flashinfer (#675 )	2024-07-20 02:43:55 -07:00
Lianmin Zheng	9592a1f3bd	Fix random dataset (#671 )	2024-07-20 01:57:43 -07:00
Lianmin Zheng	35759efa91	Support random dataset in bench_serving.py (#669 )	2024-07-20 01:06:43 -07:00
Lianmin Zheng	e1792cca24	Remove cached triton launcher (#656 )	2024-07-18 23:28:40 -07:00
Lianmin Zheng	41d1f67704	Fix flush cache (#627 )	2024-07-15 20:44:04 -07:00
Lianmin Zheng	5d264a90ac	Bump version to 0.1.20 (#618 )	2024-07-13 17:27:55 -07:00
Lianmin Zheng	0feca02dd9	Improve benchmark scripts (#615 )	2024-07-13 15:59:04 -07:00
Lianmin Zheng	65c6577696	Improve benchmark scripts & fix llava (#613 )	2024-07-13 15:00:26 -07:00
Lianmin Zheng	665815969a	Enable cuda graph by default (#612 )	2024-07-13 05:29:46 -07:00
Lianmin Zheng	396a69240f	Cleanup attention backend: flashinfer and triton (#611 )	2024-07-12 18:21:11 -07:00
Lianmin Zheng	af4e7910e7	Clean up the usage of flashinfer (#610 )	2024-07-12 13:00:03 -07:00
Lianmin Zheng	519e20cfda	Code clean up: Remove deprecated prefill move InputMetadata to infer_batch.py (#609 )	2024-07-12 12:28:09 -07:00
Lianmin Zheng	d9a6902986	Fix bench latency (#607 )	2024-07-11 14:37:01 -07:00
Lianmin Zheng	ad872feb14	bump version to 0.1.19	2024-07-09 02:23:14 -07:00
Lianmin Zheng	da2e5d6546	Fix the default argument of OpenAI Chat completion (#605 )	2024-07-09 02:04:43 -07:00
Lianmin Zheng	ce62dc73f0	Update model_support.md	2024-07-09 01:32:46 -07:00
Lianmin Zheng	d737da5f17	Update README.md	2024-07-04 00:56:58 -07:00
Lianmin Zheng	dc8cef1d0c	Update README.md	2024-07-04 00:05:40 -07:00
Lianmin Zheng	63fbef9876	fix flashinfer & http log level	2024-07-03 23:19:33 -07:00
Lianmin Zheng	c7709d3abe	Update install commands (#583 )	2024-07-03 02:10:59 -07:00
Lianmin Zheng	26294b2f3d	Update README.md	2024-07-01 09:54:08 -07:00
Lianmin Zheng	badf3fa020	Expose dtype argument (#569 )	2024-06-27 23:30:39 -07:00
Lianmin Zheng	945aa9beb2	Update readme (#568 )	2024-06-27 11:37:49 -07:00
Lianmin Zheng	2e6e62e156	Increase the number of thread limitation for tp worker managers. (#567 )	2024-06-26 09:33:45 -07:00
Lianmin Zheng	a385ee27bd	Warmup cublas (#566 )	2024-06-25 12:46:00 -07:00
Lianmin Zheng	eb1ae6ae0c	Add sglang.bench_latency for offline benchmark (#564 )	2024-06-25 03:38:04 -07:00
Lianmin Zheng	2187f36237	Add a new arguments log_level_http to control the HTTP logging (#563 )	2024-06-25 01:16:20 -07:00
Lianmin Zheng	9465b668b9	Allow running with vllm==0.4.3 (#561 )	2024-06-24 15:24:21 -07:00
Lianmin Zheng	1fa15099d8	Add LlamaForClassification (#559 )	2024-06-22 00:49:31 -07:00
Lianmin Zheng	303ef8883e	Clean up logits processor (#558 )	2024-06-22 00:25:24 -07:00
Lianmin Zheng	e94e60d6fb	make flashinfer workspace larger	2024-06-21 17:32:36 -07:00
Lianmin Zheng	d2f8bfb2e1	Follow-up fixes for flashinfer 0.0.5 (#556 )	2024-06-20 23:19:52 -07:00
Lianmin Zheng	b7e2f800ac	Update flashinfer to 0.0.5 (#554 )	2024-06-20 20:29:06 -07:00
Lianmin Zheng	53a7ebd89a	Update fused_moe (#553 )	2024-06-17 09:47:58 -07:00
Lianmin Zheng	94aead9e8d	Fix dependency (#538 )	2024-06-12 13:17:35 -07:00
Lianmin Zheng	f6dbd24043	Improve doc strings (#518 )	2024-06-08 02:39:32 -07:00
Lianmin Zheng	e8a2327d52	Update version to 0.1.17 (#515 )	2024-06-07 19:49:18 -07:00
Lianmin Zheng	91f93f141f	Crash the server when error or OOM happens (#514 )	2024-06-07 19:22:34 -07:00
Lianmin Zheng	c0ae70c8ed	Improve logging & fix litellm dependency. (#512 )	2024-06-07 13:10:32 -07:00
Lianmin Zheng Qubitium ZX	bf3e271fe0	Update vllm to v0.4.3 (#511 ) Co-authored-by: Qubitium <417764+Qubitium@users.noreply.github.com> Co-authored-by: ZX <zx@lbx.dev>	2024-06-07 12:11:31 -07:00
Lianmin Zheng	3bc01ac137	[Minor] improve code style	2024-06-03 18:11:34 -07:00
Lianmin Zheng	9f009261f2	Improve docs	2024-06-01 17:46:08 -05:00
Lianmin Zheng	159cc741e4	Make the server random by default (#493 )	2024-05-31 23:33:34 -07:00
Lianmin Zheng	80a33ce8b0	Do not set the default value of global random seed (#488 )	2024-05-29 18:41:18 -04:00
Lianmin Zheng	1a57e41679	do not launch workers in parallel	2024-05-27 23:00:16 -07:00
Lianmin Zheng	adc974268a	Update docs (#486 )	2024-05-27 22:51:05 -07:00
Lianmin Zheng	565d727409	improve logging & fix vllm version	2024-05-27 15:04:23 -07:00
Lianmin Zheng	09de730dee	Improve benchmark scripts & add more models (#484 )	2024-05-27 14:13:26 -07:00
Lianmin Zheng	55c1643627	Improve benchmark scripts & rename some scripts (#477 )	2024-05-26 12:51:45 -07:00
Lianmin Zheng	2cea6146d8	Improve logging & add logit cap (#471 )	2024-05-24 03:48:53 -07:00
Lianmin Zheng	0fafc5606b	port fp8 mixtral (#460 )	2024-05-21 11:46:35 -07:00
Lianmin Zheng	19d2135cb8	Use model loader from vllm (#459 )	2024-05-21 09:13:37 -07:00
Lianmin Zheng	ced77c6626	Rename api_num_spec_tokens -> num_api_spec_tokens (#458 )	2024-05-20 18:44:23 -07:00
Lianmin Zheng	8dbdc018a3	Abort disconnected requests (#457 )	2024-05-20 18:41:21 -07:00
Lianmin Zheng	8210ec60f4	Improve error handling & abort disconnected requests (#449 )	2024-05-17 05:49:31 -07:00
Lianmin Zheng	c05956e534	Simplify port allocation (#447 )	2024-05-16 18:07:30 -07:00
Lianmin Zheng	e0ae5d42ec	Update version to 0.1.16 (#438 )	2024-05-13 17:29:17 -07:00
Lianmin Zheng	32de16ce2f	Fix streaming (#437 )	2024-05-13 17:26:18 -07:00
Lianmin Zheng	5dc55a5f02	Handle truncation errors (#436 )	2024-05-13 15:56:00 -07:00
Lianmin Zheng	4231a42fa8	Fix import of global_config	2024-05-13 12:11:55 -07:00
Lianmin Zheng	455c9ccc4a	Update readme (#434 )	2024-05-13 00:17:02 -07:00
Lianmin Zheng	562b8857d8	Improve error handling (#433 )	2024-05-12 20:49:04 -07:00
Lianmin Zheng	6e09cf6a15	Misc fixes (#432 )	2024-05-12 15:05:40 -07:00
Lianmin Zheng	72bb344388	Update version to 0.1.15 (#431 )	2024-05-12 14:22:33 -07:00
Lianmin Zheng	2d580e7a89	Fix flashinfer (#430 )	2024-05-12 08:18:53 -07:00
Lianmin Zheng	3fc97f6709	Move openai api server into a separate file (#429 )	2024-05-12 06:41:32 -07:00
Lianmin Zheng	abc548c707	Minor fix for the import path (#428 )	2024-05-12 05:10:35 -07:00
Lianmin Zheng	aee4f523cf	Fix logit processor bugs (#427 )	2024-05-12 04:54:07 -07:00
Lianmin Zheng	7023f413c6	Clean up (#422 )	2024-05-11 20:55:00 -07:00
Lianmin Zheng	09deb20dee	Optimize the memory usage of logits processor (#420 )	2024-05-11 16:56:42 -07:00
Lianmin Zheng	a511a2d089	restrict vllm version	2024-05-09 15:49:29 -07:00
Lianmin Zheng	ba67101f99	Fix chatml template (#406 )	2024-04-30 15:53:39 -07:00
Lianmin Zheng	65501a9cf1	Fix commandr import; format code	2024-04-16 18:10:12 +00:00
Lianmin Zheng	51104cd405	Update version to v0.1.14 (#324 )	2024-03-22 13:42:22 -07:00
Lianmin Zheng	e2b2f0a213	Support oai in benchmark/mmlu (#323 )	2024-03-22 13:37:57 -07:00
Lianmin Zheng	4aa5dd2c5f	Update version to v0.1.13 (#280 )	2024-03-11 05:49:27 -07:00
Lianmin Zheng	13662fd533	Fix RuntimeEndpoint (#279 )	2024-03-11 05:24:24 -07:00
Lianmin Zheng	faba293a0d	Improve gemma and documentations (#278 )	2024-03-11 04:43:39 -07:00
Lianmin Zheng	a833de05d3	Add logo (#275 )	2024-03-10 18:51:47 -07:00
Lianmin Zheng	6493256b7d	improve print	2024-02-12 12:43:48 +00:00
Lianmin Zheng	06008bc295	Fix server launch for jupyter notebook (#186 )	2024-02-12 04:43:14 -08:00
Lianmin Zheng	bb824da41a	Add Together and AzureOpenAI examples (#184 )	2024-02-12 01:06:38 -08:00
Lianmin Zheng	624b21e742	Update version to 0.1.12 (#178 )	2024-02-11 06:43:45 -08:00
Lianmin Zheng	c51020cf0c	Fix the chat template for llava-v1.6-34b & format code (#177 )	2024-02-11 05:50:13 -08:00
Lianmin Zheng	23f05005fd	Format code & move functions (#155 )	2024-02-06 13:27:46 -08:00
Lianmin Zheng	ee1df26a77	Update README.md	2024-02-06 11:35:42 -08:00