Commit Graph

Select branches

Hide Pull Requests

dsv41-pd

dsv41-pd-visioncp

main

Mono Color

• b88ea90d4a Fix bugs of logprobs_nums (#1548) Liangsheng Yin 2024-09-30 17:09:54 -07:00
• 99ec439da4 Organize Attention Backends (#1547) Liangsheng Yin 2024-09-30 15:54:18 -07:00
• 0f4fb19bc8 [Fix, LoRA] fix LoRA with updates in main (#1545) Ying Sheng 2024-09-30 10:06:08 -07:00
• 63ba2f8d7b Clean up batch data structures: Introducing ModelWorkerBatch (#1544) Lianmin Zheng 2024-09-30 06:41:49 -07:00
• 36d5acfca5 Rename InputMetadata -> ForwardBatch (#1543) Lianmin Zheng 2024-09-30 02:41:11 -07:00
• 3f0fe08d37 Let ModelRunner take InputMetadata as input, instead of ScheduleBatch (#1541) Lianmin Zheng 2024-09-29 20:28:45 -07:00
• 55b974f96f Process image in parallel (#1539) Liangsheng Yin 2024-09-29 18:52:43 -07:00
• f86c1e611f Move scheduler code from tp_worker.py to scheduler.py (#1538) Lianmin Zheng 2024-09-29 17:42:45 -07:00
• acaffd233f [Fix] fix ipv6 url when warm up model (#1537) Xinyu Yang 2024-09-30 02:02:40 +08:00
• 048685430d Improve process creation (#1534) Lianmin Zheng 2024-09-29 02:36:12 -07:00
• fd9ad817ec Organize image inputs (#1531) Liangsheng Yin 2024-09-28 23:28:55 -07:00
• e165a9fc1b Make detokenizer_manager.py not asyncio (#1532) Lianmin Zheng 2024-09-28 19:33:09 -07:00
• 4e4459b91f Multiple minor fixes (#1530) Lianmin Zheng 2024-09-28 14:43:35 -07:00
• 065bb94753 Fix RuntimeEndpoint.select method (#1495) Jeffrey Fong 2024-09-29 05:04:06 +08:00
• f42e9bfb52 [bugfix] Add modelscope package to avoid docker image without modelscope (#1520) Kylin 2024-09-29 03:43:22 +08:00
• 840c5dbcb3 [FIX] Catch syntax error of Regex Guide to avoid crash (#1521) Ninglin Du 2024-09-29 03:42:06 +08:00
• 63e845d0bb Add float8 dynamic quant to torchao_utils (#1528) Jerry Zhang 2024-09-28 12:27:54 -07:00
• 9aa6553d2a [Feature] Support reward model LxzGordon/URM-LLaMa-3.1-8B (#1525) Ying Sheng 2024-09-27 23:32:11 -07:00
• b1e330bcb0 [Event] Update meeting link (#1529) Ying Sheng 2024-09-27 13:30:04 -07:00
• 4353acb469 minor: fix config (#1524) Liangsheng Yin 2024-09-27 01:49:16 -07:00
• 9ae1db0bdc [Fix] Ignore import error (#1513) Lianmin Zheng 2024-09-25 11:32:21 -07:00
• 37c5899fc2 Release v0.3.2 (#1512) Ying Sheng 2024-09-24 23:17:09 -07:00
• f39a0197fd Revert "kernel: use tensor cores for flashinfer gqa kernels" (#1511) Ying Sheng 2024-09-24 22:50:31 -07:00
• 3c93187caf Add support for tie_word_embeddings when loading weights + support for SmolLM (#1508) TianyiQ 2024-09-24 21:50:20 -07:00
• fb2d0680e0 [Fix] Fix clean_up_tokenization_spaces in tokenizer (#1510) Lianmin Zheng 2024-09-24 21:37:33 -07:00
• 067d8e16fc Simplify bench_latency.py (#1503) Lianmin Zheng 2024-09-24 17:42:07 -07:00
• e6692bf4a5 debug radixcache stack_overflow (#1499) luzengxiangcn 2024-09-24 19:58:01 +08:00
• 28b4d8e144 Update test_srt_backend.py (#1502) Lianmin Zheng 2024-09-24 03:17:10 -07:00
• bc068e9618 [CI] Move AMD test to a separate file (#1500) Lianmin Zheng 2024-09-24 02:06:28 -07:00
• 8d4ed42ad5 MoE torch compile (#1497) Ke Bao 2024-09-24 16:46:59 +08:00
• 2854a5ea9f Fix the overhead due to penalizer in bench_latency (#1496) Lianmin Zheng 2024-09-23 07:38:14 -07:00
• 42a2d82ba7 minor: add mla fp8 test (#1494) Yineng Zhang 2024-09-23 20:40:17 +08:00
• e4780cf839 [API, Feature] Support response prefill for openai API (#1490) Ying Sheng 2024-09-22 06:46:17 -07:00
• 39bb49d156 Update dockerfile to include datamodel_code_generator (#1492) Lianmin Zheng 2024-09-22 04:49:16 -07:00
• 6f3cf1297e [CI, AMD] Add AMD tests to CI (#1491) Ying Sheng 2024-09-22 04:45:10 -07:00
• 13f1357ef0 Add a unit test for data parallelism (#1489) Lianmin Zheng 2024-09-22 02:21:05 -07:00
• 2a99993cd9 Pr fix max workers (#1456) 3 people 2024-09-22 17:20:26 +08:00
• 167591e864 Better unit tests for adding a new model (#1488) Lianmin Zheng 2024-09-22 01:50:37 -07:00
• 441c22db8c doc: update backend (#1486) Yineng Zhang 2024-09-21 22:05:12 +08:00
• ce636ac441 fix incorrect links in documentation (#1481) Ran ChenandYineng Zhang 2024-09-21 05:36:23 -07:00
• 82136eb0b5 chore: bump v0.3.1.post3 (#1483) Yineng Zhang 2024-09-21 11:17:45 +08:00
• b8ccaf4d73 Add MLA gsm8k eval (#1484) Ke Bao 2024-09-21 11:16:13 +08:00
• a68cb201dd Fix triton head num (#1482) Ke Bao 2024-09-21 10:25:20 +08:00
• 014982b5e0 Add OLMoE (#1476) Niklas Muennighoff 2024-09-19 19:32:49 -07:00
• a6db88626e minor: add quant eval compared with base (#1475) Yineng Zhang 2024-09-20 01:57:19 +08:00
• b4408b0d16 feat: update linear deps 1/N (#1305) Yineng Zhang 2024-09-19 20:53:11 +08:00
• 2cd7e181dd Fix env vars in bench_latency (#1472) Lianmin Zheng 2024-09-19 03:19:26 -07:00
• 5ce55aee15 Release v0.3.1.post2 (#1470) Lianmin Zheng 2024-09-19 02:03:38 -07:00
• 2d346a57c2 Fix padding in the cuda graph (#1469) Lianmin Zheng 2024-09-19 01:52:15 -07:00
• 446ea33277 fix: creat new dict everytime for putting new frame (#1464) Li Bo 2024-09-19 16:31:48 +08:00
• 8f527e2940 [Event] Add public meeting invite to README (#1458) Ying Sheng 2024-09-18 08:53:22 -07:00
• 7f24ea95c3 Fuse top_k and top_k in the sampler (#1457) Lianmin Zheng 2024-09-18 04:35:35 -07:00
• 1acccb364a Fix oom issues with fp8 for llama (#1454) Lianmin Zheng 2024-09-18 03:45:19 -07:00
• aa2750beb3 [Bugfix] Enable SGLang on AMD GPUs via PyTorch for ROCm (#1419) (#1453) HAI 2024-09-18 02:01:35 -07:00
• 5e62a6b706 Add bench_server_latency.py (#1452) Lianmin Zheng 2024-09-18 00:56:06 -07:00
• 5752f25eef Fixed n>1 causing list index out of range with VLM (#1449) Xiao Yu 2024-09-18 03:46:32 -04:00
• 7c162fa9c5 Fix schedule bug (#1451) Liangsheng Yin 2024-09-17 22:59:32 -07:00
• 36078fb247 fix schedule bug (#1450) Liangsheng Yin 2024-09-17 16:33:53 -07:00
• b3710d2c93 Fix attention backend (#1448) Ke Bao 2024-09-17 22:07:53 +08:00
• c6b6d2e71b Enable MLA by default (#1447) Ke Bao 2024-09-17 19:42:48 +08:00
• 90a26be31c Release 0.3.1.post1 (#1445) Lianmin Zheng 2024-09-17 01:47:31 -07:00
• 1f4b5f770d Add OLMoE model (#1444) Jani Monoses 2024-09-17 11:14:53 +03:00
• 76524b70d1 Fix torch compile for deepseek-v2 (#1442) Ke Bao 2024-09-17 15:52:08 +08:00
• 3a6e04185b [Feature, Hardware] Enable SGLang on AMD GPUs via PyTorch for ROCm (#1420) HAI 2024-09-17 00:43:52 -07:00
• 2fa5cec775 Simplify sampler and its error handling (#1441) Lianmin Zheng 2024-09-16 21:23:31 -07:00
• 27b557aea7 Clean up model loader (#1440) Lianmin Zheng 2024-09-16 18:16:27 -07:00
• 93dffd699b Add constrained_json_whitespace_pattern to ServerArgs (#1438) zifeitong 2024-09-16 13:29:18 -07:00
• 2abe4f1cb6 Revert "[Minor] Raise exception for wrong import (#1409)" (#1432) Ying Sheng 2024-09-15 15:22:32 -07:00
• 37963394aa [Feature] Support LoRA path renaming and add LoRA serving benchmarks (#1433) Ying Sheng 2024-09-15 12:46:04 -07:00
• 899cf5c438 Remove deprecated configs (#1431) Lianmin Zheng 2024-09-15 08:52:18 -07:00
• e79f6cd73d Release v0.3.1 (#1430) Lianmin Zheng 2024-09-15 07:03:16 -07:00
• 9ba1f09760 [Fix] Fix logprob and normalized_logprob (#1428) Lianmin Zheng 2024-09-15 06:36:06 -07:00
• 282681b8a1 Update backend.md (#1429) Lianmin Zheng 2024-09-15 02:55:34 -07:00
• 58cafe23a7 Add libibverbs-dev to Dockerfile (#1427) William Arnold 2024-09-15 15:40:31 +09:00
• 9463bc1385 Enable torch.compile for triton backend (#1422) Lianmin Zheng 2024-09-14 15:38:37 -07:00
• e3fc4658f4 fix: resolve nightly eval (#1426) Yineng Zhang 2024-09-15 01:07:52 +09:00
• 33b54e7c40 Add pytorch sampling backend ut (#1425) Ke Bao 2024-09-14 23:15:30 +08:00
• 30b404ce72 Add torchao quant for mixtral and qwen_moe (#1418) Jerry Zhang 2024-09-13 23:46:55 -07:00
• 70b6802982 Optimize conflicts between CUDA graph and vocab mask tensors (#1392) Liangsheng Yin 2024-09-13 20:27:53 -07:00
• f3d32f888a ci: fix finish (#1414) Yineng Zhang 2024-09-14 00:01:30 +09:00
• 8779da95d6 Update pr-test.yml (#1412) Lianmin Zheng 2024-09-13 00:37:13 -07:00
• ad0ff62a4c Balance test in CI (#1411) Lianmin Zheng 2024-09-12 23:29:44 -07:00
• 9a903a8784 [Minor] Raise exception for wrong import (#1409) Ying Sheng 2024-09-12 23:02:36 -07:00
• 68be2f6d3b [CI] Include triton backend and online serving benchmark into CI (#1408) Lianmin Zheng 2024-09-12 21:36:41 -07:00
• b912de11b0 Make stop reason a dict instead of str (#1407) Lianmin Zheng 2024-09-12 20:47:31 -07:00
• eb02c1618a [Minor, CI] remove lora test from minimal suite (#1406) Ying Sheng 2024-09-12 16:49:50 -07:00
• 712216928f [Feature] Initial support for multi-LoRA serving (#1307) Ying Sheng 2024-09-12 16:46:14 -07:00
• c33d82a211 Add Support for XVERSE Models (Dense and MoE) to sglang (#1397) 4 people 2024-09-12 16:47:52 +08:00
• 8234e663e9 [Minor Fix] Fix llava modalities issue for single-image (#1402) Kaichen Zhang - NTU 2024-09-12 16:10:26 +08:00
• debbdb5178 kernel: use tensor cores for flashinfer gqa kernels (#1403) Zihao Ye 2024-09-12 00:38:18 -07:00
• 3efa798116 Support cuda graph in the triton attention backend (#1401) Lianmin Zheng 2024-09-12 00:36:55 -07:00
• 2a71be5e25 Fix README format (#1399) William 2024-09-12 14:46:51 +08:00
• 4462137777 Add no commit to main rule (#1393) Liangsheng Yin 2024-09-11 14:40:45 -07:00
• fec185ce0c Refactor attention backend (#1381) Lianmin Zheng 2024-09-11 11:44:26 -07:00
• c03cece42f Improve error reporting during server launch (#1390) Lianmin Zheng 2024-09-11 04:50:04 -07:00
• 15c75e4146 [Fix] Fix --disable-flashinfer (#1389) Lianmin Zheng 2024-09-11 04:36:21 -07:00
• 224200e3c2 BaiChuan2 Model (#1367) Vectoryandwanpenghan 2024-09-11 18:55:24 +08:00
• 8c0efa514d remove assertion in triton attention and add an unit test (#1385) Byron Hsu 2024-09-11 03:22:07 -07:00
• 144bc70fcc Organize flashinfer indices update (#1378) Liangsheng Yin 2024-09-10 17:38:59 -07:00
• 46094e0c1b Deprecate --disable-flashinfer and introduce --attention-backend (#1380) Lianmin Zheng 2024-09-10 17:11:16 -07:00