Commit Graph

Select branches

Hide Pull Requests

dsv41-pd

dsv41-pd-visioncp

main

Mono Color

• e3938b2f9c feat: update other MoE models deps (#2156) Yineng Zhang 2024-11-24 21:36:34 +08:00
• c211e7b669 Simplify batch update (#2154) Lianmin Zheng 2024-11-24 04:47:10 -08:00
• d90c3d6b8b fix: resolve end-of-file-fixer (#2157) Yineng Zhang 2024-11-24 20:38:26 +08:00
• 9e8f8fbf95 feat: update gitignore and add tuning config for FusedMoE (#2155) Yineng Zhang 2024-11-24 20:24:58 +08:00
• b509db5832 feat: remove the dependency on FusedMoE (#2153) Yineng Zhang 2024-11-24 20:09:27 +08:00
• dbe1729395 Merged three native APIs into one: get_server_info (#2152) Henry Hyeonmok Ko 2024-11-24 01:37:58 -08:00
• 84a1698d67 Update release-pypi-router.yml Byron Hsu 2024-11-23 17:35:25 -08:00
• 32293a299c Improve sglang router (#2148) Byron Hsu 2024-11-23 17:34:24 -08:00
• 7921690893 add prefix match for certain tenant (#2147) Byron Hsu 2024-11-23 15:35:02 -08:00
• bbb81c2457 Add more api routes (completion, health, etc) to the router (#2146) Byron Hsu 2024-11-23 15:10:26 -08:00
• 52f58fc42a fix dp_rank env (#2144) Byron Hsu 2024-11-23 11:46:21 -08:00
• 145c0ddc2d update router doc (#2143) Byron Hsu 2024-11-23 11:01:04 -08:00
• 505d7f71a6 Bump sglang-router to 0.0.5 (#2142) Byron Hsu 2024-11-23 08:35:46 -08:00
• cbedd1db1d [router] cache-aware load-balancing router v1 (#2114) Byron Hsu 2024-11-23 08:34:48 -08:00
• ad47749b82 fix: resolve bench_serving args (#2139) Yineng Zhang 2024-11-23 17:45:42 +08:00
• 751c3a037c Fix dp print message (#2138) Lianmin Zheng 2024-11-23 01:22:26 -08:00
• 60769be14d Add concurrency option for benchmark (#2136) Yunmeng 2024-11-23 17:07:07 +08:00
• a78d8f8db3 [CI] Fix test cases (#2137) Lianmin Zheng 2024-11-23 01:00:07 -08:00
• c5f865013e Fix grid size in Triton decoding kernel (#2134) Ke Bao 2024-11-23 16:51:46 +08:00
• d98fa1e93d Add simple CPU offloading support. (#2081) Jani Monoses 2024-11-23 08:23:53 +02:00
• 865233e256 Add initial support for intel Gaudi accelerators (#2121) Ankur Neog 2024-11-23 09:52:23 +05:30
• 66d4859acf Revert "Only stream output on tp rank 0" (#2130) Lianmin Zheng 2024-11-22 15:46:16 -08:00
• e1b63624d7 Only stream output on tp rank 0 (#2124) Lianmin Zheng 2024-11-22 15:13:44 -08:00
• c35cd1f8c7 Expose max total num tokens from Runtime & Engine API (#2092) Henry Hyeonmok Ko 2024-11-22 15:10:10 -08:00
• 72f87b723b feat(pre-commit): trim unnecessary notebook metadata from git history (#2127) Xuehai Pan 2024-11-23 05:04:51 +08:00
• 62a4a339eb docs: fix module docstrings and copyright headers (#2077) Xuehai Pan 2024-11-22 22:16:53 +08:00
• 2797bc3422 fix: add xgrammar dependency (#2126) Yineng Zhang 2024-11-22 20:53:11 +08:00
• 9a00e6f453 chore: bump v0.3.6 (#2120) Yineng Zhang 2024-11-22 19:27:30 +08:00
• 4f8c3aeafc minor: update gsm8k threshold (#2125) Yineng Zhang 2024-11-22 19:23:58 +08:00
• 2369e88209 [minor] Clean up unused imports (#2122) Lianmin Zhengandrinrin32 2024-11-22 01:50:42 -08:00
• ad30d5cf9a Benchmark with Pytorch Profiler easily (#2110) bjmsongandroot 2024-11-22 15:29:50 +08:00
• dfec7fca06 Rename sglang.bench_latency to sglang.bench_one_batch (#2118) Lianmin Zheng 2024-11-21 20:07:48 -08:00
• 8048c28c11 Fix #2037 - Context length check does not take into out pad tokens for visual models (#2106) Jake Poznanski 2024-11-21 19:05:41 -08:00
• 30af7dfb34 [router] add base_gpu_id server args & merged radix tree python reference (#2115) Byron Hsu 2024-11-21 17:13:33 -08:00
• f6f713797b Add support for Qwen2-VL-based embedding models (#2055) James Xu 2024-11-21 17:24:25 -05:00
• f35cb46cc3 ROCm: Fix MoE padding for none FP8 cases (#2111) HAI 2024-11-21 12:23:21 -08:00
• 7f8fcd39cd Turn off autotune for scaled mm for fp8 dynamic quant in torchao (#2116) Jerry Zhang 2024-11-21 12:19:49 -08:00
• 5c6a41facf Error out when torchao-config option is not recognized (#2107) Jerry Zhang 2024-11-20 17:37:28 -08:00
• 722530fa01 Enable overlap scheduler by default for the triton attention backend (#2105) Lianmin Zheng 2024-11-20 02:58:35 -08:00
• 56a347f7d3 Move test_session_id.py to playground (#2104) Lianmin Zheng 2024-11-20 01:28:27 -08:00
• 3295cd8af2 Allow skipping warmup in bench_offline_throughput.py (#2103) Lianmin Zheng 2024-11-20 01:25:21 -08:00
• 5942dfc00a [feat] Add session control (#2073) Ying Sheng 2024-11-20 00:36:53 -08:00
• 63a395b985 Update nightly-eval.yml (#2100) Lianmin Zheng 2024-11-19 22:15:02 -08:00
• 7d671e4ad2 Enable overlap by default (#2067) Lianmin Zheng 2024-11-19 22:07:58 -08:00
• 699384cb01 Set schedule policy more conservative for DP attention (#2096) Ke Bao 2024-11-20 12:57:18 +08:00
• ffd20fcd03 Make constrained decoding work for overlap scheduler (#2095) Lianmin Zheng 2024-11-19 15:04:43 -08:00
• 55bd97f3e5 minor: add dataset dump and questions shuffle (#2093) Yineng Zhang 2024-11-20 06:07:27 +08:00
• e57c3e12b8 Use native fp8 format on MI300X (#2094) HAI 2024-11-19 14:06:29 -08:00
• f239268fad minor: update gsm8k eval (#2091) Yineng Zhang 2024-11-19 20:36:55 +08:00
• 929c7621af Fix: incorrect top_logprobs in chat completion (#2088) Alexander Waitz 2024-11-19 04:21:36 -08:00
• b7a065eae3 Use cuda event wait and synchronization instead of busy waiting (#2089) Lianmin Zheng 2024-11-19 00:21:46 -08:00
• b110453802 Simplify logits penalizer (#2086) Lianmin Zheng 2024-11-18 17:48:28 -08:00
• 3b44bbeecf Allow passing extra request body to bench_offline_throughput.py (#2085) Lianmin Zheng 2024-11-18 14:59:15 -08:00
• 80e2c4a8de Fix chunked prefill with output logprob (#2083) Lianmin Zheng 2024-11-18 13:16:28 -08:00
• 66318ffe96 Rename layer_idx to layer_id for consistency (#2078) Jani Monoses 2024-11-18 23:00:02 +02:00
• 766192610e feat: update torch 2.5.1 (#2069) Yineng Zhang 2024-11-18 21:29:13 +08:00
• 2a3992b6f1 support set role as 'tool' (#2075) yukavioandkavioyu 2024-11-18 17:06:59 +08:00
• 4af3f889fc Simplify flashinfer indices update for prefill (#2074) 3 people 2024-11-18 00:02:36 -08:00
• df7fe4521a Crash the CI jobs on model import errors (#2072) Lianmin Zheng 2024-11-17 22:18:11 -08:00
• a7164b620f Tune the threshold for accuracy tests in CI (#2071) Lianmin Zheng 2024-11-17 21:51:00 -08:00
• 116685337e Fix cuda illegal memory access in overlap mode (#2070) Lianmin Zheng 2024-11-17 21:29:30 -08:00
• a9e90b4bce [Minor] Fix styles for overlap mode (#2068) Lianmin Zheng 2024-11-17 19:49:20 -08:00
• 8c280cee55 add phi-3 small support (#2062) TanjiroandTushar Goel 2024-11-17 18:47:43 -08:00
• 9c745d078e [Performance] Update xgrammar-related constrained decoding (#2056) DarkSharpness 2024-11-18 09:58:49 +09:00
• ebaa2f3199 Rename arguments --disable-nan-detection to --enable-nan-detection (#2066) Lianmin Zheng 2024-11-17 16:53:44 -08:00
• 62832bb272 Support cuda graph for DP attention (#2061) Ke Bao 2024-11-18 08:29:20 +08:00
• 11f881d173 Deprecate --disable-flashinfer and --disable-flashinfer-sampling (#2065) Lianmin Zheng 2024-11-17 16:20:58 -08:00
• 38625e2139 Remove monkey_patch_vllm_dummy_weight_loader (#2064) Lianmin Zheng 2024-11-17 15:48:12 -08:00
• c1f401fc58 Revert "chore: update torch v2.5.1" (#2063) Lianmin Zheng 2024-11-17 15:29:38 -08:00
• 3b878863f7 chore: update torch v2.5.1 (#1849) Yineng Zhang 2024-11-18 00:06:00 +08:00
• f719d9aebc Launch dp ranks in parallel (#2053) Lianmin ZhengandHaotian Liu 2024-11-16 17:13:36 -08:00
• edad373135 Fix illegal memory access in overlap mode & Use more fused triton kernels for building meta data (#2051) Lianmin Zheng 2024-11-16 16:14:23 -08:00
• 976bc302e5 Support DP MLA (#1970) Ke Bao 2024-11-16 17:01:43 +08:00
• 2f2e07439c Fix weight update for data parallelism (#2050) Lianmin Zheng 2024-11-16 00:30:39 -08:00
• 2ffe0a7363 Add get_amdgpu_memory_capacity() (#2049) HAI 2024-11-15 22:51:48 -08:00
• cf2489762b Add Tensor Parallel to torch_native_llama (#1876) Ke Wen 2024-11-15 21:26:00 -08:00
• e5c6715003 Fix core (MI300X) with --enable-overlap (#2048) HAI 2024-11-15 21:24:42 -08:00
• 023d0a73df fix small typos in docs (#2047) Xiaoyu Zhang 2024-11-16 03:09:10 +08:00
• 32c9a7ec11 Release v0.3.5.post2 (#2046) Lianmin Zheng 2024-11-15 06:54:00 -08:00
• b01df48cf2 [Fix] Adjust default chunked prefill size and cuda graph max bs according to GPU memory capacity (#2044) Lianmin Zheng 2024-11-15 06:21:57 -08:00
• c29b98e043 Fix json benchmark (#2043) Lianmin Zheng 2024-11-15 05:33:43 -08:00
• 954f4e6bd6 benchmark json schema (#2030) DarkSharpness 2024-11-15 22:06:19 +09:00
• 2558d6a675 Fix the default arguments of bench_offline_throughput.py & simplify detokenizer manager (#2042) Lianmin Zheng 2024-11-15 05:02:44 -08:00
• 29ebe3dff4 fix: align enable_overlap_scheduler naming between code and docs (#2038) ws 2024-11-15 19:39:10 +08:00
• f6dd648620 Offline LLM Engine Benchmark Throughput (#1968) zolinthecowandByronHsu 2024-11-14 21:59:33 -08:00
• ea53c63bad Expose no_stop_trim and skip_special_tokens in openai api (#2039) Lianmin Zheng 2024-11-14 19:09:21 -08:00
• a10d530943 Fix outlines version (#2036) Lianmin Zheng 2024-11-14 12:52:40 -08:00
• aae5434bdf Fix unit tests (#2034) Lianmin Zheng 2024-11-14 11:08:37 -08:00
• c3eac1b010 Fix torch.compile for MoE (#2033) Lianmin Zheng 2024-11-14 01:30:24 -08:00
• b275ce0043 Github runner instructions for AMD (#2031) HAI 2024-11-13 23:57:18 -08:00
• 13ce3e4b5d Add download_dir ServerArgs property (#2027) Patrick Yi 2024-11-14 02:26:56 -05:00
• df246e699d chore: open lto and optimization in release profile (#2028) Tzu Gwo 2024-11-14 15:02:39 +08:00
• fb9fb3518b set content to empty string (#2026) chottolabs 2024-11-13 20:06:02 -05:00
• c722d9bdc3 Fix dependency and error message for xgrammar (#2024) Lianmin Zheng 2024-11-13 14:04:25 -08:00
• 218ab3611d Do not let invalid grammar crash the server (#2023) Lianmin Zheng 2024-11-13 11:39:16 -08:00
• f407fcf9ef Release v0.3.5.post1 (#2022) Lianmin Zheng 2024-11-13 10:27:12 -08:00
• 54479d6f30 Fix grammar backend for tensor parallelism (#2020) Lianmin Zheng 2024-11-13 01:49:45 -08:00
• ba069a24d3 Fix grammar backend (#2018) Lianmin Zheng 2024-11-12 21:17:38 -08:00
• 125b1199c5 support parallel grammar preprocessing (#1996) DarkSharpnessandLianmin Zheng 2024-11-13 01:45:28 +09:00
• eff468dd5a fix test_embedding_models prompt length too long's bug (#2015) Xiaoyu Zhang 2024-11-12 23:21:16 +08:00