minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity
8,152 Commits 3 Branches 0 Tags
5c64a20da7dd15ca743f453c73b8d6d955a09493
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Lianmin Zheng cf81737693 Add kv_transfer_total_mb to metrics (#15667) 2025-12-23 11:16:55 -08:00
Lianmin Zheng 5e1a495c65 Improve engine customization interface (#15635) 2025-12-22 14:24:16 -08:00
Lianmin Zhenggithub-actions[bot]Hanming Lu
 		92e6b3c30e [Auto Sync] Update scheduler_runtime_checker_mixin.py (20251219) (#15437) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
 2025-12-19 01:46:03 -08:00
Lianmin Zheng f228b662a7 Update readme (#15425) 2025-12-18 23:06:00 -08:00
Lianmin Zheng d1f0063262 Clean up __init__ function of the scheduler and event loop for PD (#15298) 2025-12-18 01:35:14 -08:00
Lianmin Zheng 9d64a7b24f Minor style fixes to the scheduler.py (#15218) 2025-12-16 17:09:44 -08:00
Lianmin Zheng 702426b06a Fix import warnings (#15144) 2025-12-14 21:25:18 -08:00
Lianmin Zheng 267170bf1d Clean up server args and engine startup processes (#15015) 2025-12-12 18:46:07 -08:00
Lianmin Zheng b737a125c5 Update ci permission (#15014) 2025-12-12 14:11:40 -08:00
Lianmin Zheng 1b5e903480 Refactor of http and engine entrypoints to allow custom override (#14869) 2025-12-12 12:01:08 -08:00
Lianmin Zheng 25e97380e3 Fix CUDA version handling in ci_install_deepep.sh (#14854) 2025-12-10 16:52:30 -08:00
Lianmin Zhengandgithub-actions[bot] 4285e99da7 [Auto Sync] Update data_parallel_controller.py, detokenizer... (20251209) (#14759) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
 2025-12-09 18:38:38 -08:00
Lianmin Zheng 18bd8e8d6d Improve CI by trying a warmup before unit tests (#14669) 2025-12-09 15:17:59 -08:00
Lianmin Zheng 036e64dafa move multi-item scoring functions in tokenizer manager into a separate file (#14740) 2025-12-09 14:47:06 -08:00
Lianmin Zheng ab0048793c Revert "[Feat] Add received_time in serving_base" (#14743) 2025-12-09 06:52:27 -08:00
Lianmin ZhengandByron Hsu ce4e836be5 Add per-request decode tp size (#14678) 
Co-authored-by: Byron Hsu <byronhsu1230@gmail.com>
 2025-12-08 20:24:31 -08:00
Lianmin Zheng 46d7b35ec7 Move custom_ops under layers; move _custom_ops.py → custom_all_reduce_ops.py (#14326) 2025-12-03 10:33:37 -08:00
Lianmin Zheng ca52ed425f Clean up imports and move files (#14317) 2025-12-02 16:31:54 -08:00
Lianmin Zheng fc6fb550ec [Minor] update docs on CI (#14315) 2025-12-02 12:06:54 -08:00
Lianmin Zhenggithub-actions[bot]Hanming LuHanming Lu
 		64092c8b55 [Auto Sync] Rename is_hybrid to is_hybrid_swa (#14252) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: Hanming Lu <hanming@x.ai>
 2025-12-01 23:24:24 -08:00
Lianmin Zheng 8a7b1b8301 [Docs] Update CI docs (#14260) 2025-12-01 18:15:03 -08:00
Lianmin Zhenggithub-actions[bot]Dan Zheng
 		796d82b107 [Auto Sync] Add max_total_num_tokens metric: Update scheduler_metrics_mixin.py, collector.py (20251202) (#14256) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Dan Zheng <dzheng@x.ai>
 2025-12-01 16:34:34 -08:00
Lianmin Zhenggithub-actions[bot]gauravjain14
 		1da59e8304 [Auto Sync] optionally disable fake register in Update fp8_kernel.py (20251202) (#14255) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: gauravjain14 <41287729+gauravjain14@users.noreply.github.com>
 2025-12-01 16:11:12 -08:00
Lianmin Zheng c4293f59ac Change PR test schedule to run every 6 hours (#14218) 2025-12-01 03:50:58 -08:00
Lianmin Zheng bc3d2a85af [Minor] update docs (#14212) 2025-12-01 02:33:58 -08:00
Lianmin Zhenggithub-actions[bot]Stefan HeByron Hsu
 		6df1e8d628 [Auto Sync] Update backend.py (20251130) (#14153) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Stefan He <hebiaobuaa@gmail.com>
Co-authored-by: Byron Hsu <byronhsu1230@gmail.com>
 2025-11-30 22:15:02 -08:00
Lianmin ZhengChang SuXinyuan TongXinyuan Tong
 		155a9e7237 Fix condition for streaming output_ids in tokenizer manager (#13759) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Chang Su <chang.s.su\n@oracle.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-11-29 13:56:15 -08:00
Lianmin Zheng ce6b7dfce7 Add auto-tune workflow (#14124) 2025-11-28 23:23:53 -08:00
Lianmin Zheng 231df4b0d4 Cleanup server args (#14027) 2025-11-26 16:32:41 -08:00
Lianmin ZhengandStefan He 0a186924ba [Code sync] Fix registration of some ops in grok & Fix oss sync scripts (#13990) 
Co-authored-by: Stefan He <hebiaobuaa@gmail.com>
 2025-11-26 13:11:52 -08:00
Lianmin Zheng 1ab6ce0e62 [Auto Sync] Improve profilers and simplify bench_one_batch_server.py (#13866) 2025-11-25 12:13:31 -08:00
Lianmin Zhenggithub-actions[bot]Hanming LuHanming Lu
 		e83bd1fadc [Auto Sync] Update schedule_batch.py, schedule_policy.py, b... (20251122) (#13763) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: Hanming Lu <hanming@x.ai>
 2025-11-24 14:33:31 -08:00
Lianmin Zhenggithub-actions[bot]cctry
 		b41afa3754 [Auto Sync] Update evict_policy.py, radix_cache.py (20251120) (#13669) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: cctry <shiyang@x.ai>
 2025-11-21 22:07:36 -08:00
Lianmin Zhengandsglang-bot 99e13d189b Fix url: use https://roadmap.sglang.io for roadmap (#13733) 
Co-authored-by: sglang-bot <sglangbot@gmail.com>
 2025-11-21 05:42:32 -08:00
Lianmin Zhenggithub-actions[bot]Zhuqi Li
 		90a0133515 [Auto Sync] Update http_server.py, io_struct.py, scheduler_... (20251120) (#13679) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Zhuqi Li <zhli@x.ai>
 2025-11-20 23:17:15 -08:00
Lianmin Zheng a352e833c4 CI: Kill zombie diffusion processes in CI & minor code style fix on rotary embedding fallback (#13637) 2025-11-20 21:57:01 +08:00
Lianmin Zheng 7af9b88c6c Revert "[Feature] Introduce JIT Kernel in sglang (with hicache JIT kernel)" (#13644) 2025-11-20 02:11:12 -08:00
Lianmin Zhenggithub-actions[bot]Sehoon Kim
 		17b24aca6e [Auto Sync] Update base_grammar_backend.py, collector.py (20251116) (#13357) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sehoon Kim <sehoon@x.ai>
 2025-11-19 16:05:50 -08:00
Lianmin Zhengandsglang-bot 2e1dbdb258 Update docs (#13519) 
Co-authored-by: sglang-bot <sglangbot@gmail.com>
 2025-11-18 06:24:58 -08:00
Lianmin Zhengandsglang-bot 6d025fd35b Trigger CI retry with edit (#13516) 
Co-authored-by: sglang-bot <sglangbot@gmail.com>
 2025-11-18 05:42:31 -08:00
Lianmin Zhengandsglang-bot 63807079b9 Add docs on trigger ci (#13513) 
Co-authored-by: sglang-bot <sglangbot@gmail.com>
 2025-11-18 05:23:05 -08:00
Lianmin Zhengandsglang-bot f6cfe9f197 Use slash command to trigger CI (#13512) 
Co-authored-by: sglang-bot <sglangbot@gmail.com>
 2025-11-18 04:39:58 -08:00
Lianmin Zhengandsglang-bot 7bc99d4164 README.md -> FOLDER_README.md (#13510) 
Co-authored-by: sglang-bot <sglangbot@gmail.com>
 2025-11-18 04:10:04 -08:00
Lianmin Zhengandsglang-bot e2d6746808 Add .github/CI_PERMISSIONS.json to define the CI permissions (#13509) 
Co-authored-by: sglang-bot <sglangbot@gmail.com>
 2025-11-18 04:00:15 -08:00
Lianmin Zhengsglang-botYing Sheng
 		4d59761623 [Doc] Update CI oncall list (#13396) 
Co-authored-by: sglang-bot <sglangbot@gmail.com>
Co-authored-by: Ying Sheng <sqy1415@gmail.com>
 2025-11-16 20:43:12 -08:00
Lianmin Zhengandsglang-bot 7e626d12b7 Update docs (#13391) 
Co-authored-by: sglang-bot <sglangbot@gmail.com>
 2025-11-16 19:36:33 -08:00
Lianmin Zheng db7299aa30 [Fix] Register custom ops only if they exist (#13321) 2025-11-15 17:48:17 -08:00
Lianmin ZhengandStefan He 22f641ab4f [minor] remove debug code in python/sglang/srt/compilation/weak_ref_tensor_jit.py (#13235) 
Co-authored-by: Stefan He <hebiaobuaa@gmail.com>
 2025-11-14 12:22:24 -08:00
Lianmin Zhenggithub-actions[bot]Stefan He
 		7a2254b260 [Auto Sync] Update backend.py, forward_batch_info.py, piece... (20251113) (#13221) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Stefan He <hebiaobuaa@gmail.com>
 2025-11-13 18:54:11 -08:00
Lianmin Zhenggithub-actions[bot]gemini-code-assist[bot]
 		19f6a33ce9 [Auto Sync] Update pynccl_wrapper.py, environ.py, registry.... (20251111) (#13097) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-11-13 15:58:47 -08:00
Lianmin ZhengandStefan He d646cf6347 [Auto Sync] Update test_deterministic.py (20251112) (#13128) 
Co-authored-by: Stefan He <hebiaobuaa@gmail.com>
 2025-11-12 11:12:23 -08:00
Lianmin Zheng 40b26b456b Simplify the BatchMultimodalOutput in io_struct.py (#12993) 2025-11-10 13:54:56 -08:00
Lianmin Zhenggithub-actions[bot]Stefan He
 		9ea2c686c7 [Auto Sync] Update batch_invariant_ops.py (20251109) (#12916) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Stefan He <hebiaobuaa@gmail.com>
 2025-11-10 01:51:39 -08:00
Lianmin Zhenggithub-actions[bot]Stefan He
 		0296f1cdad [Auto Sync] Update activation.py, logits_processor.py, rota... (20251107) (#12853) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Stefan He <hebiaobuaa@gmail.com>
 2025-11-07 22:07:51 -08:00
Lianmin Zheng ae62279071 Fix data parallel controller launch for num nodes > 2 (#12822) 2025-11-07 12:13:21 -08:00
Lianmin Zheng c7d57d5bb3 Fix CI and style (#12658) 2025-11-05 15:08:15 -08:00
Lianmin Zheng fb2e816e83 Fix server args for gpt oss so users can override the moe runner backend (#12696) 2025-11-05 11:36:59 -08:00
Lianmin Zhenggithub-actions[bot]SangBin Cho
 		55dfb539cf [Auto Sync] Update scheduler_metrics_mixin.py, collector.py (20251104) (#12647) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
 2025-11-04 13:56:14 -08:00
Lianmin ZhengKan Wucctry
 		f600866a44 Improve the metrics for PD (#12580) 
Co-authored-by: Kan Wu <wukanustc@gmail.com>
Co-authored-by: cctry <shiyang@x.ai>
 2025-11-03 22:10:57 -08:00
Lianmin Zheng 243c064df2 Remove the dependency of nccl.h in symmetric memory (#12571) 2025-11-03 16:11:00 -08:00
Lianmin ZhengandNicolas Castet 7a21d8b276 Reduce the overhead of nccl symmetric memory (#12524) 
Co-authored-by: Nicolas Castet <ncastet@nvidia.com>
 2025-11-03 11:56:27 -08:00
Lianmin Zheng 20315697f4 move all get_stream in sgl_kernel to c++ to reduce the launch overhead (#12521) 2025-11-02 13:15:05 -08:00
Lianmin Zheng 2d5605e89b Fix ci install to allow prerelease (#12449) 2025-10-31 02:22:15 -07:00
Lianmin ZhengandByron Hsu c0652d907b Clean up sgl kernel (#12413) 
Co-authored-by: Byron Hsu <byronhsu1230@gmail.com>
 2025-10-31 01:13:34 -07:00
Lianmin Zheng 6a63a9852e minor code sync (#12403) 2025-10-30 12:49:31 -07:00
Lianmin Zheng 7cd716f776 Update news section in README.md (#12409) 2025-10-30 12:41:21 -07:00
Lianmin Zheng 7b36c47b3b Clean up attention backend selection code & Other minor rename (#12136) 2025-10-25 23:50:12 -07:00
Lianmin Zheng 8e70064c37 Clean up server launch code and multi tokenizer (#12132) 2025-10-25 16:40:27 -07:00
Lianmin Zhengandsglang-bot 3862661cd4 Fix token for CI monitor (#12127) 
Co-authored-by: sglang-bot <sglangbot@gmail.com>
 2025-10-25 12:38:26 -07:00
Lianmin Zhengandsglang-bot 428710c2ec clean up github tokens (#12126) 
Co-authored-by: sglang-bot <sglangbot@gmail.com>
 2025-10-25 12:23:54 -07:00
Lianmin Zheng 4caca1ba04 Clean up server args & Add CI scripts (#12124) 2025-10-25 11:53:57 -07:00
Lianmin Zhenggithub-actions[bot]Stefan He
 		ea13cb1452 [Auto Sync] Update test_deterministic.py, test_deterministi... (20251024) (#12083) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Stefan He <hebiaobuaa@gmail.com>
 2025-10-24 22:28:01 -07:00
Lianmin Zheng ffc722a690 Revert "lang: support direct video inference" (#12038) 2025-10-23 19:21:31 -07:00
Lianmin ZhengandStefan He ab07cd3e5a [Auto Sync] Update test_deterministic_utils.py (20251023) (#12022) 
Co-authored-by: Stefan He <hebiaobuaa@gmail.com>
 2025-10-23 11:20:45 -07:00
Lianmin Zheng 6d535b719f Revert "Recapture cuda graph after model weight update to resolve IMA error " (#11980) 2025-10-22 11:50:26 -07:00
Lianmin Zheng 01f14a7ad2 [code move] move pp into a separate mixin (#11838) 2025-10-20 18:46:56 -07:00
Lianmin Zhenggithub-actions[bot]Kan Wu
 		43ad05907c [Auto Sync] Update scheduler.py, server_args.py (20251020) (#11875) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Kan Wu <wukanustc@gmail.com>
 2025-10-20 17:41:19 -07:00
Lianmin Zheng 6058fb520c Update CODEOWNERS for layer quantization path (#11818) 2025-10-18 21:17:17 -07:00
Lianmin Zheng 67e34c56d7 Fix install instructions and pyproject.tomls (#11781) 2025-10-18 01:08:01 -07:00
Lianmin ZhengSangBin ChoCheng WanCheng Wan
 		9eefe2c0b7 Set CUDA_VISIBLE_DEVICES to achieve one GPU per process (#9170) 
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
Co-authored-by: Cheng Wan <cwan@x.ai>
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
 2025-10-17 17:30:06 -07:00
Lianmin Zhenggithub-actions[bot]Cheng Wan
 		fdd7c69d65 [Auto Sync] Update common.py (20251017) (#11782) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
 2025-10-17 15:03:42 -07:00
Lianmin ZhengStefan HeByron Hsu
 		b9a54e0968 [minor] sync code on python/sglang/test/test_deterministic.py and improve ci tests (#11777) 
Co-authored-by: Stefan He <hebiaobuaa@gmail.com>
Co-authored-by: Byron Hsu <byronhsu1230@gmail.com>
 2025-10-17 14:25:22 -07:00
Lianmin Zheng cd7e1bd591 Sync code and test CI; rename some env vars (#11686) 2025-10-15 18:37:03 -07:00
Lianmin Zheng d314bf6010 Update install.md (#11631) 2025-10-14 14:34:46 -07:00
Lianmin Zhenggithub-actions[bot]Byron Hsu
 		b98cf39866 [Auto Sync] Update collector.py (20251014) (#11625) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Byron Hsu <byronhsu1230@gmail.com>
 2025-10-14 13:34:33 -07:00
Lianmin Zhenggithub-actions[bot]Stefan He
 		27d710457c [Auto Sync] Update scheduler.py, server_args.py (20251014) (#11623) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Stefan He <hebiaobuaa@gmail.com>
 2025-10-14 13:20:03 -07:00
Lianmin Zheng f7f9f8eceb Update news section in README.md (#11598) 2025-10-14 00:49:39 -07:00
Lianmin Zhenggithub-actions[bot]Hanming Lu
 		ba2ce28fe9 [Auto Sync] Update model_config.py (20251014) (#11580) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
 2025-10-13 22:16:34 -07:00
Lianmin Zheng 5e3f7e7fa9 Minor: improve sampler & remove unused fields from model_config.py (#11531) 2025-10-13 11:04:44 -07:00
Lianmin Zheng 2ac46e94ef Sync changes on io_struct.py and deterministic ops (#11498) 2025-10-12 16:03:10 -07:00
Lianmin Zheng 5a6ec8f999 Fix unit tests (#11503) 2025-10-12 07:45:57 -07:00
Lianmin Zheng 6a653bb11b temporarily remove b200 tests (#11502) 2025-10-12 06:48:49 -07:00
Lianmin Zheng 548a57b1f3 Fix port conflicts in CI (#11497) 2025-10-12 06:46:36 -07:00
Lianmin Zheng 88e73ed048 Temporarily remove b200 tests (#11501) 2025-10-12 06:41:37 -07:00
Lianmin Zheng 61055cb309 Reorder PD disagg CI tests (#11438) 2025-10-10 17:56:49 -07:00
Lianmin Zheng b4408e6098 Revert "fix: fix video input for qwen3-vl" (#11437) 2025-10-10 12:44:40 -07:00
Lianmin Zheng 9b8ebb2798 move more files under srt/utils (#11285) 2025-10-09 16:46:15 -07:00
Lianmin Zheng 708f4ff490 Rename max_micro_batch_size -> pp_max_micro_batch_size (#11279) 2025-10-06 15:50:56 -07:00
Lianmin Zhenggithub-actions[bot]Sehoon Kim
 		e2daeb351c [Auto Sync] Update test_utils.py (20251006) (#11280) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sehoon Kim <sehoon@x.ai>
 2025-10-06 15:49:57 -07:00
Lianmin Zheng 0e7b353009 Fix code sync scripts (#11276) 2025-10-06 15:35:01 -07:00