minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 10 Packages Projects Releases Wiki Activity
18,275 Commits 2 Branches 0 Tags
47a157f257f13311a4408dbde3506f6a9cedaa74
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Lianmin Zheng a2b4e8888f Allow CUDA VMM feature transport with the Rust frontend (#39347) 2026-09-14 17:09:49 -07:00
Lianmin Zheng 7d37b86ff2 Remove obsolete CUDA graph buffer population methods (#37886) 2026-09-06 22:09:49 -07:00
Lianmin Zheng 613d87becd [Memory] Reuse output storage across full prefill CUDA graphs (#38038) 2026-09-04 17:23:31 -07:00
Lianmin Zheng 978cc228ca [Rust] Bound multimodal media ingress (#37967) 2026-09-04 05:22:44 -07:00
Lianmin ZhengandShiyan Deng d7f235daca [Memory] Retire graph borrow pool before updating static runs (#37966) 
Co-authored-by: Shiyan Deng <dsy842974287@meta.com>
 2026-09-04 02:59:52 -07:00
Lianmin Zheng 05dbe64dff Fix buffer-mode idle tracking and VLM memory sizing (#37567) 2026-09-03 13:56:12 -07:00
Lianmin Zheng 619ab2bcce Fix block-scale swizzling device placement (#37849) 2026-09-03 10:47:04 -07:00
Lianmin ZhengjiayisuseYinghai LuHao ZhangYichao Fu
 		87d60a2229 Improve CUDA graph and speculative execution output handling (#37329) 
Co-authored-by: jiayisuse <jiayisuse@fb.com>
Co-authored-by: Yinghai Lu <yinghai@meta.com>
Co-authored-by: Hao Zhang <zhisbug@users.noreply.github.com>
Co-authored-by: Yichao Fu <yichaofu@meta.com>
 2026-09-02 17:25:27 -07:00
Lianmin ZhengShiyan DengYinghai Lu
 		ff04a00d73 Reduce tokenizer overhead and offload CUDA VMM publication (#37330) 
Co-authored-by: Shiyan Deng <dsy842974287@meta.com>
Co-authored-by: Yinghai Lu <yinghai@meta.com>
 2026-09-02 17:21:08 -07:00
Lianmin Zheng f8f04bafa8 Rust server: align launcher and request validation behavior (#37327) 2026-09-01 23:42:36 -07:00
Lianmin ZhengPranjal ShankhdharChengze Fan
 		33428d3dae Fix GPU kernel ordering and MXFP8 quantization dispatch (#37331) 
Co-authored-by: Pranjal Shankhdhar <pranjalssh@meta.com>
Co-authored-by: Chengze Fan <fancz2002@gmail.com>
 2026-09-01 17:09:44 -07:00
Lianmin Zheng 579270d459 [Rust] Simplify request defaults and document batch header ABI (#37226) 2026-08-31 12:31:35 -07:00
Lianmin Zheng 2138494272 [Rust] Keep sampling and scheduler wire schemas in sync (#37222) 2026-08-31 12:31:07 -07:00
Lianmin Zheng 48098b5f23 [Rust] Derive server address and accept signed env values (#37221) 2026-08-31 12:30:40 -07:00
Lianmin Zheng 1da86b9801 [Rust] Split and rename embedded server components (#37220) 2026-08-31 12:28:43 -07:00
Lianmin ZhengandMing Yang 3ed3326631 Decouple speculative draft capacity from runtime state (#36897) 
Co-authored-by: Ming Yang <minos.future@gmail.com>
 2026-08-30 23:31:25 -07:00
Lianmin Zheng a0a2295271 Refactor server_args constants and layout (#36676) 2026-08-27 12:16:16 -07:00
Lianmin Zheng 94183a8d2b Move server args config parser under utils (#36681) 2026-08-27 03:34:50 -07:00
Lianmin Zheng 3402265989 [Core] Refactor server argument choices (#36586) 2026-08-27 01:56:03 -07:00
Lianmin Zhengandyangliu991 bf1e03f712 [MegaMoE] Respect padded MXFP8 scale row strides in pre-dispatch (#36237) 
Co-authored-by: yangliu991 <yangliu991@fb.com>
 2026-08-25 00:27:33 -07:00
Lianmin Zheng b498efce52 chore: move cuda_vmm_utils.py under srt/utils/ (#36053) 2026-08-23 21:47:29 -07:00
Lianmin ZhengandYichao Fu 6fd0384d42 Make draft attention backends extensible (#35932) 
Co-authored-by: Yichao Fu <yichaofu@meta.com>
 2026-08-21 23:54:44 -07:00
Lianmin Zheng f736895ce9 Make PR babysitter launcher fork-safe (#35575) 2026-08-19 17:48:34 -07:00
Lianmin ZhengandYichao Fu 99c12218c3 Support custom draft worker classes in DSpark (#35397) 
Co-authored-by: Yichao Fu <yichaofu@meta.com>
 2026-08-19 17:48:17 -07:00
Lianmin ZhengandXiaozhu Meng 9234e40aed [sampling] Fix int32 offset overflow in top-k renorm Triton kernels (#35571) 
Co-authored-by: Xiaozhu Meng <mxz297@gmail.com>
 2026-08-19 17:48:03 -07:00
Lianmin Zheng 70ee6b1714 [Rust Server] Add e2e latency metadata and fix Sarashina import (#35125) 2026-08-18 02:08:07 -07:00
Lianmin Zheng a779a2a2a5 [Chore] Move version tag helper to release scripts (#35196) 2026-08-18 00:11:12 -07:00
Lianmin Zheng 198a7b2fc9 [Misc] Clean up python/sglang package structure (#35062) 2026-08-17 14:24:35 -07:00
Lianmin Zheng af743371cc Clean up environ.py: remove dead env vars, unify deprecation handling, move examples to a unit test (#35060) 2026-08-17 06:53:34 -07:00
Lianmin Zhengandwangwenchen0407 12a455a910 Fix world-size-one aliasing in MLP batch sync (#34997) 
Co-authored-by: wangwenchen0407 <wangwenchen@meta.com>
 2026-08-17 00:04:09 -07:00
Lianmin ZhengandJialin Ouyang 9be3044b9c [Engine] Freeze GC after server warmup (#34999) 
Co-authored-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
 2026-08-16 22:41:36 -07:00
Lianmin Zheng b7eccd642f Increase post-capture decode memory reserve (#34996) 2026-08-16 15:31:36 -07:00
Lianmin Zheng f61f584347 Add explicit EPLB balancedness reporting modes (#34998) 2026-08-16 15:31:11 -07:00
Lianmin ZhengandJialin Ouyang c6ebcf39ee [VLM] Avoid synchronizing multimodal placeholder counts (#34995) 
Co-authored-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
 2026-08-16 15:15:49 -07:00
Lianmin ZhengandYonghao Zhuang 4c51248427 Support unified SWA page mapping in attention metadata (#35000) 
Co-authored-by: Yonghao Zhuang <yhzhuang@meta.com>
 2026-08-16 15:14:50 -07:00
Lianmin ZhengandYe Qi 32e6fb4fdc [Frontend] Apply request header overrides to chat completions (#35001) 
Co-authored-by: Ye (Charlotte) Qi <ye.charlotte.qi@gmail.com>
 2026-08-16 15:08:47 -07:00
Lianmin ZhengandLu Fang e49557b8da Support model-defined prefill input embedding width (#35002) 
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
 2026-08-16 15:08:19 -07:00
Lianmin Zheng 67e12131df Build Rust extensions on demand in source checkouts (#34994) 2026-08-16 14:58:06 -07:00
Lianmin Zheng 0e231d365a Clean up playground scripts and add PR babysitter launcher (#35018) 2026-08-16 14:43:57 -07:00
Lianmin Zheng 6bb73082c8 Add skill for babysitting PR CI (#35015) 2026-08-16 01:23:32 -07:00
Lianmin Zheng 8bbca87780 [Core] Organize environment variable registry (#34730) 2026-08-13 14:25:37 -07:00
Lianmin Zheng 71043b9dbb Revert parallel request lifecycle tracking from #32588 (#34160) 
Signed-off-by: Lianmin Zheng <lianminzheng@gmail.com>
 2026-08-09 03:10:02 -07:00
Lianmin Zheng bc285b2064 refactor: clean up logits processor helpers (#34158) 2026-08-09 00:22:57 -07:00
Lianmin Zheng b0fd31ba07 Multiple flexibility fixes for DP attention (#33537) 2026-08-04 15:40:40 -07:00
Lianmin Zhengandcctry 34af3ff386 Allow optimistic prefill with L2 hierarchical cache and write-back policy (#33545) 
Co-authored-by: cctry <cctry@meta.com>
 2026-08-04 13:23:08 -07:00
Lianmin Zheng 4794b401d5 [Observability] Add startup, memory, and hybrid SWA diagnostics (#33375) 2026-08-04 12:50:09 -07:00
Lianmin ZhengandJialin Ouyang 5081c063c0 fix(metrics): clear forward occupancy on idle (#33562) 
Co-authored-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
 2026-08-04 12:49:17 -07:00
Lianmin ZhengandItai Gat dea2be5ae3 [CUDA Graph] Allow custom decode graph runners (#33553) 
Co-authored-by: Itai Gat <itaigat.mail@gmail.com>
 2026-08-04 12:48:56 -07:00
Lianmin Zheng 7adf2f4a9a Inline _set_gc into _set_envs_and_config (#33538) 2026-08-04 07:31:56 -07:00
Lianmin ZhengandYinghai Lu d257b58e67 [Router] Report accelerator count in /v1/loads (#33548) 
Co-authored-by: Yinghai Lu <yinghai@meta.com>
 2026-08-04 05:55:35 -07:00
Lianmin Zheng 8f2a3ad6d7 [mem_cache] Label HiCache host pools and clarify post-capture KV sizing logs (#33445) 2026-08-04 04:21:36 -07:00
Lianmin Zhengcctrycctry
 		b57721ccf7 Enable post-capture KV sizing with DP attention (#33427) 
Co-authored-by: cctry <cctry@meta.com>
Co-authored-by: cctry <cctry@fb.com>
 2026-08-04 02:20:24 -07:00
Lianmin Zheng 16d3b118a2 Reduce startup log noise and fix Dynamo / CUDA-graph edge cases (#33428) 2026-08-04 02:19:56 -07:00
Lianmin Zheng 85484c457d docs: refresh README news highlights (#33347) 2026-08-02 22:37:01 -07:00
Rain Jiang 4af8ddb576 support rust sglang server (#29799) 2026-07-31 11:56:31 -07:00
Rain Jiang 3312645a30 wire the rust server modules into lib, runtime, and tokenizer manager (#32877) 2026-07-30 12:46:19 -07:00
Rain Jiang 047635ee35 add the rust server native api handlers and runtime threads (#32876) 2026-07-30 12:46:19 -07:00
Rain Jiang 30643f88bc add the rust server api frame codec and http server entry (#32875) 2026-07-30 12:46:19 -07:00
Rain Jiang 4facc0e18a add the rust server ingress tests, guard, and submit modules (#32874) 2026-07-30 12:46:18 -07:00
Rain Jiang e2c65af229 add the rust server ingress request validation and api server common types (#32873) 2026-07-30 12:46:17 -07:00
Rain Jiang 922d6e5542 add the rust server tokenizer, detokenizer, and egress modules (#32872) 2026-07-30 12:46:17 -07:00
Rain Jiang 35f2e6ab58 update Cargo.lock for the rust sglang-server dependencies (#32871) 2026-07-30 12:32:27 -07:00
Lianmin Zheng 16a52bff23 [Refactor] Move sampling tokenizer validation helper (#32694) 2026-07-28 16:48:03 -07:00
Lianmin Zheng 9ca4023b13 [Core] Clean up array-like msgspec structs (#32688) 2026-07-28 16:25:22 -07:00
Lianmin Zhengandcctry c19a333944 [mm] Handle per-item embeddings in cache misses (#32498) 
Co-authored-by: cctry <cctry@meta.com>
 2026-07-27 17:16:17 -07:00
Lianmin Zheng 34454c06b8 [Refactor] Tidy server_args.py section grouping and drop unused alias (#32496) 2026-07-27 04:09:09 -07:00
Lianmin Zheng 3863612023 Add oulgen to CI_PERMISSIONS.json (#32453) 2026-07-26 15:04:11 -07:00
Lianmin ZhengandAlec S 9989077f24 Use native batched llguidance mask generation (#32412) 
Co-authored-by: Alec S <10566873+alecsolder@users.noreply.github.com>
 2026-07-25 16:36:32 -07:00
Lianmin ZhengandXingyu Liu fae84ac0f9 Fix token count localization for replicated attention-TP forwards (#32411) 
Co-authored-by: Xingyu Liu <38244988+charlotte12l@users.noreply.github.com>
 2026-07-25 16:36:15 -07:00
Lianmin Zheng 5f330004bd Fix flaky test_sampling_mask: mask length can legitimately be top_k + 1 (#32410) 2026-07-25 16:05:57 -07:00
Lianmin Zheng d3cf4dfbaa Update audio container test time estimate (#32408) 
Signed-off-by: Lianmin Zheng <lianminzheng@gmail.com>
 2026-07-25 14:45:08 -07:00
Lianmin Zheng bd3f6a7935 [CI] Remove redundant Rust cache save-if settings (#32243) 2026-07-23 18:48:54 -07:00
Lianmin Zheng 429f6b6d15 Turn on breakable prefill cuda graph for dp attention by default (#31682) 2026-07-21 00:03:43 -07:00
Lianmin ZhengandJason Park e7e8aaa73c Overlap grammar (constrained decoding) with speculative decode verify (#31488) 
Co-authored-by: Jason Park <jasonjk@fb.com>
 2026-07-20 17:36:05 -07:00
Lianmin Zheng 54aaedd76d Clean up prefill CUDA graph runner (#31654) 2026-07-20 12:14:21 -07:00
Lianmin Zheng 7d64858093 Add houseroad to CI_PERMISSIONS.json (#31717) 2026-07-19 15:08:36 -07:00
Lianmin Zhengandwangwenchen0407 555267ed05 Fix ROCm fused KV and KDA paths (#31688) 
Co-authored-by: wangwenchen0407 <wangwenchen@meta.com>
 2026-07-19 15:02:35 -07:00
Lianmin Zheng c95026aed3 Upgrade llguidance to 1.7.6 (#31484) 2026-07-17 16:31:44 -07:00
Lianmin Zhenglmzhengmichael604work
 		4ae9cc3c81 Fix gate stride for 4D decode layouts (#31231) 
Co-authored-by: lmzheng <lmzheng@fb.com>
Co-authored-by: michael604work <michael604@meta.com>
 2026-07-14 20:06:50 -07:00
Lianmin Zheng b4fdce3b63 Fix post-capture KV sizing for SWA pools (#31092) 2026-07-14 20:06:15 -07:00
Lianmin ZhengandLu Fang 22c08a9bee Preserve RMSNorm shape in batch-invariant mode (#30956) 
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
 2026-07-12 17:51:53 -07:00
Lianmin Zheng b94ac87e0c Enable breakable prefill CUDA graph for DP attention (#30898) 2026-07-12 17:10:04 -07:00
Lianmin ZhengandJialin Ouyang 539253e1d5 Gate Rust extension builds (#30927) 
Co-authored-by: Jialin Ouyang <jialino@meta.com>
 2026-07-12 05:44:58 -07:00
Lianmin ZhengandYongji Wu 81d273f73b Handle coredump dirs and cache hit updates (#30897) 
Co-authored-by: Yongji Wu <30348494+libertyeagle@users.noreply.github.com>
 2026-07-12 00:37:43 -07:00
Lianmin ZhengandAbhishek Parmar 4fcc994be1 Support priority request header override (#30811) 
Co-authored-by: Abhishek Parmar <abhishek@orng.net>
 2026-07-10 14:36:43 -07:00
Lianmin Zheng bc82b06400 Add ZYHowell to CI_PERMISSIONS.json (#30810) 2026-07-10 14:09:32 -07:00
Lianmin Zheng 7998fecfd1 [kv canary] Support UnifiedRadixCache in kv-canary and bracket nested model.forward (#30574) 2026-07-10 10:58:55 -07:00
Lianmin ZhengandYonghao Zhuang 3dc93a12ca Improve EPLB dispatch handling and diagnostics (#30646) 
Co-authored-by: Yonghao Zhuang <yhzhuang@meta.com>
 2026-07-10 10:40:19 -07:00
Lianmin Zhengandtanujtiwari1998 32c8973ce8 Configurable decode retraction order (#30573) 
Co-authored-by: tanujtiwari1998 <168470992+tanujtiwari1998@users.noreply.github.com>
 2026-07-09 17:55:26 -07:00
Lianmin Zhengandtanujtiwari1998 cfc66e05c5 [tokenizer] Support pluggable tokenizer worker class in multi-tokenizer mode (#30630) 
Co-authored-by: tanujtiwari1998 <168470992+tanujtiwari1998@users.noreply.github.com>
 2026-07-09 17:31:54 -07:00
Lianmin Zheng 77b7698cad Add tanujtiwari1998 to CI_PERMISSIONS.json (#30649) 2026-07-09 17:03:46 -07:00
Lianmin Zheng a9e804623e Allow EPLB manual test to use FlashInfer A2A (#30641) 2026-07-09 03:46:07 -07:00
Lianmin Zheng ca8f15cd70 Fix FlashInfer A2A IMA by DP-synchronizing the decode graph bucket (#30242) (#30450) 2026-07-08 14:34:49 -07:00
Lianmin Zheng 24c42c90be Clean up ServerArgs post-init dispatch (#30186) 2026-07-05 23:05:07 -07:00
Lianmin Zheng 9d00385b63 Cleanup: relocate temp_set_env and consolidate multi-device/CUDA helpers in common.py (#30180) 2026-07-05 18:47:26 -07:00
Lianmin Zheng 8673e85e6c Remove # fmt: off from environ.py Envs class (#30153) 2026-07-05 12:45:24 -07:00
Lianmin Zheng 92a1f6e06c [refactor] Reorder ServerArgs sections common-first; inline LLAMA4/MIMO_V2 arch tuples (#30151) 2026-07-05 12:17:24 -07:00
Lianmin Zheng 1a5977d41b [chore] Add no-getattr rule; refine no-dataclasses rule (#29871) 2026-07-01 15:31:34 -07:00
Lianmin Zheng 76d828a4e8 Add pranjalssh to CI_PERMISSIONS.json (#29870) 2026-07-01 14:52:02 -07:00
Lianmin Zheng bb74ed4a8d Replace hasattr with isinstance in SHM feature helpers (#29549) 2026-06-28 21:31:28 -07:00