minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity
15,978 Commits 3 Branches 0 Tags
f5f021672a3ea3b312ec31c2c420db37f3016bd6
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Rain Jiang 4af8ddb576 support rust sglang server (#29799) 2026-07-31 11:56:31 -07:00
Rain Jiang 3312645a30 wire the rust server modules into lib, runtime, and tokenizer manager (#32877) 2026-07-30 12:46:19 -07:00
Rain Jiang 047635ee35 add the rust server native api handlers and runtime threads (#32876) 2026-07-30 12:46:19 -07:00
Rain Jiang 30643f88bc add the rust server api frame codec and http server entry (#32875) 2026-07-30 12:46:19 -07:00
Rain Jiang 4facc0e18a add the rust server ingress tests, guard, and submit modules (#32874) 2026-07-30 12:46:18 -07:00
Rain Jiang e2c65af229 add the rust server ingress request validation and api server common types (#32873) 2026-07-30 12:46:17 -07:00
Rain Jiang 922d6e5542 add the rust server tokenizer, detokenizer, and egress modules (#32872) 2026-07-30 12:46:17 -07:00
Rain Jiang 35f2e6ab58 update Cargo.lock for the rust sglang-server dependencies (#32871) 2026-07-30 12:32:27 -07:00
Lianmin Zheng 16a52bff23 [Refactor] Move sampling tokenizer validation helper (#32694) 2026-07-28 16:48:03 -07:00
Lianmin Zheng 9ca4023b13 [Core] Clean up array-like msgspec structs (#32688) 2026-07-28 16:25:22 -07:00
Lianmin Zhengandcctry c19a333944 [mm] Handle per-item embeddings in cache misses (#32498) 
Co-authored-by: cctry <cctry@meta.com>
 2026-07-27 17:16:17 -07:00
Lianmin Zheng 34454c06b8 [Refactor] Tidy server_args.py section grouping and drop unused alias (#32496) 2026-07-27 04:09:09 -07:00
Lianmin Zheng 3863612023 Add oulgen to CI_PERMISSIONS.json (#32453) 2026-07-26 15:04:11 -07:00
Lianmin ZhengandAlec S 9989077f24 Use native batched llguidance mask generation (#32412) 
Co-authored-by: Alec S <10566873+alecsolder@users.noreply.github.com>
 2026-07-25 16:36:32 -07:00
Lianmin ZhengandXingyu Liu fae84ac0f9 Fix token count localization for replicated attention-TP forwards (#32411) 
Co-authored-by: Xingyu Liu <38244988+charlotte12l@users.noreply.github.com>
 2026-07-25 16:36:15 -07:00
Lianmin Zheng 5f330004bd Fix flaky test_sampling_mask: mask length can legitimately be top_k + 1 (#32410) 2026-07-25 16:05:57 -07:00
Lianmin Zheng d3cf4dfbaa Update audio container test time estimate (#32408) 
Signed-off-by: Lianmin Zheng <lianminzheng@gmail.com>
 2026-07-25 14:45:08 -07:00
Lianmin Zheng bd3f6a7935 [CI] Remove redundant Rust cache save-if settings (#32243) 2026-07-23 18:48:54 -07:00
Lianmin Zheng 429f6b6d15 Turn on breakable prefill cuda graph for dp attention by default (#31682) 2026-07-21 00:03:43 -07:00
Lianmin ZhengandJason Park e7e8aaa73c Overlap grammar (constrained decoding) with speculative decode verify (#31488) 
Co-authored-by: Jason Park <jasonjk@fb.com>
 2026-07-20 17:36:05 -07:00
Lianmin Zheng 54aaedd76d Clean up prefill CUDA graph runner (#31654) 2026-07-20 12:14:21 -07:00
Lianmin Zheng 7d64858093 Add houseroad to CI_PERMISSIONS.json (#31717) 2026-07-19 15:08:36 -07:00
Lianmin Zhengandwangwenchen0407 555267ed05 Fix ROCm fused KV and KDA paths (#31688) 
Co-authored-by: wangwenchen0407 <wangwenchen@meta.com>
 2026-07-19 15:02:35 -07:00
Lianmin Zheng c95026aed3 Upgrade llguidance to 1.7.6 (#31484) 2026-07-17 16:31:44 -07:00
Lianmin Zhenglmzhengmichael604work
 		4ae9cc3c81 Fix gate stride for 4D decode layouts (#31231) 
Co-authored-by: lmzheng <lmzheng@fb.com>
Co-authored-by: michael604work <michael604@meta.com>
 2026-07-14 20:06:50 -07:00
Lianmin Zheng b4fdce3b63 Fix post-capture KV sizing for SWA pools (#31092) 2026-07-14 20:06:15 -07:00
Lianmin ZhengandLu Fang 22c08a9bee Preserve RMSNorm shape in batch-invariant mode (#30956) 
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
 2026-07-12 17:51:53 -07:00
Lianmin Zheng b94ac87e0c Enable breakable prefill CUDA graph for DP attention (#30898) 2026-07-12 17:10:04 -07:00
Lianmin ZhengandJialin Ouyang 539253e1d5 Gate Rust extension builds (#30927) 
Co-authored-by: Jialin Ouyang <jialino@meta.com>
 2026-07-12 05:44:58 -07:00
Lianmin ZhengandYongji Wu 81d273f73b Handle coredump dirs and cache hit updates (#30897) 
Co-authored-by: Yongji Wu <30348494+libertyeagle@users.noreply.github.com>
 2026-07-12 00:37:43 -07:00
Lianmin ZhengandAbhishek Parmar 4fcc994be1 Support priority request header override (#30811) 
Co-authored-by: Abhishek Parmar <abhishek@orng.net>
 2026-07-10 14:36:43 -07:00
Lianmin Zheng bc82b06400 Add ZYHowell to CI_PERMISSIONS.json (#30810) 2026-07-10 14:09:32 -07:00
Lianmin Zheng 7998fecfd1 [kv canary] Support UnifiedRadixCache in kv-canary and bracket nested model.forward (#30574) 2026-07-10 10:58:55 -07:00
Lianmin ZhengandYonghao Zhuang 3dc93a12ca Improve EPLB dispatch handling and diagnostics (#30646) 
Co-authored-by: Yonghao Zhuang <yhzhuang@meta.com>
 2026-07-10 10:40:19 -07:00
Lianmin Zhengandtanujtiwari1998 32c8973ce8 Configurable decode retraction order (#30573) 
Co-authored-by: tanujtiwari1998 <168470992+tanujtiwari1998@users.noreply.github.com>
 2026-07-09 17:55:26 -07:00
Lianmin Zhengandtanujtiwari1998 cfc66e05c5 [tokenizer] Support pluggable tokenizer worker class in multi-tokenizer mode (#30630) 
Co-authored-by: tanujtiwari1998 <168470992+tanujtiwari1998@users.noreply.github.com>
 2026-07-09 17:31:54 -07:00
Lianmin Zheng 77b7698cad Add tanujtiwari1998 to CI_PERMISSIONS.json (#30649) 2026-07-09 17:03:46 -07:00
Lianmin Zheng a9e804623e Allow EPLB manual test to use FlashInfer A2A (#30641) 2026-07-09 03:46:07 -07:00
Lianmin Zheng ca8f15cd70 Fix FlashInfer A2A IMA by DP-synchronizing the decode graph bucket (#30242) (#30450) 2026-07-08 14:34:49 -07:00
Lianmin Zheng 24c42c90be Clean up ServerArgs post-init dispatch (#30186) 2026-07-05 23:05:07 -07:00
Lianmin Zheng 9d00385b63 Cleanup: relocate temp_set_env and consolidate multi-device/CUDA helpers in common.py (#30180) 2026-07-05 18:47:26 -07:00
Lianmin Zheng 8673e85e6c Remove # fmt: off from environ.py Envs class (#30153) 2026-07-05 12:45:24 -07:00
Lianmin Zheng 92a1f6e06c [refactor] Reorder ServerArgs sections common-first; inline LLAMA4/MIMO_V2 arch tuples (#30151) 2026-07-05 12:17:24 -07:00
Lianmin Zheng 1a5977d41b [chore] Add no-getattr rule; refine no-dataclasses rule (#29871) 2026-07-01 15:31:34 -07:00
Lianmin Zheng 76d828a4e8 Add pranjalssh to CI_PERMISSIONS.json (#29870) 2026-07-01 14:52:02 -07:00
Lianmin Zheng bb74ed4a8d Replace hasattr with isinstance in SHM feature helpers (#29549) 2026-06-28 21:31:28 -07:00
Lianmin Zheng f76e707f59 Clean up follow-ups for eagle hidden dim clean up (#29546) 2026-06-28 21:30:53 -07:00
Lianmin Zheng 828411e6f1 Fix FlashInfer A2A dispatcher during CUDA graph capture (#29461) 2026-06-28 02:21:26 -07:00
Lianmin Zheng 6eedc8f376 Fix EAGLE draft hidden dim extraction and centralize spec helpers (#29464) 2026-06-27 21:48:26 -07:00
Lianmin ZhengandYinghai Lu 5747ed3b19 Fix DP-attention SHM feature finalization race (#29543) 
Co-authored-by: Yinghai Lu <yinghai@meta.com>
 2026-06-27 21:36:58 -07:00
Lianmin Zheng e6cbc8f5fe Fix SWA cache loc slicing for all attention backends (#29460) 2026-06-27 20:37:58 -07:00
Lianmin Zheng 09ca4fc96b Skip FlashInfer FP8 autotune for MXFP8 quantized models (#29462) 2026-06-26 18:15:08 -07:00
Lianmin Zheng 9c3227b689 Fix IPv6 wildcard bootstrap address resolution in disagg (#29459) 2026-06-26 16:05:36 -07:00
Lianmin ZhengXingyu Liuxingyuliu
 		267d165ad0 shm_broadcast: retry bind on EADDRINUSE (fix dp-attention port race) (#29455) 
Co-authored-by: Xingyu Liu <38244988+charlotte12l@users.noreply.github.com>
Co-authored-by: xingyuliu <xingyuliu@fb.com>
 2026-06-26 14:49:39 -07:00
Lianmin Zheng 118d6b2e5e [Cleanup] Style and type annotation improvements extracted from #28688 (#29224) 2026-06-25 12:51:15 -07:00
Lianmin Zheng 6c839368e0 [Cleanup] IPC struct renames, better typing, and SenderWrapper removal (#29214) 2026-06-24 17:25:24 -07:00
Lianmin ZhengandYinghai Lu 7e63feee6f Add scheduler metrics extension hooks (#29207) 
Co-authored-by: Yinghai Lu <yinghai@meta.com>
 2026-06-24 15:50:05 -07:00
Lianmin ZhengandRain Jiang fd87a85388 Convert SamplingParams to msgspec Struct (#29198) 
Co-authored-by: Rain Jiang <96632942+rainj-me@users.noreply.github.com>
 2026-06-24 13:42:45 -07:00
Lianmin Zheng d5c566e59b Extract profile request cleanups (#29098) 2026-06-24 11:22:58 -07:00
Lianmin ZhengandCheng Wan d6aacd2801 Handle input-embed-only batches in eager runner (#29121) 
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
 2026-06-24 10:33:52 -07:00
Lianmin Zheng 34dd9c28ca [Refactor] Introduce sock_send/sock_recv wrappers for zmq IPC (#29012) 2026-06-23 15:54:36 -07:00
Lianmin Zheng ecab3f322e Revert "Improve MFU metrics for prefill and verify timing" (#29079) 2026-06-23 15:46:22 -07:00
Lianmin ZhengandPranjal Shankhdhar b60185c41c Improve MFU metrics for prefill and verify timing (#29000) 
Co-authored-by: Pranjal Shankhdhar <pranjal.ssh@gmail.com>
 2026-06-23 12:26:56 -07:00
Lianmin Zheng b28e990161 Migrate all ServerArgs fields to Annotated style, reduce add_cli_args by ~2400 lines (#28919) 2026-06-22 08:34:37 -07:00
Lianmin Zheng ad9723af03 Clean up CUDA graph capture logs (#28937) 2026-06-22 06:15:26 -07:00
Lianmin Zheng 886b96621d Migrate more server args to annotated style (#28830) 2026-06-21 20:50:17 -07:00
Lianmin Zhengandhnyls2002 a4d0ff3def [misc] Make NaN-logit sanitization opt-in (default off) (#28829) 
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
 2026-06-21 14:31:35 -07:00
Lianmin Zheng 7942d546d1 Revert "[Spec] Split init_backends; account draft weights in --mem-fraction-static" (#28841) 2026-06-21 07:52:35 -07:00
Lianmin Zheng 3975ea5ac7 Fix H20 torch import reinstall fallback (#28818) 2026-06-21 04:50:46 -07:00
Lianmin Zheng c9488241e9 [Refactor] Auto-derive CLI args from dataclass fields to eliminate duplication (#28814) 2026-06-21 00:51:08 -07:00
Lianmin Zheng d331fdd2ba Add project rule: prefer msgspec.Struct over dataclasses (#28816) 2026-06-20 18:43:16 -07:00
Lianmin Zheng 54b9b9d0c9 Remove threading atexit monkey patch (#28812) 2026-06-20 18:30:50 -07:00
Lianmin Zheng 8a3d6c3403 Sort pyproject dependency lists (#28811) 2026-06-20 17:24:57 -07:00
Lianmin Zheng 95fb1ef697 [CI] Remove deprecated test/srt legacy CI setup (#28810) 2026-06-20 15:09:33 -07:00
Lianmin Zheng fe428dd845 Clean up startup log noise (#28807) 2026-06-20 15:02:52 -07:00
Lianmin ZhengandYinghai Lu 45d203fb08 Fix tokenizer state cleanup on dispatch failure (#28694) 
Co-authored-by: Yinghai Lu <yinghai@meta.com>
 2026-06-19 21:55:39 -07:00
Lianmin Zheng 7cead0fb8f Add JonnyKong to CI_PERMISSIONS.json (#28550) 2026-06-17 11:42:45 -07:00
Lianmin ZhengandIan O'Connell 7e629a2f8c Allow overriding tokenizer path in benchmark harness (#28280) 
Co-authored-by: Ian O'Connell <ianoc@meta.com>
 2026-06-15 13:07:50 -07:00
Lianmin Zhengandgemini-code-assist[bot] 3b419f66da [JIT] Track angle-bracket includes in source hash (#28273) 
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-06-15 02:58:14 -07:00
Lianmin Zheng f18d38d040 Revert "[AMD][Quantization] Online MXFP4 quantization 2/N - FP8 to MXFP4 requantization on AMD GPUs" (#28213) 2026-06-14 13:34:06 -07:00
Lianmin Zheng b001d3e815 Add prajjwal1 to CI_PERMISSIONS.json (#28120) 2026-06-12 20:26:38 -07:00
Lianmin Zheng 165331a200 Share BCG output buffers across capture sizes (#27659) 2026-06-09 20:33:46 -07:00
Lianmin Zheng ca716f4734 Add TP server GPU process regression test (#27721) 2026-06-09 16:25:27 -07:00
Lianmin Zhengandlmzheng eb8dceda44 Defer DeepGEMM PDL setup to worker init (#27671) 
Co-authored-by: lmzheng <lmzheng@fb.com>
 2026-06-09 13:52:30 -07:00
Lianmin Zheng fca4ef9d69 Fix SWA pool resolution for EAGLE draft workers (#27491) 2026-06-08 11:00:29 -07:00
Lianmin Zheng bcb5645629 Fix TRTLLM target verify query metadata (#27495) 2026-06-08 10:58:48 -07:00
Lianmin Zheng a07d813ec8 Revert "Fix TRTLLM target verify query metadata (#27473)" (#27494) 2026-06-07 02:48:52 -07:00
Lianmin ZhengandMing Yang db58e76c33 Add all_to_all_single to GroupCoordinator (#27492) 
Co-authored-by: Ming Yang <minos.future@gmail.com>
 2026-06-07 02:48:47 -07:00
Lianmin Zheng 5be0b0c8c0 Fix TRTLLM target verify query metadata (#27473) 2026-06-07 02:46:01 -07:00
Lianmin Zhengcctrycctry
 		88a7b0fd30 Classify malformed-multimodal rejects as invalid_request (#27451) 
Co-authored-by: cctry <cctry@meta.com>
Co-authored-by: cctry <cctry@fb.com>
 2026-06-06 10:19:22 -07:00
Lianmin Zheng 448d3afb76 fix(spec): complete CustomSpecAlgo duck-typing interface and guard against drift (#27300) 2026-06-04 15:52:46 -07:00
Lianmin Zheng ff5c4d7b57 Add zx3xyy to CI_PERMISSIONS.json (#27213) 2026-06-03 22:12:00 -07:00
Lianmin ZhengHanming LuGrigory Sizov
 		14ed9b448e Add ZMQ IPv6 support, bench_serving sampling params, and reduce routed_dp_rank log noise (#27180) 
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: Grigory Sizov <grisha.sizov@gmail.com>
 2026-06-03 17:49:34 -07:00
Lianmin Zheng 53b8378307 Fix weights_checker checksum for 0-dim tensors and multi-GPU (#26863) 2026-05-31 21:27:03 -07:00
Lianmin ZhengandJaewon a779791b3f Add random-ids dataset, round-robin expert simulation, and kill_process_tree logging (#26862) 
Co-authored-by: Jaewon <52840625+jaewonlee-fb@users.noreply.github.com>
 2026-05-31 20:50:29 -07:00
Lianmin Zhengandcctry 4ff1296f5e Optimize get load calls (/v1/loads) using shared-memory load snapshots (#26348) 
Co-authored-by: cctry <cctry@meta.com>
 2026-05-29 13:40:26 -07:00
Lianmin Zheng dc4e7bc479 Fix TRTLLM MHA draft decode cache seqlens replay (#26655) 2026-05-28 20:58:16 -07:00
Lianmin Zheng 93fa577bb9 Clean up server startup log noise (#26205) 2026-05-24 14:35:15 -07:00
Lianmin ZhengandJaewon 8fa56a0ab1 Fix FlashInfer A2A token cap sizing (#25907) 
Co-authored-by: Jaewon <52840625+jaewonlee-fb@users.noreply.github.com>
 2026-05-20 23:01:28 -07:00
Lianmin Zheng c4a7d12092 Enable breakable CUDA graph for eagle (#25795) 2026-05-20 18:02:04 -07:00