minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity
16,359 Commits 2 Branches 0 Tags
430f38ea2530453c0940b94da8251adf2d0ea047
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Liangsheng YinandBrayden Zhong b51bf9ec9e [Spec] Budget the DFLASH draft KV pool from its own attention geometry (#34234) 
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
 2026-08-10 01:28:49 -07:00
Liangsheng Yin 3bb72bc72a [CI] Keep the torch compilation cache instead of wiping it on install (#34231) 2026-08-10 01:21:20 -07:00
Liangsheng YinandBrayden Zhong aea78d1e73 [misc] Pass FP8 scales in FlashInfer SWA prefill, autotune fp8 on SM120, and tighten is_image_understandable_model (#34217) 
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
 2026-08-09 23:45:11 -07:00
Liangsheng YinandBrayden Zhong d6a066131c [Feature] Support NVFP4 token embedding in ModelOpt mixed-precision checkpoints (#34222) 
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
 2026-08-09 23:37:54 -07:00
Liangsheng Yin 7c90840bad [CI] Key scheduled CUDA suites by runner_config instead of hand-written jobs (#34186) 2026-08-09 16:44:53 -07:00
Liangsheng YinandDeleter-D 4a5d7d3c67 [DSV4] Fix silent KV corruption when speculative draft tokens > 4 (#34189) 
Co-authored-by: Deleter-D <867909454@qq.com>
 2026-08-09 15:54:41 -07:00
Liangsheng Yin 1ebd6fab6c [CI] Gate Kimi-K3 acceptance length on the GSM8K average (#34145) 2026-08-08 20:28:42 -07:00
Liangsheng Yin 4792ab1e90 [CI] Pin the rust frontend parity test to eager prefill (#34146) 2026-08-08 20:07:19 -07:00
Liangsheng Yin a1ca76b24b [Scheduler] Unify WAR read-done gating behind shared-read boundary declarations (#34052) 2026-08-08 03:26:36 -07:00
Liangsheng Yin 2c0188cc78 [Fix] Give the piecewise CUDA graph test stub an hf_config (#34100) 2026-08-08 02:04:16 -07:00
Liangsheng YinandBaizhou Zhang f6a6f5bf1e [CI] Trim redundant nightly test registrations (#34070) 
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
 2026-08-08 01:42:46 -07:00
Liangsheng Yin c59d2b4329 [Fix] Judge the phase-checker device-assert test by its FAIL line, not the exit code (#34017) 2026-08-07 16:19:05 -07:00
Liangsheng Yingithub-actions[bot]Harmya BhattharmyaXinyuanXinyuan TongXinyuan Tong
 		3c51e29deb Responses support (#32689) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Harmya Bhatt <harmyacs@gmail.com>
Co-authored-by: harmya <harmya@modal.com>
Co-authored-by: Xinyuan <xinyuan@radixark.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
 2026-08-07 13:21:46 -07:00
Liangsheng Yin 7395ee833e [CI] Share VLM engines and prune launch matrices on the per-commit H100/H200 suites (#33944) 2026-08-07 01:22:22 -07:00
Liangsheng Yin afa79330b8 [misc] Remove break-graph debug log; reclaim pid-less /dev/shm leaks in CI (#33929) 2026-08-06 21:18:01 -07:00
Liangsheng Yin 163b739b34 [CI] Refresh the CPU HF cache base only on main-ref runs (#33904) 2026-08-06 19:37:59 -07:00
Liangsheng Yin efc99a86ff [CI] Restore the full prefill CUDA graph capture range in test launches (#33847) 2026-08-06 01:40:10 -07:00
Liangsheng Yin 9bd1461757 [CI] Bound the CUDA graph capture range in test launches and lift the spec fixture's admission cap (#33776) 2026-08-05 20:00:30 -07:00
Liangsheng Yin 3869fe556f [CI] Collapse the EAGLE launch matrix and the scoring engine boots on the per-commit runners (#33756) 2026-08-05 16:43:19 -07:00
Liangsheng Yin c0ef548eef [misc] Unify MLA scaling init and remove dead buffer / scaling code (#33363) 2026-08-05 15:59:33 -07:00
Liangsheng Yin 2d27133fcf [CI] Skip apt-get when the required packages are already installed (#33757) 2026-08-05 15:06:01 -07:00
Liangsheng Yin de34dd11e9 [CI] Fold duplicate-server suites and prune the retract matrix on 1-gpu-5090 (#33745) 2026-08-05 12:41:51 -07:00
Liangsheng Yin 1a045669e4 [CI] Merge tokenizer worker tests and drop redundant triton attention e2e (#33641) 2026-08-05 11:55:11 -07:00
Liangsheng Yin 4a3d6ca88c [CI] Skip sglang-kernel and sgl-deep-gemm reinstall on version match (#33637) 2026-08-05 01:46:16 -07:00
Liangsheng Yin c0d5ebd6c4 [CI] Move CPU-only unit tests to the CPU suite and trim dead 5090 registrations (#33654) 2026-08-05 01:43:29 -07:00
Liangsheng Yin eac1f78568 [CI] Free hosted-runner disk space only when it is low (#33644) 2026-08-04 21:55:06 -07:00
Liangsheng Yin 198a3bc29b [Test] Route GEMM backend UTs through real layer modules and weight loaders (#33615) 2026-08-04 20:53:26 -07:00
Liangsheng Yin 1033cae8d5 [CI] Speed up dependency install: dual-ABI Rust ext cache and prevalidation pruning (#33619) 2026-08-04 20:33:48 -07:00
Liangsheng Yin 76dc89f5aa [Test] Replace NVFP4 MoE runner backend e2e matrix with a layer-level unit test (#33611) 2026-08-04 16:03:29 -07:00
Liangsheng Yin 0d99d91e49 [CI] Make B200 base-b suites single-GPU as prep for 1-gpu B200 runners (#33605) 2026-08-04 15:53:01 -07:00
Liangsheng Yin a0b3f1dde6 [Test] Replace GEMM backend e2e matrices with layer-level unit tests (#33596) 2026-08-04 15:50:41 -07:00
Liangsheng Yin 58da9859c4 [CI] Extract download-rust-ext and give every install step a cache fallback (#33597) 2026-08-04 15:33:34 -07:00
+26
Liangsheng YinDarkSharpnessXiaoyu ZhangMickYuhao YangCheng WanKe BaoBaizhou ZhangChunan ZengKhoa PhamZiyi XuZijie XiaYuwei AnzhangxiaohaoYangmin LiJulien LinHao PhanThomas WangRolaoDenthupigeonsoupHaiShawXinyuan TongPranjal ShankhdharLee NauHMINGelvischenvByron HsuByron HsuClaude Opus 5Thomas WangXinyi SongMohammad Miadh AngkadCheng WanBBufHanming LuXinyi Song
 		abddb1c7e9 [Kimi] Support kimi-k3 (#32541) 
Co-authored-by: DarkSharpness <76582120+DarkSharpness@users.noreply.github.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
Co-authored-by: Yuhao Yang <47235274+yhyang201@users.noreply.github.com>
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
Co-authored-by: Ke Bao <ispobaoke@gmail.com>
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
Co-authored-by: Chunan Zeng <zcnrex@gmail.com>
Co-authored-by: Khoa Pham <khoa.pham@radixark.ai>
Co-authored-by: Ziyi Xu <ziyi.xu@radixark.ai>
Co-authored-by: Zijie Xia <37504505+zijiexia@users.noreply.github.com>
Co-authored-by: Yuwei An <ayw.sirius19@gmail.com>
Co-authored-by: zhangxiaohao <1024393531@qq.com>
Co-authored-by: Yangmin Li <yangminl@nvidia.com>
Co-authored-by: Julien Lin <jullin@nvidia.com>
Co-authored-by: Hao Phan <htphan@nvidia.com>
Co-authored-by: Thomas Wang <1am9trash@gmail.com>
Co-authored-by: RolaoDenthu <xinyisong0111@gmail.com>
Co-authored-by: pigeonsoup <32922982+pigeonsoup@users.noreply.github.com>
Co-authored-by: HaiShaw <hixiao@gmail.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
Co-authored-by: Pranjal Shankhdhar <pranjal.ssh@gmail.com>
Co-authored-by: Lee Nau <lee.nau@gmail.com>
Co-authored-by: HMING <126185151+Hearum@users.noreply.github.com>
Co-authored-by: elvischenv <219235043+elvischenv@users.noreply.github.com>
Co-authored-by: Byron Hsu <byronhsu1230@gmail.com>
Co-authored-by: Byron Hsu <byron+per@periodiclabs.ai>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: Xinyi Song <86638975+RolaoDenthu@users.noreply.github.com>
Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
Co-authored-by: BBuf <xiaoyu.zhang@radixark.ai>
Co-authored-by: Hanming Lu <hanminglu@meta.com>
Co-authored-by: Xinyi Song <xinyis10@illinois.edu>
 2026-08-04 13:22:49 -07:00
Liangsheng Yin 0753663b8e [CI] Trim redundant B200 test registrations (#33586) 2026-08-04 13:22:00 -07:00
Liangsheng Yin b6d548afd7 [Fix] Resolve VLM test image placeholders from the model's own chat template (#33509) 2026-08-04 02:01:39 -07:00
Liangsheng Yin 157401f050 [CI] Build the Rust extensions on the 5090 pool and seed the cache from main (#33460) 2026-08-03 23:33:20 -07:00
Liangsheng YinandMick afc868517b [Perf] Speed up the Kimi-K2.5 vision path and match PIL bicubic in the GPU resize (#33349) 
Co-authored-by: Mick <mickjagger19@icloud.com>
 2026-08-03 21:57:49 -07:00
Liangsheng Yin 7ba393dd15 [CI] Dispatch base-a-test-cpu through its own reusable stage workflow (#33461) 2026-08-03 20:49:23 -07:00
Liangsheng Yin cdff33d738 [CI] Build the Rust extension modules once per run instead of in every CUDA job (#33384) 2026-08-03 18:56:48 -07:00
Liangsheng Yin c949e91f18 [CI] Remove the orphaned site-packages sglang skeleton that shadows the checkout (#33441) 2026-08-03 16:31:25 -07:00
Liangsheng Yin 22c2e2bcad [CI] Persist the cargo build cache across CUDA CI jobs (#33361) 2026-08-03 16:23:09 -07:00
Liangsheng Yin b8f6181bff [CI] Build the Rust extensions with the pinned toolchain instead of the image default (#33437) 2026-08-03 16:16:34 -07:00
Liangsheng Yin 45c00daa1b [misc] Deep-merge nested config overrides and parse request bodies with orjson (#33351) 2026-08-02 23:55:06 -07:00
Liangsheng Yin fcc4de9e5f [CI] Size the CPU stage from the live partition model (#33329) 2026-08-02 22:26:58 -07:00
Liangsheng Yin f5f021672a [Fix] Treat an empty grammar constraint as unset in SamplingParams (#33328) 2026-08-02 21:45:51 -07:00
Liangsheng YinandMick dd6ddc053b [Fix] Drop deprecated multimodal processor residency state (#33308) 
Co-authored-by: Mick <mickjagger19@icloud.com>
 2026-08-02 20:02:53 -07:00
Liangsheng YinandMick 4bc593fdc8 [Perf] Broadcast single-image DP vision embedding instead of pad-to-max all-gather (#33307) 
Co-authored-by: Mick <mickjagger19@icloud.com>
 2026-08-02 19:39:53 -07:00
Liangsheng Yin 2a7a299c27 [Spec] Support sampling in the DSPARK graph-folded draft proposal (#33298) 2026-08-02 18:46:39 -07:00
Liangsheng Yin 558c9bdcc2 [misc] Improve benchmark determinism and dataset API coverage (#33255) 2026-08-02 01:39:50 -07:00
Liangsheng Yin ca07917c58 [Fix] Bound FULL_MASK verify-mask reuse by the captured max_bs (#33127) 2026-07-31 17:50:40 -07:00
Liangsheng YinandKaixi Matteo Chen 5c6635d8f3 [Spec] Compact the target-verify mask when nothing reads it (#32920) 
Co-authored-by: Kaixi Matteo Chen <kaiximatteoc@nvidia.com>
 2026-07-30 23:13:21 -07:00
Liangsheng Yin 9f56553408 [Perf] Fast-path chain-style draft token organization in multi-layer EAGLE (#32887) 2026-07-30 02:55:21 -07:00
Liangsheng YinandKaixi 6ab3231b97 [Perf] Skip the target-verify tree mask fill when the backend never reads it (#32886) 
Co-authored-by: Kaixi <kaiximatteoc@nvidia.com>
 2026-07-30 02:32:38 -07:00
Liangsheng Yin c192145830 [Kernel] Fuse KV-cache writes for asymmetric K/V (head_dim != v_head_dim) (#32813) 2026-07-30 00:26:10 -07:00
Liangsheng Yin 2625fdfe6b [Fix] Count multi-layer draft-extend replays in the fwd-occupancy device timer (#32867) 2026-07-30 00:21:34 -07:00
Liangsheng Yin f4e0ac382e [misc] Remove unused multi_layer_draft_forward_cg module (#32881) 2026-07-29 21:17:16 -07:00
Liangsheng Yin 313a518bee [Spec] Emit step trace span for multi-layer draft-extend graph replays (#32850) 2026-07-29 19:22:58 -07:00
Liangsheng Yin e5c46ff07d [Fix] Route asymmetric-KV models to fa4 on SM100 and pin MiMoV2 FP8 MoE to flashinfer_trtllm (#32818) 2026-07-29 16:37:43 -07:00
Liangsheng Yin cce5873513 [CI] Fail lint when a registered file's TestCase classes never run (#32735) 2026-07-29 00:45:01 -07:00
Liangsheng Yin c4fc241fd3 [Perf] Free KV pages by segment in the paged allocator without a device sync (#32701) 2026-07-28 22:24:16 -07:00
Liangsheng Yin 14bd315d6e [Refactor] Remove dead allocator backup_state / restore_state (#32709) 2026-07-28 19:49:47 -07:00
Liangsheng Yin 85618cc798 docs: update sglang cookbook (#32654) 2026-07-28 15:33:24 -07:00
Liangsheng Yinkpham-sglZijie XiaispobockMickBaizhou ZhangthomawanBBuf
 		7dafacca49 docs(cookbook): add the Kimi-K3 serving cookbook (#32542) 
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai>
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
Co-authored-by: ispobock <ispobaoke@gmail.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
Co-authored-by: thomawan <thomawan@amd.com>
Co-authored-by: BBuf <1182563586@qq.com>
 2026-07-27 08:37:30 -07:00
Liangsheng Yin 3da1071d56 [Spec] Hold the grammar bitmask in one GrammarMask type across all decode paths (#32409) 2026-07-25 15:27:43 -07:00
Liangsheng Yin 17afd8421f [Spec] Share the grammar mask build and verify-tree staging across spec workers (#32393) 2026-07-25 03:40:21 -07:00
Liangsheng Yin 3c5bf1f6d2 [Spec] Derive NGRAM grammar tree links on the host instead of reading back retrive_next_token (#32380) 2026-07-25 02:25:16 -07:00
Liangsheng Yin cff20a2fbb [Spec] Consolidate the grammar sync decision into ScheduleBatch.grammar_needs_sync (#32353) 2026-07-24 20:42:38 -07:00
Liangsheng Yin a31542ebd9 [Feature] Add leveled invariant-check primitive for nan/inf/oob validity checks (#32308) 2026-07-24 04:27:20 -07:00
Liangsheng Yin f4f15162bc [Fix] Fail fast when a safetensors index references missing shard files (#32279) 2026-07-24 02:18:56 -07:00
Liangsheng Yin d059b0f56e [Fix] Clamp degenerate all-sentinel draft rows to token 0 in dspark _online_combine_kernel (#32277) 2026-07-24 00:53:56 -07:00
Liangsheng Yin 99b29bf188 [Fix] Support ENCODER_ONLY target-verify in the trtllm_mha backend (#32178) 2026-07-23 20:33:39 -07:00
Liangsheng Yin 59ef3b15cc [Fix] Reserve the mamba pool's +1 padding slot in the memory budget solve (#32184) 2026-07-23 14:55:35 -07:00
Liangsheng Yin f35411ee81 [Spec] Enable grammar overlap scheduling for STANDALONE speculative decoding (#32110) 2026-07-23 01:47:27 -07:00
Liangsheng Yin 9b853e6832 [Scheduler] Enable decode retraction ordering under speculative decoding (#32023) 2026-07-23 00:42:56 -07:00
Liangsheng Yin 1b63155efe [Perf] Skip blocks past per-request live length in full-width Triton kernels (#32109) 2026-07-22 22:00:25 -07:00
Liangsheng Yin 98cc8d91cf [Fix] Evict only the KV shortfall in evict_from_tree_cache (#32016) 2026-07-22 12:41:06 -07:00
Liangsheng Yin 5c6a29b3c3 [Fix] Unify pinned host pool release on graceful shutdown (#32029) 2026-07-22 12:29:34 -07:00
Liangsheng Yin 024639a372 [Perf] Stack dspark dense draft per-layer ctx KV projection into one GEMM (#31986) 2026-07-21 17:46:13 -07:00
Liangsheng Yin 8ae0eb83fc [Perf] Fold dspark dense draft embedding into the draft graph via forward_embed (#31985) 2026-07-21 17:00:42 -07:00
Liangsheng Yin 08cb081ed0 [Perf] Skip page-table columns past kv length in DSA draft-extend metadata kernel (#31981) 2026-07-21 16:20:14 -07:00
Liangsheng Yin cb324cc958 Add wangfakang to CI_PERMISSIONS.json (#31980) 2026-07-21 14:03:00 -07:00
Liangsheng Yin 7fe9ad25ac [Spec] Extract DFlash compact draft-cache rebuild helpers (#31677) 2026-07-20 15:07:55 -07:00
Liangsheng Yin ff6c755952 [Refactor] Unify logprob results into a single LogprobResult and rename chunk env vars (#31733) 2026-07-20 12:44:27 -07:00
Liangsheng Yin b15a83983c [Fix] Release hierarchical cache host pool on graceful shutdown (#31746) 2026-07-19 22:27:29 -07:00
Liangsheng Yin b3570a4531 [Refactor] Unify input logprob processing on a single chunked path (#31655) 2026-07-19 16:03:23 -07:00
Liangsheng Yin cce5fe7696 [Scheduler] Move the WAR barrier to right after each run_batch launch (#31687) 2026-07-18 20:38:34 -07:00
Liangsheng Yin 10908a6793 [Fix] Respect cache_protected_len in ChunkCache and disabled-radix release paths (#31662) 2026-07-18 12:04:25 -07:00
Liangsheng Yin 639261f7b2 [Refactor] Move output logprob processing into the logprob_processor layer (#31624) 2026-07-17 22:33:52 -07:00
Liangsheng Yin 19c53c44a0 [Fix] Account zero-logprob sequences correctly in chunked logprob stitching (#31639) 2026-07-17 22:33:27 -07:00
Liangsheng Yin 2c856abbe3 [Fix] Enable chunked input-logprob processing by default to cap peak memory (#31498) 2026-07-17 13:19:03 -07:00
Liangsheng Yin 19f4859b30 [CI] Exclude current process memory from GPU idle check (#31571) 2026-07-17 02:11:41 -07:00
Liangsheng Yin 6e3be088a9 [Spec] Allocate the verify tree-mask scratch on the target backend only (#31527) 2026-07-16 23:23:21 -07:00
Liangsheng Yin 45824c69ca fa3: build the topk>1 verify replay page table on-device (#31381) 2026-07-16 21:21:01 -07:00
Liangsheng Yin 27ad9d11b1 [CI] Wait for GPU memory release before each test class setUpClass (#31509) 2026-07-16 21:06:29 -07:00
Liangsheng Yin dfa6278370 [Fix] Publish idle scheduler metrics immediately when the running-reqs gauge is stale (#31517) 2026-07-16 21:04:10 -07:00
Liangsheng Yin 9910ef8167 flashmla: sync-free spec via device-side draft-extend (#31090) 2026-07-16 19:38:12 -07:00
Liangsheng Yin 640101da13 [Fix] Deflake score engine determinism tests (#31494) 2026-07-16 15:42:47 -07:00
Liangsheng Yin 9a4d640244 [Perf] Cache uniform ragged-verify layout for DSpark verify-all compact (#31434) 2026-07-16 15:00:41 -07:00
Liangsheng Yin fc1e3797b7 [Spec] Split the capture width from num_tokens_per_req and gate replay on it (#31255) 2026-07-16 14:59:11 -07:00
Liangsheng Yin 44f4ea917c fa3: sync-free eagle spec via fixed-window draft-extend metadata (#31364) 2026-07-16 12:31:50 -07:00