minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 10 Packages Projects Releases Wiki Activity
14,567 Commits 2 Branches 0 Tags
9214b9338fcb1df5b1a9ab5835aa631539f55556
Commit Graph
62 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Khoa Phamkpham-sglJackey Hua
 		9214b9338f [DSA] Enable draft-extend CUDA graph for DeepSeek Sparse Attention (#29413) 
Co-authored-by: kpham-sgl <kpham-sgl@users.noreply.github.com>
Co-authored-by: Jackey Hua <107608053+zhendonghua@users.noreply.github.com>
 2026-06-26 23:53:46 -07:00
Khoa PhamCursorClaude Opus 4.8
 		999199f9ff [DeepSeek V3] Run routed experts on main stream in dual-stream MoE (#29142) 
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-26 00:48:59 -07:00
Khoa PhamandCursor eeee3abbbf [CI] Fix false spec accept length failure after profiling phase (#29329) 
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-06-25 22:19:30 -07:00
Khoa PhamandClaude Opus 4.7 6c92f9f328 fix(bench): pass DCP_RANK/DCP_WORLD_SIZE to set_mla_kv_buffer_kernel (#29356) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-06-25 18:13:57 -07:00
Khoa PhamandClaude Opus 4.7 c8c6757bed test(dp-attn): drop --enable-torch-compile from TestDPAttentionDP2TP2 (#29345) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-06-25 16:41:35 -07:00
Khoa PhamClaude Opus 4.8Xinyuan Tong
 		ed26a109ee [Fix] Return streaming logprobs when reasoning/tool parser is active (#28601) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
 2026-06-23 09:56:09 -07:00
Khoa PhamandCursor 0642cd5020 (chore): bump tokenspeed_mla to 0.1.7 (#28759) 
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-06-21 21:25:37 -07:00
Khoa PhamandClaude Opus 4.7 e4fd613def [Spec] Fix return_hidden_states under spec V2 (issue #26163) (#28496) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-06-17 16:21:36 -07:00
Khoa Pham 3c4130c741 [Kimi K2.5] Fix eagle3 aux capture for tp>1 when AR fusion is enabled (#28343) 2026-06-17 12:59:11 -07:00
Khoa PhamCursorQiaolin Yu
 		b54f8432ad Batch EAGLE draft/draft-extend replay memcpys via grouped foreach copy (#28465) 
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Qiaolin Yu <liin1211@outlook.com>
 2026-06-17 00:34:21 -07:00
Khoa Pham a0c6e0b3a4 chore: bump tokenspeed_mla 0.1.1 -> 0.1.6 (#28116) 2026-06-12 19:57:36 -07:00
Khoa PhamandQiaolin Yu d1a39b0c74 [DeepSeek V3] Defer moe finalize and fused it with main stream add (#27720) 
Co-authored-by: Qiaolin Yu <liin1211@outlook.com>
 2026-06-12 19:00:51 -07:00
Khoa PhamandMohammad Miadh Angkad 82eedd5bd0 [DeepEP] Enable fabric handles automatically when supported (#27107) 
Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
 2026-06-12 17:17:44 -07:00
Khoa PhamRichard WangClaude Opus 4.8
 		1cf8efdd08 docs: Diffusion Gemma cookbook (#27824) 
Co-authored-by: Richard Wang <wangrichard08@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-10 09:33:01 -07:00
Khoa PhamandCursor 9a3e845fc1 [Spec] Add nvtx to spec regions (#27615) 
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-06-08 22:58:03 -07:00
Khoa Pham 0f8673851c test: fix gemma GSM8K thresholds in nightly text eval (#27342) 2026-06-08 19:58:42 -07:00
Khoa Pham c95179bc85 [Spec] Fuse small kenrels under gather_spec_extras (#27233) 2026-06-08 15:02:04 -07:00
Khoa PhamandClaude Opus 4.8 bf172c492a Cookbook for QAT (#27396) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-05 11:18:11 -07:00
Khoa Pham b5560ffc36 Fix flashinfer autotune oom glm51 (#24195) 2026-06-02 23:28:57 -07:00
Khoa PhamandClaude Opus 4.8 6d53615699 [Gemma4] Use hard GSM8K accuracy floor for 31B MTP test (#27101) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-03 01:57:12 -04:00
Khoa Pham a711c57a32 [Spec] Fix Gemma 4 MTP with trtllm_mha crash issue (#26966) 2026-06-02 14:37:51 -07:00
Khoa Pham 22bb9a6421 test: disable test_gemma4_mtp_26b_a4b_extra from CI (#27082) 2026-06-02 14:00:48 -07:00
Khoa PhamandCursor 08526c7fca [Spec] FrozenKVMTP fold assistant seed into captured draft graph (#25539) 
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-06-01 22:27:25 -07:00
Khoa Pham da01f2974e [Log] include max_token_num and hidden_dim in FlashInfer workspace init log (#26605) 2026-06-01 13:26:52 -07:00
Khoa PhamandClaude Opus 4.8 cb8a103b81 chore: add @pyc96 as codeowner for FrozenKVMTP module (#26953) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-01 13:06:58 -07:00
Khoa PhamandCursor 621a79728c test: stabilize Gemma4 26B-A4B MTP GSM8K test with deterministic inference + tuned threshold (#26653) 
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-05-28 20:58:48 -07:00
Khoa PhamandCursor 93445e6359 [spec decoding] support kimi-k2.6-eagle3.1-mla draft (#26506) 
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-05-28 14:26:39 -07:00
Khoa Pham a38620b92c chore: add @pyc96 as codeowner for gemma4 files (#26601) 2026-05-28 12:37:50 -07:00
Khoa PhamandClaude Opus 4.7 9040feebd8 [Gemma4] Add test for MTP models (#24552) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-27 21:36:28 -07:00
Khoa PhamandClaude Opus 4.7 b0ce16d0c5 [CP] 1/N: Support MLA Prefill Context Parallel (#23292) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-23 03:07:09 -07:00
Khoa PhamandClaude Opus 4.7 f829cafa3e [perf] skip add_special_tokens=False kwarg on chat-template tokenize for slow tokenizers (#25953) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 22:45:54 -07:00
Khoa PhamandClaude Opus 4.7 b29e41e8b3 Respect user override for Gemma4 attention backend (#25547) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-18 10:46:25 -07:00
Khoa PhamandClaude Opus 4.7 856f13a916 chore(codeowners): add kpham-sgl to frozen_kv_mtp files (#25211) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 20:01:05 -07:00
Khoa PhamandClaude Opus 4.7 371cb2ade2 [Bench] Add MEM profile activity to bench_serving (#25026) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 16:22:11 -07:00
Khoa Pham c665edec6e [env] Make max KV chunk capacity configurable via SGLANG_MAX_KV_CHUNK_CAPACITY (#25120) 2026-05-12 22:37:45 -07:00
Khoa Pham 49ac447c94 [bench] Agentic support for bench_serving.py (#25016) 2026-05-12 16:00:45 -07:00
Khoa Pham 1610aa77ab Reduce gemma4 moe deterministic test runtime (#24754) 2026-05-08 20:46:56 -07:00
Khoa PhamandPengyu Chen d2c1034163 [Gemma 4] Adding MTP support (#24436) 
Co-authored-by: Pengyu Chen <pychen96@gmail.com>
 2026-05-07 14:08:41 -07:00
Khoa Pham 932d89690a Gemma4-mtp cookbook (#24433) 2026-05-05 23:04:41 +08:00
Khoa Pham d22853480d Fix deterministic inference on models with SWAKVPool (#24395) 2026-05-05 20:20:46 +08:00
Khoa Pham ef2b1b6d89 Fix flashinfer workspace OOM (#24172) 2026-05-04 01:26:35 -07:00
Khoa PhamandClaude Opus 4.7 c2d60c3667 chore(codeowners): add @kpham-sgl as codeowner for ngram files (#24349) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-03 23:27:33 -07:00
Khoa Pham ddcacaf1bd Fix failing test_nvidia_nemotron_3_nano by fixing test_grouped_topk (#23874) 2026-04-28 15:03:58 -07:00
Khoa PhamandClaude Opus 4.7 826f2d0620 chore(codeowners): add @kpham-sgl as owner for gemma4 files (#23916) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-28 11:43:32 -07:00
Khoa Pham 5a0eea8ac5 [CI] Adding Gemma 4 to Nightly CI (#22408) 2026-04-16 19:30:16 -07:00
Khoa PhamandClaude Opus 4.6 1f8df97054 Fix broken streaming response with --incremental-streaming-output (#22549) 
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-12 15:05:58 -07:00
Khoa Pham aeeff58cd4 [Spec][Ngram] Clean up unused stateless batchMatch (#22487) 2026-04-10 21:52:56 -07:00
Khoa PhamandClaude Opus 4.6 04bd8e1218 [Spec][Ngram] Return token counts in list_external_corpora API (#22471) 
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-10 21:50:02 -07:00
Khoa Pham f127d67823 [Spec][Ngram] Misc enhance support for multiple SAMs (#22294) 2026-04-08 19:56:23 -07:00
Khoa Pham 12272b6791 [Spec][Ngram] 6/N: Load an external corpus and construct a Suffix Automaton (#21425) 2026-04-06 00:11:14 -07:00
Khoa Pham b2008bf9e0 [Spec][Ngram] 5/N: Store and advance anchor match state across decode steps (#21243) 2026-04-05 22:21:05 -07:00
Khoa PhamandClaude Opus 4.6 cd75d54fc5 [Bugfix] Fix CUDA graph replay issues in trtllm_mla draft_extend (#21987) 
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-03 01:45:13 -07:00
Khoa Pham 2b5aed94f5 Remove maxItems=1 restriction when tool_choice is specified (#20208) 2026-04-03 02:35:24 +00:00
Khoa Pham f836658077 [Spec][Ngram] 4/N: Remove max_match_window_size and min_match_window_size, matching all suffixes of the Trie (#21225) 2026-04-01 22:09:46 -07:00
Khoa Pham 153359b4dd Multi tool streaming fix (#20004) 2026-04-01 21:53:05 -07:00
Khoa Pham 8d4fca5908 [Security] 1/N: Bind ZMQ sockets to localhost to prevent unauthenticated remote access (#21435) 2026-03-26 23:33:49 -07:00
kpham-sglandClaude Opus 4.6 59cb9a9da6 [Spec][Ngram] 3/N: Fix synchronization issues in Ngram.cpp (#21186) 
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-23 00:37:07 -07:00
kpham-sglandClaude Opus 4.6 bc4aaab6a1 [Spec][Ngram] 2/N: Rename branch length to max trie depth (#21181) 
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-22 23:35:25 -07:00
kpham-sgl 6d160b42bb [Spec][Ngram] 1/N: Reference based Speculative Decoding refactor (#20393) 2026-03-22 00:55:10 -07:00
kpham-sglandhnyls2002 346a4131cf [Spec] Refactor NaN/OOB checks to async maybe_detect_* with env-var control (#19899) 
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
 2026-03-05 13:51:05 -08:00
kpham-sgl 376dfb03f7 Fix issue 19717 by making qo_indptr uniform strided instead of packed (#19807) 2026-03-04 15:27:10 -08:00
kpham-sgl 88290690f0 Add kpham-sgl into CI Permission list (#19819) 2026-03-03 18:55:10 -08:00