minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 10 Packages Projects Releases Wiki Activity
14,599 Commits 2 Branches 0 Tags
b9b860652e36e2ab70f776bd851b933dbb9b502b
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Brayden ZhongandBrayden Zhong b9b860652e Fix DSA indexer fusion bug causing excessive memory consumption. (#29576) 
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
 2026-06-28 14:11:07 -07:00
Brayden ZhongBrayden ZhongKaixi
 		073de15053 Fuse the DSA (V3.2, GLM-5.x) indexer Q/K paths into single kernels (#27705) 
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
Co-authored-by: Kaixi <kaiximatteoc@nvidia.com>
 2026-06-27 14:29:56 -07:00
Brayden ZhongandBrayden Zhong e4253b39e2 Support JIT fused A GEMM (MLA down projection) and support GLM-5 hidden size, SM120 (#27397) 
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
 2026-06-27 13:32:06 -07:00
Brayden ZhongandBrayden Zhong 9495737d82 Fix CI broken by #28450 (#29308) 
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
 2026-06-25 10:35:34 -07:00
Brayden ZhongandBrayden Zhong 5d4e63d49e Sync the changes in #23402 (#29063) 
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
 2026-06-24 17:01:02 -07:00
Brayden ZhongandBrayden Zhong f82addd4a8 Support online MXFP8 quantization for ungated MoE (#27939) 
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
 2026-06-24 16:58:48 -07:00
Brayden ZhongandBrayden Zhong 3c95a87b66 Fix the CuDNN failure on bmm_fp8 when two libcudart.so exists. (#29201) 
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
 2026-06-24 15:26:46 -07:00
Brayden ZhongandBrayden Zhong d46afbf8b4 Fix nightly CI test for GLM-4.6 + B200 (#28749) 
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
 2026-06-24 12:34:11 -07:00
Brayden ZhongandBrayden Zhong 2c697daf5f [Cookbook] Nemotron3-Ultra: align MTP draft depth with NVIDIA reference (num_steps 5) (#29200) 
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
 2026-06-24 19:12:34 +00:00
Brayden Zhong ba9d5aed98 Fix nightly CI test for Kimi K2.5 INT4 + H200 (#28746) 2026-06-23 00:27:33 +00:00
Brayden ZhongBrayden Zhongzijiexia
 		99c18cceec Sync server arguments and environment variables + update various documentation (#28674) 
Co-authored-by: Brayden Zhong <brayden.zhong@radixark.ai>
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
 2026-06-22 10:23:46 -07:00
Brayden ZhongandBrayden Zhong 34e5e38604 [Cookbook] Nemotron3-Ultra: Add mamba-backend and SSM dtype flags (#28675) 
Co-authored-by: Brayden Zhong <brayden.zhong@radixark.ai>
 2026-06-22 09:50:22 -07:00
Brayden ZhongandBrayden Zhong 05ee93c44f Remove redundant cast and copy in calling trtllm_fp8_block_scale_moe (#28555) 
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
 2026-06-18 18:11:29 -07:00
Brayden ZhongandBrayden Zhong ea407df4b0 Use Flashinfer allreduce fusion for MNNVL allreduce for Nemotron (#28346) 
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
 2026-06-18 17:30:12 -07:00
Brayden ZhongandBrayden Zhong b8a73bfba0 Call Flashinfer mm_fp8 for per-tensor FP8 GEMMs on SM100 (#28333) 
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
 2026-06-16 20:50:03 -07:00
Brayden ZhongBrayden Zhongshaunkotek
 		13537f8e20 Unskip Marlin NVFP4 tests (#27589) 
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
Co-authored-by: shaunkotek <shaunkotek@users.noreply.github.com>
 2026-06-16 11:58:22 -07:00
Brayden ZhongandBrayden Zhong 25e696aa8d Fix Stage B CUDA CI (#28367) 
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
 2026-06-16 02:39:17 -07:00
Brayden ZhongandBrayden Zhong 1a19f66acb Fix DP attention + EP mode of Nemotron (#28102) 
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
 2026-06-12 18:39:04 -07:00
Brayden ZhongandBrayden Zhong 95867f0932 [Doc] Fix some inconsistencies in the Nemotron Cookbook (#28087) 
Co-authored-by: Brayden Zhong <brayden.zhong@radixark.ai>
 2026-06-12 14:51:58 -07:00
Brayden ZhongandBrayden Zhong 8bfcc0c39c Use the correct wrapper for fp4_quantize (#27956) 
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
 2026-06-11 20:23:09 -07:00
Brayden ZhongandBrayden Zhong 493f828bfa Add DeepGEMM prerelease wheel tests (#27075) 
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
 2026-06-11 15:14:50 -07:00
Brayden ZhongandBrayden Zhong 38ae22e08c Nemotron perf changes (#26733) 
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
 2026-06-05 22:31:46 -07:00
Brayden ZhongandBrayden Zhong d381ec7997 [CI] Fix Nemotron nightly mixed precision checkpoints test (#27284) 
Co-authored-by: Brayden Zhong <brayden.zhong@radixark.ai>
 2026-06-05 13:51:26 -07:00
Brayden ZhongandBrayden Zhong 3b62286fca Reland "Support NextN = 2/4 in DSV32" (#27166) 
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
 2026-06-05 13:43:28 -07:00
Brayden ZhongMartin VitClaude Opus 4.7Brayden Zhong
 		e76d36214b Changes for SM120 perf and usability for NVFP4 (#26496) 
Co-authored-by: Martin Vit <martin@voipmonitor.org>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Brayden Zhong <brayden.zhong@radixark.ai>
 2026-06-04 15:29:25 -07:00
Brayden Zhongb8zhongBrayden Zhong
 		5bf90ad988 Enable DeepGEMM PDL on by default (#23979) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
 2026-06-04 14:13:45 -07:00
Brayden Zhongandb8zhong 13852d3f31 Support NextN = 2/4 in DSV32 (#24870) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
 2026-06-02 19:06:06 -07:00
Brayden Zhongandb8zhong 50e0b3b77f Support Flashinfer Cute-DSL MLA attention (#24737) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
 2026-05-28 00:21:32 -07:00
Brayden Zhongandb8zhong e31ea50df8 Remove DeepGEMM for indexer GEMM in piecewise NSA path (#26494) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
 2026-05-28 00:15:23 -07:00
Brayden Zhongandb8zhong b4808d44da Use Cute-DSL MXFP8 quantize kernels (#25486) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
 2026-05-28 00:01:31 -07:00
Brayden Zhong 97dd6aad60 Add a little env var for disabling Flashinfer autotune cache (#26193) 2026-05-27 23:59:59 -07:00
Brayden Zhongandb8zhong d523ae127f Fix Mistral Large 3 nightly test (#25407) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
 2026-05-15 17:28:35 -07:00
Brayden Zhongandb8zhong d5f3254ed1 [Dependency] Flashinfer 0.6.8post1 -> 0.6.11 (#24452) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
 2026-05-12 14:38:32 -07:00
Brayden Zhongandb8zhong 1df9edcd01 Use Torch torch.mm for Deepseek V3.2 Indexer GEMM (#23856) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
 2026-05-11 00:41:05 -07:00
Brayden Zhongandb8zhong 1d80a1a9fe Use Cute-DSL NVFP4 quantization kernels (#23745) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
 2026-05-11 00:40:02 -07:00
Brayden Zhongandb8zhong 8acb0270fd Followup fix for Custom AR V2 in non NVL scenarios (#24742) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
 2026-05-10 16:57:46 -07:00
Brayden Zhongandb8zhong 4b23f6bdc5 Fix performance regression on Deepseek V3 on moe-runner-backend=triton on SM90 (#24562) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
 2026-05-09 03:49:12 -07:00
Brayden Zhongandb8zhong 05d1ab51e8 Enable PDL for various kernels in DSV32/GLM5 (#23965) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
 2026-05-09 03:42:56 -07:00
Brayden Zhongandb8zhong f4b7e73699 Enable trtllm-gen BF16 MoE for MTP (#24260) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
 2026-05-09 03:14:17 -07:00
Brayden Zhongandb8zhong 8f33bee31b Reland Cute-DSL FP4 dense GEMM (#23590) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
 2026-05-09 02:20:58 -07:00
Brayden Zhongandb8zhong 9ee830346f Disable Custom AR V2 when in multi-node (#24729) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
 2026-05-08 17:50:05 -07:00
Brayden Zhongandb8zhong 80d0226b68 Turn on JIT custom AR implementation by default (#24363) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
 2026-05-08 02:05:31 -07:00
Brayden Zhongb8zhonggemini-code-assist[bot]
 		5fa3bb2eaf Enable flashinfer::trtllm_allreduce_fusion with PDL (#23765) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-05-08 10:41:10 +08:00
Brayden Zhongandb8zhong 3fe8bc987e Support Triton MLA FP8 KV cache (#20479) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
 2026-05-06 18:32:39 -07:00
Brayden ZhongKangyan ZhouClaude Opus 4.7Baizhou Zhangb8zhongMick
 		88bb5dffe4 [Dependency] Upgrade to Torch 2.11.0 (#21247) 
Co-authored-by: Kangyan Zhou <zky314343421@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
 2026-05-02 12:25:36 -07:00
Brayden Zhong 6aafe756b9 Revert "[Feature] NVFP4 Marlin fallback for non-Blackwell GPUs (SM75+… (#22047) 2026-04-03 13:12:30 -07:00
Brayden Zhong cb0c2cbfdb Enable multi-thread weight loading by default (#20289) 2026-04-01 21:27:20 -07:00
Brayden Zhong 6a9b09847c CUTLASS NVFP4 GEMM improvement of SM120 (#21314) 2026-04-01 09:04:34 +08:00
Brayden Zhong 009eee85a0 CUTLASS FP8 Blockwise GEMM improvement of SM120 (#20887) 2026-03-22 17:55:54 +08:00
Brayden Zhong b42b9f6e1a Support CuteDSL mm_fp4 backend (#18801) 2026-03-19 14:20:01 -07:00
Brayden Zhong 88c40ec16d Use Flashinfer for target_verify in GDN model for SM120 (#20604) 2026-03-17 22:40:56 -07:00
Brayden Zhong 97d5386a21 Use TRTLLM allreduce fusion for Qwen 3.5 (#19889) 2026-03-17 22:40:22 -07:00
Brayden Zhong 591e61245a [Doc] Add smal table for GEMM backends (#20213) 2026-03-09 22:19:57 -07:00
Brayden Zhong e2af840c3d Various SM120 improvements (#19721) 2026-03-03 16:46:13 -08:00
Brayden Zhongandamittell 1388680+amittell@users.noreply.github.com 9305f0e58d Support triton_kernels for GPT-OSS on SM120 (#19718) 
Co-authored-by: amittell 1388680+amittell@users.noreply.github.com
 2026-03-03 14:14:01 -08:00
Brayden ZhongBaidu-AIAKzengpai
 		80a6b32703 [Perf] Optimize NSA backend metadata under MTP (#19536) 
Co-authored-by: Baidu-AIAK <Baidu_AIAK@163.com>
Co-authored-by: zengpai <zengpai@baidu.com>
 2026-03-01 01:59:26 -08:00
b8zhongandCheng Wan 78bf13db44 MoE Refactor: Refactor modelopt_quant.py -> flashinfer_trllm.py (#16685) 
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
 2026-02-02 20:45:14 -08:00
b8zhongandBrayden Zhong 398d13a189 [Perf] Add Flashinfer DeepGEMM SM90 for SwapAB Optimization (#15514) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2026-02-01 08:56:23 +08:00
b8zhong ef134d407d [Fix] Revert back to using CUTLASS mm_fp4 backend (#17369) 2026-01-31 23:01:29 +08:00
b8zhong 22498e10c0 [Fix] Triton TP MoE Dpsk V3/Qwen3 Coder with SwapAB (#17965) 2026-01-31 15:56:26 +08:00
b8zhongandBrayden Zhong 22df62d586 add weightless qk norm to RMSNorm interface for Llama 4 (#12813) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2026-01-29 19:09:55 -08:00
b8zhong 2ff0880a0e [Fix] GLM 4.7 + NVFP4 + MTP (#17166) 2026-01-21 21:34:18 +08:00
b8zhong 7dc3cbe7ca [Docker] Fix CUDA 13 installing wrong nvidia-nccl-cu13 due to nixl-cu13 not breaking system package (#17370) 2026-01-20 09:52:30 +08:00
b8zhong f374623fa9 [Refactor] Set fp4-gemm-backend=auto on SM100 and rename fp4-gemm-backend with flashinfer_ prefix (#17309) 2026-01-19 20:09:07 +08:00
b8zhongandVincent Zhong 4df74eb576 [Refactor] Add -fp4-gemm-backend to replace SGLANG_FLASHINFER_FP4_GEMM_BACKEND (#16534) 
Co-authored-by: Vincent Zhong <207368749+vincentzed@users.noreply.github.com>
 2026-01-18 23:25:46 +08:00
b8zhong d36f6f043c [Fix] flashinfer_trtllm intermediate_size assertion with Qwen3 + TP=8 (#16824) 2026-01-17 09:24:05 +08:00
b8zhong d44f09ad98 [Benchmark] Add GSM8K Platinum Eval (#14565) 2026-01-16 11:06:14 +08:00
b8zhong 3d72944fb8 [Doc] Add tip on how to use Spec V2 (#15455) 2026-01-16 05:30:18 +08:00
b8zhong 4346db5faf [Fix] Remove assertion for padding for NVFP4 weight scales to fix GLM 4.5 NVFP4 (#12497) 2026-01-15 08:57:14 +08:00
b8zhongandBrayden Zhong 24b30f7757 MoE Refactor: Refactor fp8.py -> flashinfer_trllm.py (#15151) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2026-01-07 15:35:00 -08:00
b8zhong 74a166cb86 [Fix] Only add SM90 and SM100 to check for auto-enabling TRT Allreduce Fusion (#16283) 2026-01-03 11:43:17 +08:00
b8zhongandEven Zhou a36142aa71 Fix Mindspore model import warning (#15287) 
Co-authored-by: Even Zhou <even.y.zhou@outlook.com>
 2025-12-18 23:15:33 -05:00
b8zhong e88e75a9df [Deepseek V3.2] Fix Deepseek MTP in V1 mode (#15429) 2025-12-18 17:56:07 -08:00
b8zhong e72b02db28 tiny fix lint on main (#15424) 2025-12-18 15:55:02 -08:00
b8zhongandBrayden Zhong d20699a33c [Deepseek V3.2] Support Overlap Spec + NSA (#15307) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-12-17 13:35:39 -08:00
b8zhong ffa7e03506 [Piecewise CUDA Graph] Support INT8 (#14918) 2025-12-17 18:20:57 +08:00
b8zhong 79ab57bd7a Revert "direct register custom op for mm_fp4 (#13699)" (#15284) 2025-12-16 23:09:44 -08:00
b8zhong 4b8901ac0f Update FP4 GEMM Benchmark (#14449) 2025-12-16 23:04:56 -08:00
b8zhongandBrayden Zhong 1ab9b8e0a3 Enable TRT AllReduce Fusion by default (#14764) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-12-14 20:01:49 -08:00
b8zhongandBrayden Zhong fe6d38d2fa fix: trtllm mha attention auto-selection on sm120 (#14842) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-12-11 22:35:25 -08:00
b8zhongandBrayden Zhong dab31e4cc7 direct register custom op for mm_fp4 (#13699) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-12-11 22:33:17 -08:00
b8zhongandBrayden Zhong 6107268fe7 extend timeout for b200 test (#14925) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-12-11 15:43:53 -08:00
b8zhongandBrayden Zhong c51efb8b84 fix fp8 gemm nightly CI (#14844) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-12-10 15:57:51 -08:00
b8zhong da9b801eb7 fix lora target all + csgmv backend (#14796) 2025-12-10 11:42:51 -08:00
b8zhong 56e5c07424 fix b200 fa4 ci (#14788) 2025-12-10 00:03:43 -08:00
b8zhong b0a25d0913 fix b200 ci (#14786) 2025-12-09 23:08:41 -08:00
b8zhong 55504df2f7 Add FP8 Blockwise GEMM Backend Flag --fp8-gemm-backend (#14379) 2025-12-09 12:05:56 -08:00
b8zhong 0f8bd55f3e [CI] Fix Llama 3.1 8B FP4 CI (#14699) 2025-12-08 22:27:15 -08:00
b8zhong 32f8b6064e improve default glm mtp setting (#14457) 
Signed-off-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-12-08 13:27:13 -08:00
b8zhongandBrayden Zhong 3b47973af8 [CI] Tiny speed up VLM CI (#14517) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-12-07 13:30:41 -08:00
b8zhong 6d5d76ad97 remove unecessary dual stream token threshold from the rest of models (qwen moe, kimi linear, etc.) (#14337) 2025-12-06 19:57:26 -08:00
b8zhong dd91d38e6a [Doc] Add short explanation on page size (#14557) 2025-12-06 17:26:30 -08:00
b8zhong ec7b2c16d9 tiny remove deprecated endpoint call (#13607) 2025-12-05 09:54:49 -08:00
b8zhongBrayden ZhongHo-Ren Chuang
 		88d1bab537 add doc for quantized kv cache (#14348) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
Co-authored-by: Ho-Ren (Jack) Chuang <horenchuang@bytedance.com>
 2025-12-04 13:01:05 -08:00
b8zhong 9d82340298 Revert "Revert "enable csgmv automatically on cuda"" (#14277) 2025-12-03 13:12:30 -08:00
b8zhongandBrayden Zhong 65c8568c4a sync attention, deepseek doc (#14335) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-12-02 21:19:40 -08:00
b8zhongandBrayden Zhong 236a7c2370 fix trtllm mla spec (#13738) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-12-01 22:16:25 -08:00
b8zhongandBrayden Zhong e6420100ee sync attention doc and ep doc to doctree (#14257) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-12-01 21:15:22 -08:00
b8zhongandBrayden Zhong a24aefe562 add some more labels (#13701) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-11-21 11:29:33 -08:00
b8zhongBrayden Zhongluoyuan.luo
 		85ffce30af [Piecewise CUDA Graph] Support Kimi-K2 (non-Thinking) (#13466) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
 2025-11-21 09:39:59 -08:00