minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 10 Packages Projects Releases Wiki Activity
12,434 Commits 2 Branches 0 Tags
4b23f6bdc50e45b8ae27c8e59bec3f62ea31bd9b
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Brayden Zhongandb8zhong 4b23f6bdc5 Fix performance regression on Deepseek V3 on moe-runner-backend=triton on SM90 (#24562) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
 2026-05-09 03:49:12 -07:00
Brayden Zhongandb8zhong 05d1ab51e8 Enable PDL for various kernels in DSV32/GLM5 (#23965) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
 2026-05-09 03:42:56 -07:00
Brayden Zhongandb8zhong f4b7e73699 Enable trtllm-gen BF16 MoE for MTP (#24260) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
 2026-05-09 03:14:17 -07:00
Brayden Zhongandb8zhong 8f33bee31b Reland Cute-DSL FP4 dense GEMM (#23590) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
 2026-05-09 02:20:58 -07:00
Brayden Zhongandb8zhong 9ee830346f Disable Custom AR V2 when in multi-node (#24729) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
 2026-05-08 17:50:05 -07:00
Brayden Zhongandb8zhong 80d0226b68 Turn on JIT custom AR implementation by default (#24363) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
 2026-05-08 02:05:31 -07:00
Brayden Zhongb8zhonggemini-code-assist[bot]
 		5fa3bb2eaf Enable flashinfer::trtllm_allreduce_fusion with PDL (#23765) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-05-08 10:41:10 +08:00
Brayden Zhongandb8zhong 3fe8bc987e Support Triton MLA FP8 KV cache (#20479) 
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
 2026-05-06 18:32:39 -07:00
Brayden ZhongKangyan ZhouClaude Opus 4.7Baizhou Zhangb8zhongMick
 		88bb5dffe4 [Dependency] Upgrade to Torch 2.11.0 (#21247) 
Co-authored-by: Kangyan Zhou <zky314343421@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
Co-authored-by: b8zhong <b8zhong@users.noreply.github.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
 2026-05-02 12:25:36 -07:00
Brayden Zhong 6aafe756b9 Revert "[Feature] NVFP4 Marlin fallback for non-Blackwell GPUs (SM75+… (#22047) 2026-04-03 13:12:30 -07:00
Brayden Zhong cb0c2cbfdb Enable multi-thread weight loading by default (#20289) 2026-04-01 21:27:20 -07:00
Brayden Zhong 6a9b09847c CUTLASS NVFP4 GEMM improvement of SM120 (#21314) 2026-04-01 09:04:34 +08:00
Brayden Zhong 009eee85a0 CUTLASS FP8 Blockwise GEMM improvement of SM120 (#20887) 2026-03-22 17:55:54 +08:00
Brayden Zhong b42b9f6e1a Support CuteDSL mm_fp4 backend (#18801) 2026-03-19 14:20:01 -07:00
Brayden Zhong 88c40ec16d Use Flashinfer for target_verify in GDN model for SM120 (#20604) 2026-03-17 22:40:56 -07:00
Brayden Zhong 97d5386a21 Use TRTLLM allreduce fusion for Qwen 3.5 (#19889) 2026-03-17 22:40:22 -07:00
Brayden Zhong 591e61245a [Doc] Add smal table for GEMM backends (#20213) 2026-03-09 22:19:57 -07:00
Brayden Zhong e2af840c3d Various SM120 improvements (#19721) 2026-03-03 16:46:13 -08:00
Brayden Zhongandamittell 1388680+amittell@users.noreply.github.com 9305f0e58d Support triton_kernels for GPT-OSS on SM120 (#19718) 
Co-authored-by: amittell 1388680+amittell@users.noreply.github.com
 2026-03-03 14:14:01 -08:00
Brayden ZhongBaidu-AIAKzengpai
 		80a6b32703 [Perf] Optimize NSA backend metadata under MTP (#19536) 
Co-authored-by: Baidu-AIAK <Baidu_AIAK@163.com>
Co-authored-by: zengpai <zengpai@baidu.com>
 2026-03-01 01:59:26 -08:00
b8zhongandCheng Wan 78bf13db44 MoE Refactor: Refactor modelopt_quant.py -> flashinfer_trllm.py (#16685) 
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
 2026-02-02 20:45:14 -08:00
b8zhongandBrayden Zhong 398d13a189 [Perf] Add Flashinfer DeepGEMM SM90 for SwapAB Optimization (#15514) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2026-02-01 08:56:23 +08:00
b8zhong ef134d407d [Fix] Revert back to using CUTLASS mm_fp4 backend (#17369) 2026-01-31 23:01:29 +08:00
b8zhong 22498e10c0 [Fix] Triton TP MoE Dpsk V3/Qwen3 Coder with SwapAB (#17965) 2026-01-31 15:56:26 +08:00
b8zhongandBrayden Zhong 22df62d586 add weightless qk norm to RMSNorm interface for Llama 4 (#12813) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2026-01-29 19:09:55 -08:00
b8zhong 2ff0880a0e [Fix] GLM 4.7 + NVFP4 + MTP (#17166) 2026-01-21 21:34:18 +08:00
b8zhong 7dc3cbe7ca [Docker] Fix CUDA 13 installing wrong nvidia-nccl-cu13 due to nixl-cu13 not breaking system package (#17370) 2026-01-20 09:52:30 +08:00
b8zhong f374623fa9 [Refactor] Set fp4-gemm-backend=auto on SM100 and rename fp4-gemm-backend with flashinfer_ prefix (#17309) 2026-01-19 20:09:07 +08:00
b8zhongandVincent Zhong 4df74eb576 [Refactor] Add -fp4-gemm-backend to replace SGLANG_FLASHINFER_FP4_GEMM_BACKEND (#16534) 
Co-authored-by: Vincent Zhong <207368749+vincentzed@users.noreply.github.com>
 2026-01-18 23:25:46 +08:00
b8zhong d36f6f043c [Fix] flashinfer_trtllm intermediate_size assertion with Qwen3 + TP=8 (#16824) 2026-01-17 09:24:05 +08:00
b8zhong d44f09ad98 [Benchmark] Add GSM8K Platinum Eval (#14565) 2026-01-16 11:06:14 +08:00
b8zhong 3d72944fb8 [Doc] Add tip on how to use Spec V2 (#15455) 2026-01-16 05:30:18 +08:00
b8zhong 4346db5faf [Fix] Remove assertion for padding for NVFP4 weight scales to fix GLM 4.5 NVFP4 (#12497) 2026-01-15 08:57:14 +08:00
b8zhongandBrayden Zhong 24b30f7757 MoE Refactor: Refactor fp8.py -> flashinfer_trllm.py (#15151) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2026-01-07 15:35:00 -08:00
b8zhong 74a166cb86 [Fix] Only add SM90 and SM100 to check for auto-enabling TRT Allreduce Fusion (#16283) 2026-01-03 11:43:17 +08:00
b8zhongandEven Zhou a36142aa71 Fix Mindspore model import warning (#15287) 
Co-authored-by: Even Zhou <even.y.zhou@outlook.com>
 2025-12-18 23:15:33 -05:00
b8zhong e88e75a9df [Deepseek V3.2] Fix Deepseek MTP in V1 mode (#15429) 2025-12-18 17:56:07 -08:00
b8zhong e72b02db28 tiny fix lint on main (#15424) 2025-12-18 15:55:02 -08:00
b8zhongandBrayden Zhong d20699a33c [Deepseek V3.2] Support Overlap Spec + NSA (#15307) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-12-17 13:35:39 -08:00
b8zhong ffa7e03506 [Piecewise CUDA Graph] Support INT8 (#14918) 2025-12-17 18:20:57 +08:00
b8zhong 79ab57bd7a Revert "direct register custom op for mm_fp4 (#13699)" (#15284) 2025-12-16 23:09:44 -08:00
b8zhong 4b8901ac0f Update FP4 GEMM Benchmark (#14449) 2025-12-16 23:04:56 -08:00
b8zhongandBrayden Zhong 1ab9b8e0a3 Enable TRT AllReduce Fusion by default (#14764) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-12-14 20:01:49 -08:00
b8zhongandBrayden Zhong fe6d38d2fa fix: trtllm mha attention auto-selection on sm120 (#14842) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-12-11 22:35:25 -08:00
b8zhongandBrayden Zhong dab31e4cc7 direct register custom op for mm_fp4 (#13699) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-12-11 22:33:17 -08:00
b8zhongandBrayden Zhong 6107268fe7 extend timeout for b200 test (#14925) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-12-11 15:43:53 -08:00
b8zhongandBrayden Zhong c51efb8b84 fix fp8 gemm nightly CI (#14844) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-12-10 15:57:51 -08:00
b8zhong da9b801eb7 fix lora target all + csgmv backend (#14796) 2025-12-10 11:42:51 -08:00
b8zhong 56e5c07424 fix b200 fa4 ci (#14788) 2025-12-10 00:03:43 -08:00
b8zhong b0a25d0913 fix b200 ci (#14786) 2025-12-09 23:08:41 -08:00
b8zhong 55504df2f7 Add FP8 Blockwise GEMM Backend Flag --fp8-gemm-backend (#14379) 2025-12-09 12:05:56 -08:00
b8zhong 0f8bd55f3e [CI] Fix Llama 3.1 8B FP4 CI (#14699) 2025-12-08 22:27:15 -08:00
b8zhong 32f8b6064e improve default glm mtp setting (#14457) 
Signed-off-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-12-08 13:27:13 -08:00
b8zhongandBrayden Zhong 3b47973af8 [CI] Tiny speed up VLM CI (#14517) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-12-07 13:30:41 -08:00
b8zhong 6d5d76ad97 remove unecessary dual stream token threshold from the rest of models (qwen moe, kimi linear, etc.) (#14337) 2025-12-06 19:57:26 -08:00
b8zhong dd91d38e6a [Doc] Add short explanation on page size (#14557) 2025-12-06 17:26:30 -08:00
b8zhong ec7b2c16d9 tiny remove deprecated endpoint call (#13607) 2025-12-05 09:54:49 -08:00
b8zhongBrayden ZhongHo-Ren Chuang
 		88d1bab537 add doc for quantized kv cache (#14348) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
Co-authored-by: Ho-Ren (Jack) Chuang <horenchuang@bytedance.com>
 2025-12-04 13:01:05 -08:00
b8zhong 9d82340298 Revert "Revert "enable csgmv automatically on cuda"" (#14277) 2025-12-03 13:12:30 -08:00
b8zhongandBrayden Zhong 65c8568c4a sync attention, deepseek doc (#14335) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-12-02 21:19:40 -08:00
b8zhongandBrayden Zhong 236a7c2370 fix trtllm mla spec (#13738) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-12-01 22:16:25 -08:00
b8zhongandBrayden Zhong e6420100ee sync attention doc and ep doc to doctree (#14257) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-12-01 21:15:22 -08:00
b8zhongandBrayden Zhong a24aefe562 add some more labels (#13701) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-11-21 11:29:33 -08:00
b8zhongBrayden Zhongluoyuan.luo
 		85ffce30af [Piecewise CUDA Graph] Support Kimi-K2 (non-Thinking) (#13466) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
 2025-11-21 09:39:59 -08:00
b8zhong 42028af614 enable csgmv automatically on cuda (#13600) 2025-11-20 12:53:02 -08:00
b8zhong 92ad2ff9ce Flashinfer TRTLLM-GEN-MoE + Qwen3 (#13489) 2025-11-18 14:18:29 -08:00
b8zhong f33860777c [Piecewise CUDA Graph] Support ModelOpt FP8 (#13094) 2025-11-18 12:46:24 +08:00
b8zhong d5fa58c4dd fix nightly docker build (#13386) 2025-11-16 11:21:09 -08:00
b8zhong f35f7f1245 [Piecewise CUDA Graph] Support W4A8 (#13179) 2025-11-16 11:53:50 +08:00
b8zhong 24a25ffa20 [Piecewise CUDA Graph] Support ModelOpt FP4 (#13101) 2025-11-16 11:03:19 +08:00
b8zhong 7b79cc4fe2 ci: speed up b200 ci (#13237) 2025-11-13 23:57:33 -08:00
b8zhong e523e2167a remove deprecated tile_tokens_dim (#13186) 2025-11-13 17:09:27 -08:00
b8zhong cc2e36c352 overlap shared + routed expert computation in kimi linear (#12660) 2025-11-11 14:52:58 -08:00
b8zhong 49653c8896 use fast stream instead of torch.cuda.current_stream in llama 4 shared experts overlap (#12811) 2025-11-08 15:04:37 -08:00
b8zhongandBrayden Zhong 55e8e3999c add back flashinfer jit cache to dev docker (#12851) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-11-07 14:51:24 -08:00
b8zhong 32ec68faf9 keep attention backend document up to date (#12741) 2025-11-05 20:41:20 -08:00
b8zhong d31d48b341 update usage of trtllm_fp8_per_tensor_scale_moe (#12569) 2025-11-03 14:25:32 -08:00
b8zhong a076ec1a7a Revert "fix llama4 kv cache layout" (#12437) 2025-10-30 22:33:37 -07:00
b8zhongandBrayden Zhong 34c286b811 Fix the shared expert & routed expert overlap in Llama 4 (#12405) 
Co-authored-by: Brayden Zhong <b8zhong@users.noreply.github.com>
 2025-10-30 20:14:49 -07:00
b8zhong 7ed8ba05cb [CI] Add Llama 3.1 8B FP4 to B200 CI (#12182) 2025-10-29 14:56:41 -07:00
b8zhong bacb3825fe fix: llama 4 + trtllm gen + fp8 kv cache incompatibility (#12347) 2025-10-29 11:31:02 -07:00
b8zhong 83a7c89c3f followup fix for llama 4 trtllm flashinfer backend (#12314) 2025-10-28 22:17:08 -07:00
b8zhong c143f416ce fix: Llama 4 BF16 load on Blackwell (#12308) 2025-10-28 18:59:01 -07:00
b8zhong 77225d602a Use Flashinfer TRT-LLM as Llama 4 compatible MoE backend (#11928) 2025-10-28 10:39:43 -07:00
b8zhong f80371ff8c Use flashinfer_trtllm moe runner backend to gain around 10% perf on b200 fp8 dpsk (#11816) 2025-10-23 19:12:15 -07:00
b8zhong 47e12e082e Enable Llama 4 + TRTLLM MHA (#12003) 2025-10-23 18:22:58 -07:00
b8zhong 8ae9d4bb41 Revert "[ROCm] Remove vLLM rope dependency & use AITER impl" (#12028) 2025-10-23 12:42:59 -07:00
b8zhong 4d4feccbb2 [ROCm] Remove vLLM rope dependency & use AITER impl (#11322) 2025-10-22 19:17:34 -07:00
b8zhong d0a64c7e2c vlm: enforce pybase64 for image and str encode/decode (#10700) 2025-10-21 19:05:32 +08:00
b8zhong f4f8a1b4d8 ci: update lmms-eval to speed up multimodal CI (#11000) 2025-10-19 02:51:19 +08:00
b8zhong f9a7d9b3dc support server arg override KV cache to bf16 to avoid slow cases (#11749) 2025-10-19 02:49:48 +08:00
b8zhong 6bc503af73 [Doc] Update support matrix for attn and hybrid attn (#11293) 2025-10-14 22:43:11 -07:00
b8zhong a2faf8940c [1/n] Enable DCA CUDA graph capture (#9537) 2025-10-03 11:30:00 +08:00
b8zhong 948278f173 fix cpp JIT compilation issue of ngram speculative decoding (#10837) 2025-10-02 21:05:01 +08:00
b8zhong b2435be682 Cache the result of is_blackwell platform check (#10498) 2025-09-15 22:30:28 -07:00
b8zhong 716e682721 [Fix] Add undefined update_tensor_inplace function (#6307) 2025-08-18 11:11:00 +08:00
Brayden Zhong bc938ea13f Fix DP load for embedding (#9165) 2025-08-15 23:58:44 -07:00
Brayden Zhong 2871eacc05 Add Triton Fused MoE kernel config for E=16 on B200 (#7004) 2025-08-13 23:12:27 -07:00
Brayden Zhong 4a9f3eef90 Tiny Llama4 type error in constructor (#6752) 2025-08-09 01:03:59 -07:00
Brayden Zhong 4acf690206 [Optimization][Perf] Disable the GC during CUDA graph capture to speed up by up to 3x (#8577) 2025-07-31 11:31:21 -07:00