minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity
12,794 Commits 3 Branches 0 Tags
b463740953171494d0a80ae0bd63a2f8a54747fa
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Cheng WanandClaude Opus 4.7 7158a255eb [MoE Refactor] Migrate flashinfer_cutedsl + DeepEP to MoeRunner (#25525) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-17 14:48:17 -07:00
Cheng WanandClaude Opus 4.7 be3c425788 [MoE] Unify DeepEPMoE+MoriEPMoE through AITER MoeRunner pre/post-permute (#23760) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-17 02:25:52 -07:00
Cheng Wan 568ba7216a Fix logging for inplace setting in the flashInfer-trtllm backend (#25522) 2026-05-17 02:20:14 -07:00
Cheng Wan 229cadec04 Update logging for inplace setting in MoE layer (#25499) 2026-05-16 17:12:11 -07:00
Cheng Wan 0c017db916 Update kl_div_thres to 0.02 in swa_radix_cache (#25497) 2026-05-16 16:43:12 -07:00
Cheng WanandClaude Opus 4.7 54221dd998 feat(moe): reuse prev-layer output as symm_output for FP4 routed MoE (#25379) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-15 12:05:40 -07:00
Cheng WanandClaude Opus 4.7 ee93795476 perf(mla): hybrid Triton fused cat+FP8-quantize for MLA chunked-prefill K/V (#25333) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-15 10:51:00 -07:00
Cheng Wan 3f7e538b2f Add NPU condition for cosine and sine caching (#25399) 2026-05-15 05:21:53 -07:00
Cheng WanandClaude Opus 4.7 ad4994dc1d DeepseekV2MoE: defer shared experts when routed kernel is non-mutating (#25279) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-14 22:20:55 -07:00
Cheng Wan dca9ba6321 perf(mla): TMA bulk-store set_mla_kv_buffer (up to 12× over baseline) (#25311) 2026-05-14 18:23:41 -07:00
Cheng WanandClaude Opus 4.7 1913cb4dbb Skip CI tests added in #24816 (broken on main) (#25329) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-14 18:12:13 -07:00
Cheng WanandClaude Opus 4.7 6c0633b0b1 fix(nvfp4): make process_weights_after_loading hot-reload-safe via alias-when-same-shape (#25190) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 16:57:39 -07:00
Cheng WanandClaude Opus 4.7 72b49bfac6 docker, ci: swap GB DeepEP source from fzyzcjy fork to deepseek-ai/DeepEP@hybrid-ep (#25113) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 13:25:15 -07:00
Cheng WanandClaude Opus 4.7 f2a90094c9 bench: fix wrong flag names in bench_one_batch{,_server} docstrings (#25143) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-12 23:46:47 -07:00
Cheng WanandClaude Opus 4.7 4fb40bffac perf(nvfp4): free unused source scales after weight processing (#25107) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-12 16:53:50 -07:00
Cheng WanandClaude Opus 4.7 044bb88a97 fix: SGLANG_RADIX_FORCE_MISS chunk-cache passthrough (#24950) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-11 00:07:46 -07:00
Cheng WanCheng WanClaude Opus 4.7
 		c7f674e427 [Bug] Add dsv4 state_type branch to mooncake disaggregation (#24878) 
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-10 01:13:46 -07:00
Cheng Wansihan-zzzClaude Opus 4.7
 		d1c5937428 env: add SGLANG_RADIX_FORCE_MISS to force radix prefix-cache miss (#24726) 
Co-authored-by: sihan-zzz <228612289+sihan-zzz@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-08 17:46:38 -07:00
Cheng Wan 7b707c9222 disable the combination of --enable-two-batch-overlap and --enforce-s… (#24720) 2026-05-08 14:27:35 -07:00
Cheng Wan b47fab6f5d [bugfix] Support MIXED forward mode in TBO splitter for DP attention (#24241) 2026-05-01 16:01:23 -07:00
Cheng Wan d48095ba53 Bypass torch.cuda.use_mem_pool generator-CM in SymmetricMemoryContext (#24190) 2026-05-01 01:25:49 -07:00
Cheng WanandClaude Opus 4.7 108bfd8b6a [MoE] Add Aiter MoE runner backend and purge aiter.fused_moe from quant methods (#23597) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-30 19:50:52 -07:00
Cheng WanandClaude Opus 4.7 c7878dbb6d [MoE] Deprecate act_and_mul_triton; fold filter_expert into JIT silu/gelu_and_mul (#23707) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-26 01:41:35 -07:00
Cheng Wan d9c72bdd2b Skip unselected experts in flashinfer_trtllm (#23493) 2026-04-23 17:30:19 -07:00
Cheng WanandClaude Opus 4.7 000a2525e1 Move expert_mask_gpu from FusedMoE layer to StandardDispatcher (#23585) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 17:17:27 -07:00
Cheng WanandClaude Opus 4.7 ebcc2b3eec ci: run weekly est_time update on Monday using p90 of last 15 runs (#23120) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-19 14:39:27 -07:00
Cheng WanandClaude Opus 4.7 5f7aee726a refactor(moe): de-duplicate triton MoE runner path into shared helpers (#23019) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-17 17:05:13 -07:00
Cheng Wan b5e4ae7b1a fix: match est_time updates by backend, not just suite (#22563) 2026-04-10 17:54:50 -07:00
Cheng Wan 0011d2aec0 fix: track est_time per suite instead of per backend (#22557) 2026-04-10 16:58:40 -07:00
Cheng WanandClaude Opus 4.6 3f39b3d811 feat: add weekly workflow to update CI test est_time values (#22545) 
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-10 15:03:37 -07:00
Cheng WanandClaude Opus 4.6 6d95602ea3 Reduce GPU memory for MoE parallel groups (#22515) 
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-10 13:23:23 -07:00
Cheng WanandClaude Opus 4.6 37107bee6f [Observability] Add pending token count to prefill log and get_load (#22480) 
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-10 02:05:21 -07:00
Cheng Wan c01ee848b0 Revert "fix: use consistent time denominator for throughput metrics in bench_one_batch_server" (#21276) 2026-03-23 22:14:54 -07:00
Cheng Wan d6b12c401c Revert "[bugfix] Fix PPMissingLayer AttributeError when Using PP" (#21189) 2026-03-22 23:28:36 -07:00
Cheng Wan 6e54361608 Refactor CUDA graph input buffers with shared buffer pool (#19180) 2026-02-24 02:24:40 -08:00
Cheng Wan 84c67c8be0 Refactor graph input buffers (#18991) 2026-02-20 18:09:31 -08:00
Cheng Wan 38ee749dd9 Fix adjust_num_token_non_padded_for_attn_tp returning CPU tensor (#19051) 2026-02-20 23:23:38 +08:00
Cheng Wan b59a22f781 fix lint on main (#19052) 2026-02-20 15:30:57 +08:00
Cheng Wan 13a4a0406e Fix flashinfer autotune to only wrap run_once() (#19004) 2026-02-19 20:02:21 -08:00
Cheng Wan 64bca5315f Fix long prompt KV allocation by falling back to torch native APIs when exceeding Triton tensor limit (#18250) 2026-02-19 19:15:05 -08:00
Cheng Wan 73a7f0d049 Revert "Add SDAR model support" (#19032) 2026-02-19 16:03:56 -08:00
Cheng Wan 84c09913eb Moving _alloc_extend_naive out of npu allocator (#18200) 2026-02-04 02:09:55 -08:00
Cheng Wan ab8b99eb23 Refine logprob logic for request handling (#17986) 2026-02-01 19:11:52 -08:00
Cheng Wan 6a6b36367e Fix logprob_start_len handling for prefill-only requests (#17395) 2026-01-29 15:14:43 -08:00
Cheng Wan a416af4be7 Fix capture_sizes range for pcg (#17956) 2026-01-29 12:46:35 -08:00
Cheng Wan 2b461c15b4 Update logprob_start_len handling in scheduler (#16240) 2025-12-31 14:11:24 -08:00
Cheng Wan 5fb734f1a5 Enhance comments in set_extend_input_len method (#16130) 2025-12-29 20:47:33 -08:00
Cheng Wan 60f1ca6925 Refactor: Moving extend_logprob_start_len calculation out of prepare_for_extend (#16105) 2025-12-30 12:38:33 +08:00
Cheng Wan c2e0913e17 Fix extend_input_len calculation in decode.py (#16103) 2025-12-29 13:13:24 -08:00
Cheng Wan 6f9d0a89a0 [scheduler] fix: correcting extend_logprob_start_len calculation (#15922) 2025-12-28 14:57:04 -08:00
Cheng Wan c457aad54a Update test parameters for deepep_large test (#16001) 2025-12-28 00:58:19 -08:00
Cheng Wan 2ec57cefd9 hotfix: add type hints to scheduler mixins (#15916) 2025-12-26 17:08:11 -08:00
Cheng Wan 988b14ca0e refactor: add type hints to scheduler mixins (#15913) 2025-12-26 16:50:07 -08:00
Cheng Wan 5290cef97c Fix condition check for require_gathered_buffer (#15328) 2025-12-17 12:42:47 -08:00
Cheng Wan 96cc10834a [CI] update estimated elapsed time of some unittests (#14347) 2025-12-03 01:21:40 -08:00
Cheng WanandKangyan Zhou 0fe74af563 Remove incorrect deep_gemm assertions from server_args.py (#14113) 
Co-authored-by: Kangyan Zhou <zky314343421@gmail.com>
 2025-11-28 20:25:39 -08:00
Cheng Wan b12c9e5c0a Fix installation for nvidia-nvshmem-cu12 (#14033) 2025-11-26 18:27:12 -08:00
Cheng Wan bf10869203 [Doc] Add an Introduction to Expert Parallelism (#13783) 2025-11-24 14:46:51 -08:00
Cheng WanandBaizhou Zhang 6d0e0b9bfc [11/N] MoE Refactor: Simplifying SBO Implementation with Dispatcher Hooks (#13327) 
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
 2025-11-21 01:11:37 -08:00
Cheng WanandMick a5ad0069b2 fix: change performance log directory to cache path (#13482) 
Co-authored-by: Mick <mickjagger19@icloud.com>
 2025-11-18 15:18:43 +08:00
Cheng Wan 2b8b9d8496 [CI] use cached deepep installation in gb200 CI (#13388) 2025-11-16 12:49:01 -08:00
Cheng Wan 13366843a4 [CI] check unit-test-backend-8-gpu-h20 in workflow (#13355) 2025-11-15 16:44:10 -08:00
Cheng Wan 83087247d1 [hotfix] missing w13_weight_fp8 and w2_weight_fp8 in UE8M0 requantization (#12259) 2025-10-28 19:10:38 -07:00
Cheng Wan b1e13e7cea [hotfix] Incorrect CombineOverlapArgs in SBO (#12230) 2025-10-28 01:23:06 -07:00
Cheng Wan 649949807f [10/N] MoE Refactor: reorganize deepgemm runner in DeepEPMoE (#12054) 2025-10-24 19:16:17 -07:00
Cheng Wan bfc3b3f786 [9/N] MoE Refactor: cleanup dispatcher interfaces (#11847) 2025-10-20 10:11:46 -07:00
Cheng Wan 5b214b50b6 [Refactor] move deep_gemm_wrapper out of quantization (#11784) 2025-10-17 18:57:54 -07:00
Cheng Wan 1bdd010291 Revert "Deprecate global_server_args_dict" (#11520) 2025-10-12 17:40:40 -07:00
Cheng Wan 6cd296940a [lint] Fix the lint issue (#11516) 2025-10-12 16:22:46 -07:00
Cheng Wan 52fcbbb8bd Revert "perf: optimize qwen-vl with symm mem allreduce" (#11436) 2025-10-10 12:30:05 -07:00
Cheng Wan 97cd38e58d Skip weight loading in deepgemm compilation (#11312) 2025-10-07 21:52:46 -07:00
Cheng Wan 3c06b673af [8/N] MoE Refactor: deprecate EPMoE (#11211) 2025-10-07 21:51:41 -07:00
Cheng Wan 41763ba079 Remove gdrcopy check in ci_install_deepep.sh (#11237) 2025-10-05 17:35:22 -07:00
Cheng Wan 6535fda127 [Profile] dump memory trace when cuda graph profile is enabled (#11083) 2025-09-29 17:36:48 -07:00
Cheng Wan 4844fac91d Refactor TopK to ensure readability and extensibility (#9338) 2025-09-14 19:16:25 -07:00
Cheng Wan 2f8ba6fe82 [Fix] MoE: fix w8a8_fp8 MoE and add tests to cover this code path (#10429) 2025-09-14 17:34:28 -07:00
Cheng Wan 5a7e10fe4c [MoE] fix: incorrect weight initialization for cutlass_fused_experts_fp8 (#10144) 2025-09-07 19:43:59 -07:00
Cheng Wan a5a03209e9 Fix circular import (#10107) 2025-09-06 01:34:17 -07:00
Cheng Wanandgemini-code-assist[bot] 21af5c0404 [Fix] Compatibility between DP attention and pipeline parallelism (#10100) 
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-09-06 01:34:10 -07:00
Cheng Wan 3fa62da78c [7/N] MoE Refactor: the implementation of new framework (#9269) 2025-09-05 21:09:09 -07:00
Cheng Wan 453511acc7 Save memory for expert model parallel (#9957) 2025-09-04 13:31:47 -07:00
Cheng Wan 84b006b278 Cleanup MoE Refactor (#9223) 2025-08-15 02:28:33 -07:00
Cheng Wan e3e75a786a Fix the deprecation warning for enable_flashinfer_mxfp4_moe (#9214) 2025-08-14 23:59:35 -07:00
Cheng Wan 295895120d [6/N] MoE Refactor: Cleanup MoE-related configs (#8849) 2025-08-14 21:14:53 -07:00
Cheng Wan b87aacb5c5 [DP Attention] Refactor: adding some utility functions (#9136) 2025-08-13 21:08:06 -07:00
Cheng Wan 5f5b3b2449 [5/n] DP Enhancement: Correct num_token_non_padded (#9107) 2025-08-12 12:23:46 -07:00
Cheng Wanandgemini-code-assist[bot] f003cd3548 [CI] Fix CI tests (#9050) 
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-08-10 23:52:05 -07:00
Cheng Wan 5018809222 [DP] fix: engine crash when decode batch is padded (#8995) 2025-08-09 01:29:29 -07:00
Cheng Wan a47baff12c [hotfix] use the original implementation in 8785 (#8994) 2025-08-08 21:47:25 -07:00
Cheng Wan fd7e15b76d Revert "[bug fix] Ensure local token and global token buffers are pointing to different storage " (#8993) 2025-08-08 21:34:17 -07:00
Cheng Wan 1d24db8348 Expert Parallelism for GPT-OSS (#8944) 2025-08-08 00:46:42 -07:00
Cheng Wan 5b6acc1495 fix glm4 moe (#8883) 2025-08-06 18:02:31 -07:00
Cheng Wan 78aad91037 [CI] fix pip upgrade (#8881) 2025-08-06 15:02:32 -07:00
Cheng Wan b102353f8f [MoE] Enable renormalize=False in Triton kernels (#8735) 2025-08-03 17:03:04 -07:00
Cheng Wan 0e0eef00ce [DP] fix the compatibility issue between DP attention and --attention-backend triton (#8723) 2025-08-03 03:06:57 -07:00
Cheng Wan cb099d2095 [CUDA Graph] save cuda graph memory by using next_token_logits_buffer (#8579) 2025-08-03 03:06:47 -07:00
Cheng Wan 7a91330149 Save cuda graph memory for fa3 (#8567) 2025-08-03 03:06:31 -07:00
Cheng Wan a437aa9987 [hotfix] fix mixtral with tensor-level compressed-tensor quantization (#8721) 2025-08-02 22:59:25 -07:00
Cheng Wan 6c88f6c8d9 [5/N] MoE Refactor: Update MoE parallelism arguments (#8658) 2025-08-01 01:20:03 -07:00
Cheng Wan 7a1f7fc504 [Feature] Hybrid EP and TP (#8590) 2025-07-31 02:53:25 -07:00