minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity
13,680 Commits 3 Branches 0 Tags
db58e76c334548416265c9c7a55b8ff6bf21b8f8
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Cheng WanandClaude Opus 4.8 9097647090 Route the eager forward path through the CUDA graph input-buffer registry (#27407) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-06 14:35:53 -07:00
Cheng WanandClaude Opus 4.8 7dc7376697 fix(attn): delegate init_mha_chunk_metadata in HybridLinearAttnBackend (#27316) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-04 17:44:23 -07:00
Cheng WanandClaude Opus 4.8 0aa72a9e76 Replace skip_attn_backend_init with a batch-carried attention plan marker (+ staleness re-plan) (#27193) 
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-04 17:13:18 -07:00
Cheng WanandClaude Opus 4.8 10ab7c919f [refactor] Retire DecodeInputBuffers / PrefillInputBuffers in favor of CudaGraphBufferRegistry (#27192) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-03 20:52:56 -07:00
Cheng WanandClaude Opus 4.8 c9ca56da8c Unify full→SWA index translation in init_forward_metadata; drop pool caches (#27091) 
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-03 16:12:27 -07:00
Cheng Wan 61aa3293d3 Revert "Fix TokenizerManager crash on top_logprobs with tensor values" (#27187) 2026-06-03 14:53:28 -07:00
Cheng WanandClaude Opus 4.8 e485ad6ac1 Fix hybrid linear attention dispatch by layer id with draft-worker awareness (#27120) 
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-03 14:11:14 -07:00
Cheng WanandClaude Opus 4.8 45604a0f4a [refactor] Unify CUDA graph runner input buffers behind CudaGraphBufferRegistry (#26742) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-03 10:54:10 -07:00
Cheng Wan 73b53e7a87 Revert "Support NextN = 2/4 in DSV32" (#27138) 2026-06-03 01:29:52 -07:00
Cheng Wan 202e618898 Revert "Fix hybrid linear attention misrouting plain-RadixAttention linear layers to the full backend (Ring-2.5-1T)" (#27116) 2026-06-02 23:27:57 -07:00
Cheng WanandClaude Opus 4.7 99da43b900 [refactor] init_forward_metadata 3-method ABC + side-channel removal + ForwardMetadata type rename (#26735) 
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-02 10:33:33 -07:00
Cheng WanandClaude Sonnet 4.6 ff8ed7a302 [refactor] unify cuda-graph capture/replay across attention backends (#26665) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-29 12:46:42 -07:00
Cheng WanandClaude Sonnet 4.6 ec075d8bc5 Fix DRAFT_EXTEND_V2 CG metadata: align test fixture and Triton with production seq_lens convention (#26651) 
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-29 02:46:45 -07:00
Cheng WanandClaude Sonnet 4.6 0fb0ea7aac test: add trtllm_mha EAGLE-draft CG runner coverage (chain) (#26669) 
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-28 23:46:06 -07:00
Cheng WanandClaude Sonnet 4.6 2dfbc3d781 test: strengthen CG-replay coverage with prod-fill padding, metadata invariants, and pad-ratio sweep (#26658) 
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-28 22:43:29 -07:00
Cheng Wan 84698b21e7 rename unittest as unittests (#26654) 2026-05-28 21:14:06 -07:00
Cheng WanandClaude Sonnet 4.6 e381312664 Revert "Fix FA DRAFT_EXTEND_V2 cache extent" (#26628) 
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-28 18:45:33 -07:00
Cheng WanandClaude Opus 4.7 f66f56c6bd Add attention-backend unit-test suite under test/registered/attention/unittest (#26517) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 17:30:31 -07:00
Cheng WanandClaude Opus 4.7 4f92e63c99 Let unittest._ShouldStop propagate through retry() so subTest+failfast works (#26616) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 16:26:17 -07:00
Cheng WanandClaude Opus 4.7 12e28bdf0c Fix FlashInfer SWA EXTEND-with-prefix correctness in merge_state path (#26513) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 01:16:58 -07:00
Cheng WanandClaude Opus 4.7 00cd6fb3d9 Add sliding-window mask support to TorchNativeAttnBackend (#26516) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 01:06:57 -07:00
Cheng WanandClaude Opus 4.7 8ca09a30f1 Allow Optional key/value in unified_attention_with_output split-op (MLA absorb fix) (#26515) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 01:04:07 -07:00
Cheng WanandClaude Opus 4.7 b429a30428 Expose Flex attention causal/decode masks as static methods (#26514) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 01:03:32 -07:00
Cheng WanandClaude Opus 4.7 e5f5d84780 Fix FA DRAFT_EXTEND_V2 cache extent (#26512) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 00:56:23 -07:00
Cheng Wan 51840ca459 Add xutizhou as code owner for eplb directory (#26479) 2026-05-27 03:32:35 -07:00
Cheng WanandCheng Wan 44922de48a fix(swa): downgrade translate_loc_from_full_to_swa key-change log from warning to debug (#26225) 
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
 2026-05-24 11:37:46 -07:00
Cheng WanandCheng Wan 83a18e687d Revert "[refactor] unify cuda-graph capture/replay across attention backends (#26134)" (#26166) 
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
 2026-05-23 02:32:08 -07:00
Cheng WanandCheng Wan 5964d30233 fix(swa): eliminate spurious translate_loc_from_full_to_swa warning in BCG and CG paths (#26152) 
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
 2026-05-23 00:01:12 -07:00
Cheng WanCheng WanClaude Sonnet 4.6
 		d226f75669 [refactor] unify cuda-graph capture/replay across attention backends (#26134) 
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-22 18:51:13 -07:00
Cheng WanandClaude Sonnet 4.6 d765dfd043 refactor(attn): init hisparse_coordinator before attn_backend; replace lazy property with init-time capture (#26012) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-21 16:03:42 -07:00
Cheng WanandClaude Sonnet 4.6 c5251a98a9 feat(model_runner): remove pool/backend refs from ForwardBatch via ForwardContext (#25983) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-21 14:01:49 -07:00
Cheng WanandClaude Opus 4.7 b765faee30 [MoE Refactor] deprecate forward_npu and NpuFuseEPMoE (#25678) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 10:25:17 -07:00
Cheng WanandClaude Sonnet 4.6 888a8794ef [Fix] DSV4 cached_loc invalidated when SWA mapping is rebuilt (#25889) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-20 22:38:12 -07:00
Cheng WanandClaude Sonnet 4.6 79b937aefb [Refactor] Encapsulate SWA loc translation inside SWAKVPool with per-batch cache invalidation (#25824) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-20 21:26:32 -07:00
Cheng WanandClaude Sonnet 4.6 a528eb7564 fix: rustfmt service_discovery.rs warn! line length (#25927) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-20 19:25:11 -07:00
Cheng Wan bdacb1be4d Update CODEOWNERS to replace 'nsa' with 'dsa' (#25861) 2026-05-20 00:31:23 -07:00
Cheng WanandClaude Sonnet 4.6 8131641bc6 [Refactor] Rename NSA → DSA: user-facing aliases, file/class/import rename (#25821) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-20 00:18:04 -07:00
Cheng WanandClaude Sonnet 4.6 052abcc0dd [Refactor] Pass PP start_layer via model constructor instead of forward_batch.token_to_kv_pool (#25825) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-19 22:16:07 -07:00
Cheng Wan a4b51d35ef Revert "[codex] Update Wan2.2 ModelOpt CI checkpoints" (#25845) 2026-05-19 21:45:20 -07:00
Cheng Wan b7267e8fce [CI] Enable weight prefetch for 8-gpu-h200 basic tests (#25684) 2026-05-18 14:12:45 -07:00
Cheng WanandClaude Opus 4.7 7158a255eb [MoE Refactor] Migrate flashinfer_cutedsl + DeepEP to MoeRunner (#25525) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-17 14:48:17 -07:00
Cheng WanandClaude Opus 4.7 be3c425788 [MoE] Unify DeepEPMoE+MoriEPMoE through AITER MoeRunner pre/post-permute (#23760) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-17 02:25:52 -07:00
Cheng Wan 568ba7216a Fix logging for inplace setting in the flashInfer-trtllm backend (#25522) 2026-05-17 02:20:14 -07:00
Cheng Wan 229cadec04 Update logging for inplace setting in MoE layer (#25499) 2026-05-16 17:12:11 -07:00
Cheng Wan 0c017db916 Update kl_div_thres to 0.02 in swa_radix_cache (#25497) 2026-05-16 16:43:12 -07:00
Cheng WanandClaude Opus 4.7 54221dd998 feat(moe): reuse prev-layer output as symm_output for FP4 routed MoE (#25379) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-15 12:05:40 -07:00
Cheng WanandClaude Opus 4.7 ee93795476 perf(mla): hybrid Triton fused cat+FP8-quantize for MLA chunked-prefill K/V (#25333) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-15 10:51:00 -07:00
Cheng Wan 3f7e538b2f Add NPU condition for cosine and sine caching (#25399) 2026-05-15 05:21:53 -07:00
Cheng WanandClaude Opus 4.7 ad4994dc1d DeepseekV2MoE: defer shared experts when routed kernel is non-mutating (#25279) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-14 22:20:55 -07:00
Cheng Wan dca9ba6321 perf(mla): TMA bulk-store set_mla_kv_buffer (up to 12× over baseline) (#25311) 2026-05-14 18:23:41 -07:00
Cheng WanandClaude Opus 4.7 1913cb4dbb Skip CI tests added in #24816 (broken on main) (#25329) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-14 18:12:13 -07:00
Cheng WanandClaude Opus 4.7 6c0633b0b1 fix(nvfp4): make process_weights_after_loading hot-reload-safe via alias-when-same-shape (#25190) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 16:57:39 -07:00
Cheng WanandClaude Opus 4.7 72b49bfac6 docker, ci: swap GB DeepEP source from fzyzcjy fork to deepseek-ai/DeepEP@hybrid-ep (#25113) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 13:25:15 -07:00
Cheng WanandClaude Opus 4.7 f2a90094c9 bench: fix wrong flag names in bench_one_batch{,_server} docstrings (#25143) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-12 23:46:47 -07:00
Cheng WanandClaude Opus 4.7 4fb40bffac perf(nvfp4): free unused source scales after weight processing (#25107) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-12 16:53:50 -07:00
Cheng WanandClaude Opus 4.7 044bb88a97 fix: SGLANG_RADIX_FORCE_MISS chunk-cache passthrough (#24950) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-11 00:07:46 -07:00
Cheng WanCheng WanClaude Opus 4.7
 		c7f674e427 [Bug] Add dsv4 state_type branch to mooncake disaggregation (#24878) 
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-10 01:13:46 -07:00
Cheng Wansihan-zzzClaude Opus 4.7
 		d1c5937428 env: add SGLANG_RADIX_FORCE_MISS to force radix prefix-cache miss (#24726) 
Co-authored-by: sihan-zzz <228612289+sihan-zzz@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-08 17:46:38 -07:00
Cheng Wan 7b707c9222 disable the combination of --enable-two-batch-overlap and --enforce-s… (#24720) 2026-05-08 14:27:35 -07:00
Cheng Wan b47fab6f5d [bugfix] Support MIXED forward mode in TBO splitter for DP attention (#24241) 2026-05-01 16:01:23 -07:00
Cheng Wan d48095ba53 Bypass torch.cuda.use_mem_pool generator-CM in SymmetricMemoryContext (#24190) 2026-05-01 01:25:49 -07:00
Cheng WanandClaude Opus 4.7 108bfd8b6a [MoE] Add Aiter MoE runner backend and purge aiter.fused_moe from quant methods (#23597) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-30 19:50:52 -07:00
Cheng WanandClaude Opus 4.7 c7878dbb6d [MoE] Deprecate act_and_mul_triton; fold filter_expert into JIT silu/gelu_and_mul (#23707) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-26 01:41:35 -07:00
Cheng Wan d9c72bdd2b Skip unselected experts in flashinfer_trtllm (#23493) 2026-04-23 17:30:19 -07:00
Cheng WanandClaude Opus 4.7 000a2525e1 Move expert_mask_gpu from FusedMoE layer to StandardDispatcher (#23585) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 17:17:27 -07:00
Cheng WanandClaude Opus 4.7 ebcc2b3eec ci: run weekly est_time update on Monday using p90 of last 15 runs (#23120) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-19 14:39:27 -07:00
Cheng WanandClaude Opus 4.7 5f7aee726a refactor(moe): de-duplicate triton MoE runner path into shared helpers (#23019) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-17 17:05:13 -07:00
Cheng Wan b5e4ae7b1a fix: match est_time updates by backend, not just suite (#22563) 2026-04-10 17:54:50 -07:00
Cheng Wan 0011d2aec0 fix: track est_time per suite instead of per backend (#22557) 2026-04-10 16:58:40 -07:00
Cheng WanandClaude Opus 4.6 3f39b3d811 feat: add weekly workflow to update CI test est_time values (#22545) 
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-10 15:03:37 -07:00
Cheng WanandClaude Opus 4.6 6d95602ea3 Reduce GPU memory for MoE parallel groups (#22515) 
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-10 13:23:23 -07:00
Cheng WanandClaude Opus 4.6 37107bee6f [Observability] Add pending token count to prefill log and get_load (#22480) 
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-10 02:05:21 -07:00
Cheng Wan c01ee848b0 Revert "fix: use consistent time denominator for throughput metrics in bench_one_batch_server" (#21276) 2026-03-23 22:14:54 -07:00
Cheng Wan d6b12c401c Revert "[bugfix] Fix PPMissingLayer AttributeError when Using PP" (#21189) 2026-03-22 23:28:36 -07:00
Cheng Wan 6e54361608 Refactor CUDA graph input buffers with shared buffer pool (#19180) 2026-02-24 02:24:40 -08:00
Cheng Wan 84c67c8be0 Refactor graph input buffers (#18991) 2026-02-20 18:09:31 -08:00
Cheng Wan 38ee749dd9 Fix adjust_num_token_non_padded_for_attn_tp returning CPU tensor (#19051) 2026-02-20 23:23:38 +08:00
Cheng Wan b59a22f781 fix lint on main (#19052) 2026-02-20 15:30:57 +08:00
Cheng Wan 13a4a0406e Fix flashinfer autotune to only wrap run_once() (#19004) 2026-02-19 20:02:21 -08:00
Cheng Wan 64bca5315f Fix long prompt KV allocation by falling back to torch native APIs when exceeding Triton tensor limit (#18250) 2026-02-19 19:15:05 -08:00
Cheng Wan 73a7f0d049 Revert "Add SDAR model support" (#19032) 2026-02-19 16:03:56 -08:00
Cheng Wan 84c09913eb Moving _alloc_extend_naive out of npu allocator (#18200) 2026-02-04 02:09:55 -08:00
Cheng Wan ab8b99eb23 Refine logprob logic for request handling (#17986) 2026-02-01 19:11:52 -08:00
Cheng Wan 6a6b36367e Fix logprob_start_len handling for prefill-only requests (#17395) 2026-01-29 15:14:43 -08:00
Cheng Wan a416af4be7 Fix capture_sizes range for pcg (#17956) 2026-01-29 12:46:35 -08:00
Cheng Wan 2b461c15b4 Update logprob_start_len handling in scheduler (#16240) 2025-12-31 14:11:24 -08:00
Cheng Wan 5fb734f1a5 Enhance comments in set_extend_input_len method (#16130) 2025-12-29 20:47:33 -08:00
Cheng Wan 60f1ca6925 Refactor: Moving extend_logprob_start_len calculation out of prepare_for_extend (#16105) 2025-12-30 12:38:33 +08:00
Cheng Wan c2e0913e17 Fix extend_input_len calculation in decode.py (#16103) 2025-12-29 13:13:24 -08:00
Cheng Wan 6f9d0a89a0 [scheduler] fix: correcting extend_logprob_start_len calculation (#15922) 2025-12-28 14:57:04 -08:00
Cheng Wan c457aad54a Update test parameters for deepep_large test (#16001) 2025-12-28 00:58:19 -08:00
Cheng Wan 2ec57cefd9 hotfix: add type hints to scheduler mixins (#15916) 2025-12-26 17:08:11 -08:00
Cheng Wan 988b14ca0e refactor: add type hints to scheduler mixins (#15913) 2025-12-26 16:50:07 -08:00
Cheng Wan 5290cef97c Fix condition check for require_gathered_buffer (#15328) 2025-12-17 12:42:47 -08:00
Cheng Wan 96cc10834a [CI] update estimated elapsed time of some unittests (#14347) 2025-12-03 01:21:40 -08:00
Cheng WanandKangyan Zhou 0fe74af563 Remove incorrect deep_gemm assertions from server_args.py (#14113) 
Co-authored-by: Kangyan Zhou <zky314343421@gmail.com>
 2025-11-28 20:25:39 -08:00
Cheng Wan b12c9e5c0a Fix installation for nvidia-nvshmem-cu12 (#14033) 2025-11-26 18:27:12 -08:00
Cheng Wan bf10869203 [Doc] Add an Introduction to Expert Parallelism (#13783) 2025-11-24 14:46:51 -08:00
Cheng WanandBaizhou Zhang 6d0e0b9bfc [11/N] MoE Refactor: Simplifying SBO Implementation with Dispatcher Hooks (#13327) 
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
 2025-11-21 01:11:37 -08:00
Cheng WanandMick a5ad0069b2 fix: change performance log directory to cache path (#13482) 
Co-authored-by: Mick <mickjagger19@icloud.com>
 2025-11-18 15:18:43 +08:00