minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity
13,653 Commits 3 Branches 0 Tags
bd7fea07409e788379735905c073a834165453b6
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Cheng WanandClaude Opus 4.8 7dc7376697 fix(attn): delegate init_mha_chunk_metadata in HybridLinearAttnBackend (#27316) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-04 17:44:23 -07:00
Cheng WanandClaude Opus 4.8 0aa72a9e76 Replace skip_attn_backend_init with a batch-carried attention plan marker (+ staleness re-plan) (#27193) 
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-04 17:13:18 -07:00
Cheng WanandClaude Opus 4.8 10ab7c919f [refactor] Retire DecodeInputBuffers / PrefillInputBuffers in favor of CudaGraphBufferRegistry (#27192) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-03 20:52:56 -07:00
Cheng WanandClaude Opus 4.8 c9ca56da8c Unify full→SWA index translation in init_forward_metadata; drop pool caches (#27091) 
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-03 16:12:27 -07:00
Cheng Wan 61aa3293d3 Revert "Fix TokenizerManager crash on top_logprobs with tensor values" (#27187) 2026-06-03 14:53:28 -07:00
Cheng WanandClaude Opus 4.8 e485ad6ac1 Fix hybrid linear attention dispatch by layer id with draft-worker awareness (#27120) 
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-03 14:11:14 -07:00
Cheng WanandClaude Opus 4.8 45604a0f4a [refactor] Unify CUDA graph runner input buffers behind CudaGraphBufferRegistry (#26742) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-03 10:54:10 -07:00
Cheng Wan 73b53e7a87 Revert "Support NextN = 2/4 in DSV32" (#27138) 2026-06-03 01:29:52 -07:00
Cheng Wan 202e618898 Revert "Fix hybrid linear attention misrouting plain-RadixAttention linear layers to the full backend (Ring-2.5-1T)" (#27116) 2026-06-02 23:27:57 -07:00
Cheng WanandClaude Opus 4.7 99da43b900 [refactor] init_forward_metadata 3-method ABC + side-channel removal + ForwardMetadata type rename (#26735) 
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-02 10:33:33 -07:00
Cheng WanandClaude Sonnet 4.6 ff8ed7a302 [refactor] unify cuda-graph capture/replay across attention backends (#26665) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-29 12:46:42 -07:00
Cheng WanandClaude Sonnet 4.6 ec075d8bc5 Fix DRAFT_EXTEND_V2 CG metadata: align test fixture and Triton with production seq_lens convention (#26651) 
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-29 02:46:45 -07:00
Cheng WanandClaude Sonnet 4.6 0fb0ea7aac test: add trtllm_mha EAGLE-draft CG runner coverage (chain) (#26669) 
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-28 23:46:06 -07:00
Cheng WanandClaude Sonnet 4.6 2dfbc3d781 test: strengthen CG-replay coverage with prod-fill padding, metadata invariants, and pad-ratio sweep (#26658) 
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-28 22:43:29 -07:00
Cheng Wan 84698b21e7 rename unittest as unittests (#26654) 2026-05-28 21:14:06 -07:00
Cheng WanandClaude Sonnet 4.6 e381312664 Revert "Fix FA DRAFT_EXTEND_V2 cache extent" (#26628) 
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-28 18:45:33 -07:00
Cheng WanandClaude Opus 4.7 f66f56c6bd Add attention-backend unit-test suite under test/registered/attention/unittest (#26517) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 17:30:31 -07:00
Cheng WanandClaude Opus 4.7 4f92e63c99 Let unittest._ShouldStop propagate through retry() so subTest+failfast works (#26616) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 16:26:17 -07:00
Cheng WanandClaude Opus 4.7 12e28bdf0c Fix FlashInfer SWA EXTEND-with-prefix correctness in merge_state path (#26513) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 01:16:58 -07:00
Cheng WanandClaude Opus 4.7 00cd6fb3d9 Add sliding-window mask support to TorchNativeAttnBackend (#26516) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 01:06:57 -07:00
Cheng WanandClaude Opus 4.7 8ca09a30f1 Allow Optional key/value in unified_attention_with_output split-op (MLA absorb fix) (#26515) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 01:04:07 -07:00
Cheng WanandClaude Opus 4.7 b429a30428 Expose Flex attention causal/decode masks as static methods (#26514) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 01:03:32 -07:00
Cheng WanandClaude Opus 4.7 e5f5d84780 Fix FA DRAFT_EXTEND_V2 cache extent (#26512) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 00:56:23 -07:00
Cheng Wan 51840ca459 Add xutizhou as code owner for eplb directory (#26479) 2026-05-27 03:32:35 -07:00
Cheng WanandCheng Wan 44922de48a fix(swa): downgrade translate_loc_from_full_to_swa key-change log from warning to debug (#26225) 
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
 2026-05-24 11:37:46 -07:00
Cheng WanandCheng Wan 83a18e687d Revert "[refactor] unify cuda-graph capture/replay across attention backends (#26134)" (#26166) 
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
 2026-05-23 02:32:08 -07:00
Cheng WanandCheng Wan 5964d30233 fix(swa): eliminate spurious translate_loc_from_full_to_swa warning in BCG and CG paths (#26152) 
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
 2026-05-23 00:01:12 -07:00
Cheng WanCheng WanClaude Sonnet 4.6
 		d226f75669 [refactor] unify cuda-graph capture/replay across attention backends (#26134) 
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-22 18:51:13 -07:00
Cheng WanandClaude Sonnet 4.6 d765dfd043 refactor(attn): init hisparse_coordinator before attn_backend; replace lazy property with init-time capture (#26012) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-21 16:03:42 -07:00
Cheng WanandClaude Sonnet 4.6 c5251a98a9 feat(model_runner): remove pool/backend refs from ForwardBatch via ForwardContext (#25983) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-21 14:01:49 -07:00
Cheng WanandClaude Opus 4.7 b765faee30 [MoE Refactor] deprecate forward_npu and NpuFuseEPMoE (#25678) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 10:25:17 -07:00
Cheng WanandClaude Sonnet 4.6 888a8794ef [Fix] DSV4 cached_loc invalidated when SWA mapping is rebuilt (#25889) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-20 22:38:12 -07:00
Cheng WanandClaude Sonnet 4.6 79b937aefb [Refactor] Encapsulate SWA loc translation inside SWAKVPool with per-batch cache invalidation (#25824) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-20 21:26:32 -07:00
Cheng WanandClaude Sonnet 4.6 a528eb7564 fix: rustfmt service_discovery.rs warn! line length (#25927) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-20 19:25:11 -07:00
Cheng Wan bdacb1be4d Update CODEOWNERS to replace 'nsa' with 'dsa' (#25861) 2026-05-20 00:31:23 -07:00
Cheng WanandClaude Sonnet 4.6 8131641bc6 [Refactor] Rename NSA → DSA: user-facing aliases, file/class/import rename (#25821) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-20 00:18:04 -07:00
Cheng WanandClaude Sonnet 4.6 052abcc0dd [Refactor] Pass PP start_layer via model constructor instead of forward_batch.token_to_kv_pool (#25825) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-19 22:16:07 -07:00
Cheng Wan a4b51d35ef Revert "[codex] Update Wan2.2 ModelOpt CI checkpoints" (#25845) 2026-05-19 21:45:20 -07:00
Cheng Wan b7267e8fce [CI] Enable weight prefetch for 8-gpu-h200 basic tests (#25684) 2026-05-18 14:12:45 -07:00
Cheng WanandClaude Opus 4.7 7158a255eb [MoE Refactor] Migrate flashinfer_cutedsl + DeepEP to MoeRunner (#25525) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-17 14:48:17 -07:00
Cheng WanandClaude Opus 4.7 be3c425788 [MoE] Unify DeepEPMoE+MoriEPMoE through AITER MoeRunner pre/post-permute (#23760) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-17 02:25:52 -07:00
Cheng Wan 568ba7216a Fix logging for inplace setting in the flashInfer-trtllm backend (#25522) 2026-05-17 02:20:14 -07:00
Cheng Wan 229cadec04 Update logging for inplace setting in MoE layer (#25499) 2026-05-16 17:12:11 -07:00
Cheng Wan 0c017db916 Update kl_div_thres to 0.02 in swa_radix_cache (#25497) 2026-05-16 16:43:12 -07:00
Cheng WanandClaude Opus 4.7 54221dd998 feat(moe): reuse prev-layer output as symm_output for FP4 routed MoE (#25379) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-15 12:05:40 -07:00
Cheng WanandClaude Opus 4.7 ee93795476 perf(mla): hybrid Triton fused cat+FP8-quantize for MLA chunked-prefill K/V (#25333) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-15 10:51:00 -07:00
Cheng Wan 3f7e538b2f Add NPU condition for cosine and sine caching (#25399) 2026-05-15 05:21:53 -07:00
Cheng WanandClaude Opus 4.7 ad4994dc1d DeepseekV2MoE: defer shared experts when routed kernel is non-mutating (#25279) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-14 22:20:55 -07:00
Cheng Wan dca9ba6321 perf(mla): TMA bulk-store set_mla_kv_buffer (up to 12× over baseline) (#25311) 2026-05-14 18:23:41 -07:00
Cheng WanandClaude Opus 4.7 1913cb4dbb Skip CI tests added in #24816 (broken on main) (#25329) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-14 18:12:13 -07:00
Cheng WanandClaude Opus 4.7 6c0633b0b1 fix(nvfp4): make process_weights_after_loading hot-reload-safe via alias-when-same-shape (#25190) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 16:57:39 -07:00
Cheng WanandClaude Opus 4.7 72b49bfac6 docker, ci: swap GB DeepEP source from fzyzcjy fork to deepseek-ai/DeepEP@hybrid-ep (#25113) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 13:25:15 -07:00
Cheng WanandClaude Opus 4.7 f2a90094c9 bench: fix wrong flag names in bench_one_batch{,_server} docstrings (#25143) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-12 23:46:47 -07:00
Cheng WanandClaude Opus 4.7 4fb40bffac perf(nvfp4): free unused source scales after weight processing (#25107) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-12 16:53:50 -07:00
Cheng WanandClaude Opus 4.7 044bb88a97 fix: SGLANG_RADIX_FORCE_MISS chunk-cache passthrough (#24950) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-11 00:07:46 -07:00
Cheng WanCheng WanClaude Opus 4.7
 		c7f674e427 [Bug] Add dsv4 state_type branch to mooncake disaggregation (#24878) 
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-10 01:13:46 -07:00
Cheng Wansihan-zzzClaude Opus 4.7
 		d1c5937428 env: add SGLANG_RADIX_FORCE_MISS to force radix prefix-cache miss (#24726) 
Co-authored-by: sihan-zzz <228612289+sihan-zzz@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-08 17:46:38 -07:00
Cheng Wan 7b707c9222 disable the combination of --enable-two-batch-overlap and --enforce-s… (#24720) 2026-05-08 14:27:35 -07:00
Cheng Wan b47fab6f5d [bugfix] Support MIXED forward mode in TBO splitter for DP attention (#24241) 2026-05-01 16:01:23 -07:00
Cheng Wan d48095ba53 Bypass torch.cuda.use_mem_pool generator-CM in SymmetricMemoryContext (#24190) 2026-05-01 01:25:49 -07:00
Cheng WanandClaude Opus 4.7 108bfd8b6a [MoE] Add Aiter MoE runner backend and purge aiter.fused_moe from quant methods (#23597) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-30 19:50:52 -07:00
Cheng WanandClaude Opus 4.7 c7878dbb6d [MoE] Deprecate act_and_mul_triton; fold filter_expert into JIT silu/gelu_and_mul (#23707) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-26 01:41:35 -07:00
Cheng Wan d9c72bdd2b Skip unselected experts in flashinfer_trtllm (#23493) 2026-04-23 17:30:19 -07:00
Cheng WanandClaude Opus 4.7 000a2525e1 Move expert_mask_gpu from FusedMoE layer to StandardDispatcher (#23585) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 17:17:27 -07:00
Cheng WanandClaude Opus 4.7 ebcc2b3eec ci: run weekly est_time update on Monday using p90 of last 15 runs (#23120) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-19 14:39:27 -07:00
Cheng WanandClaude Opus 4.7 5f7aee726a refactor(moe): de-duplicate triton MoE runner path into shared helpers (#23019) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-17 17:05:13 -07:00
Cheng Wan b5e4ae7b1a fix: match est_time updates by backend, not just suite (#22563) 2026-04-10 17:54:50 -07:00
Cheng Wan 0011d2aec0 fix: track est_time per suite instead of per backend (#22557) 2026-04-10 16:58:40 -07:00
Cheng WanandClaude Opus 4.6 3f39b3d811 feat: add weekly workflow to update CI test est_time values (#22545) 
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-10 15:03:37 -07:00
Cheng WanandClaude Opus 4.6 6d95602ea3 Reduce GPU memory for MoE parallel groups (#22515) 
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-10 13:23:23 -07:00
Cheng WanandClaude Opus 4.6 37107bee6f [Observability] Add pending token count to prefill log and get_load (#22480) 
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-10 02:05:21 -07:00
Cheng Wan c01ee848b0 Revert "fix: use consistent time denominator for throughput metrics in bench_one_batch_server" (#21276) 2026-03-23 22:14:54 -07:00
Cheng Wan d6b12c401c Revert "[bugfix] Fix PPMissingLayer AttributeError when Using PP" (#21189) 2026-03-22 23:28:36 -07:00
Cheng Wan 6e54361608 Refactor CUDA graph input buffers with shared buffer pool (#19180) 2026-02-24 02:24:40 -08:00
Cheng Wan 84c67c8be0 Refactor graph input buffers (#18991) 2026-02-20 18:09:31 -08:00
Cheng Wan 38ee749dd9 Fix adjust_num_token_non_padded_for_attn_tp returning CPU tensor (#19051) 2026-02-20 23:23:38 +08:00
Cheng Wan b59a22f781 fix lint on main (#19052) 2026-02-20 15:30:57 +08:00
Cheng Wan 13a4a0406e Fix flashinfer autotune to only wrap run_once() (#19004) 2026-02-19 20:02:21 -08:00
Cheng Wan 64bca5315f Fix long prompt KV allocation by falling back to torch native APIs when exceeding Triton tensor limit (#18250) 2026-02-19 19:15:05 -08:00
Cheng Wan 73a7f0d049 Revert "Add SDAR model support" (#19032) 2026-02-19 16:03:56 -08:00
Cheng Wan 84c09913eb Moving _alloc_extend_naive out of npu allocator (#18200) 2026-02-04 02:09:55 -08:00
Cheng Wan ab8b99eb23 Refine logprob logic for request handling (#17986) 2026-02-01 19:11:52 -08:00
Cheng Wan 6a6b36367e Fix logprob_start_len handling for prefill-only requests (#17395) 2026-01-29 15:14:43 -08:00
Cheng Wan a416af4be7 Fix capture_sizes range for pcg (#17956) 2026-01-29 12:46:35 -08:00
Cheng Wan 2b461c15b4 Update logprob_start_len handling in scheduler (#16240) 2025-12-31 14:11:24 -08:00
Cheng Wan 5fb734f1a5 Enhance comments in set_extend_input_len method (#16130) 2025-12-29 20:47:33 -08:00
Cheng Wan 60f1ca6925 Refactor: Moving extend_logprob_start_len calculation out of prepare_for_extend (#16105) 2025-12-30 12:38:33 +08:00
Cheng Wan c2e0913e17 Fix extend_input_len calculation in decode.py (#16103) 2025-12-29 13:13:24 -08:00
Cheng Wan 6f9d0a89a0 [scheduler] fix: correcting extend_logprob_start_len calculation (#15922) 2025-12-28 14:57:04 -08:00
Cheng Wan c457aad54a Update test parameters for deepep_large test (#16001) 2025-12-28 00:58:19 -08:00
Cheng Wan 2ec57cefd9 hotfix: add type hints to scheduler mixins (#15916) 2025-12-26 17:08:11 -08:00
Cheng Wan 988b14ca0e refactor: add type hints to scheduler mixins (#15913) 2025-12-26 16:50:07 -08:00
Cheng Wan 5290cef97c Fix condition check for require_gathered_buffer (#15328) 2025-12-17 12:42:47 -08:00
Cheng Wan 96cc10834a [CI] update estimated elapsed time of some unittests (#14347) 2025-12-03 01:21:40 -08:00
Cheng WanandKangyan Zhou 0fe74af563 Remove incorrect deep_gemm assertions from server_args.py (#14113) 
Co-authored-by: Kangyan Zhou <zky314343421@gmail.com>
 2025-11-28 20:25:39 -08:00
Cheng Wan b12c9e5c0a Fix installation for nvidia-nvshmem-cu12 (#14033) 2025-11-26 18:27:12 -08:00
Cheng Wan bf10869203 [Doc] Add an Introduction to Expert Parallelism (#13783) 2025-11-24 14:46:51 -08:00
Cheng WanandBaizhou Zhang 6d0e0b9bfc [11/N] MoE Refactor: Simplifying SBO Implementation with Dispatcher Hooks (#13327) 
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
 2025-11-21 01:11:37 -08:00
Cheng WanandMick a5ad0069b2 fix: change performance log directory to cache path (#13482) 
Co-authored-by: Mick <mickjagger19@icloud.com>
 2025-11-18 15:18:43 +08:00
Cheng Wan 2b8b9d8496 [CI] use cached deepep installation in gb200 CI (#13388) 2025-11-16 12:49:01 -08:00