minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity
14,768 Commits 3 Branches 0 Tags
75cdaf432a88f4fcca6e220ac4e4415fd69bd1e0
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Cheng Wanandlch1475369 bc25abb786 perf(triton): avoid per-step D2H .item() sync in cuda-graph loc translate (#29921) 
Co-authored-by: lch1475369 <lch1475369@gmail.com>
 2026-07-02 13:20:00 -07:00
Cheng Wan b558cc1abb feat(short-conv): shared ShortConvAttnBackend for ZAYA1 CCA + LFM2 short conv (#29867) 2026-07-01 20:08:00 -07:00
Cheng Wanandlch1475369 4a8e76805c feat(mem_cache): unified memory pool for hybrid Mamba / SWA models (#29678) 
Co-authored-by: lch1475369 <lch1475369@gmail.com>
 2026-07-01 13:21:59 -07:00
Cheng Wanandlch1475369 fc96edd297 feat(mem_cache): page-major (layer-major within a page) KV/state layout (#29533) 
Co-authored-by: lch1475369 <lch1475369@gmail.com>
 2026-06-29 14:49:54 -07:00
Cheng Wan 7b02eab7a6 Revert "[DeepSeek V3] Run routed experts on main stream in dual-stream MoE" (#29452) 2026-06-26 13:59:36 -07:00
Cheng Wan c4376aaa88 [Refactor] Remove dead out_cache_loc_swa buffers (#28968) 2026-06-23 00:03:30 -07:00
Cheng Wan 6fdcb9934c fix(runner): size eager static buffers for prefill budget and MLP-sync autotune (#28677) 2026-06-19 13:23:26 -07:00
Cheng Wan 2aa7b58aa7 refactor(runner): reuse a prepared static buffer for every dummy run (#28740) 2026-06-19 13:16:07 -07:00
Cheng WanandClaude Opus 4.8 856b0dc74b refactor(runner): move kernel warmup into the shared runner lifecycle (warmup()) (#28739) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-19 13:13:13 -07:00
Cheng WanandClaude Opus 4.8 d705a91de1 refactor(runner): add EagerRunner, own the eager path, polymorphic dispatch (#28386) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-19 13:05:43 -07:00
Cheng Wan a6db86d535 refactor(runner): split BaseRunner (shared) from BaseCudaGraphRunner (#28385) 2026-06-19 02:04:15 -07:00
Cheng Wan 1c6331cbd6 refactor(runner): rename runner replay/load/can_run for the shared surface (#28384) 2026-06-19 01:45:04 -07:00
Cheng Wan f83e4d5968 refactor(runner): unify eager-forward DP/MLP-sync padding into one helper (#28383) 2026-06-18 13:41:30 -07:00
Cheng Wan 67db2ac3e7 refactor(runner): unify pp_proxy_tensors forward kwarg into one helper (#28382) 2026-06-18 02:23:47 -07:00
Cheng Wan 53318911ca Add get_parallel(): a structured accessor for parallel-topology state (#28567) 2026-06-17 20:23:43 -07:00
Cheng Wan 448af67a98 ci: run AMD and NPU PR tests on PRs not targeting main (#28368) 2026-06-15 19:10:51 -07:00
Cheng Wan 27ba13358e [Spec] Clear dead DRAFT_EXTEND objects left after EAGLE v1 removal (#28133) 2026-06-13 13:04:51 -07:00
Cheng Wan bde6bccf39 [Spec] Remove deprecated EAGLE v1 DRAFT_EXTEND forward mode (#28129) 2026-06-13 13:00:21 -07:00
Cheng Wan 3a1417a0c1 [refactor] Fold FrozenKVMTPCudaGraphRunner onto the shared DecodeCudaGraphRunner base (#28081) 2026-06-12 16:55:53 -07:00
Cheng WanandClaude Opus 4.8 d601edab73 [Spec] Fix EagleDraftWorker draft-extend attn backend assignment (#28096) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-12 15:47:05 -07:00
Cheng Wan 2e74ff192c [DSV4] Use int64 for compressor out_loc tensors (#27973) 2026-06-11 17:45:34 -07:00
Cheng Wan 97a0031799 [lint] Enable Ruff UP037 to drop redundant quoted annotations (#27984) 2026-06-11 17:38:05 -07:00
Cheng Wan 9425ba2478 Remove extra_pytest_path from pr-test.yml (#27856) 2026-06-10 21:16:14 -07:00
Cheng Wan f4f30d7d23 [Fix] Use int64 seq_lens across all CUDA graph runners and backends (#27840) 2026-06-10 19:54:58 -07:00
Cheng Wan 70c71ba183 [NPU] Fix dead patch_model monkey-patch breaking NPU torch.compile capture (#27774) 2026-06-10 01:15:50 -07:00
Cheng Wan 95d8a75bc9 Bundle set_kv_buffer write targets into KVWriteLoc (loc + swa_loc) (#27695) 2026-06-09 23:09:51 -07:00
Cheng Wan 758fd4bb9a [SWA] Cache full→SWA out_cache_loc per forward across attention backends (#27617) 2026-06-09 22:57:51 -07:00
Cheng WanandClaude Opus 4.8 9097647090 Route the eager forward path through the CUDA graph input-buffer registry (#27407) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-06 14:35:53 -07:00
Cheng WanandClaude Opus 4.8 7dc7376697 fix(attn): delegate init_mha_chunk_metadata in HybridLinearAttnBackend (#27316) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-04 17:44:23 -07:00
Cheng WanandClaude Opus 4.8 0aa72a9e76 Replace skip_attn_backend_init with a batch-carried attention plan marker (+ staleness re-plan) (#27193) 
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-04 17:13:18 -07:00
Cheng WanandClaude Opus 4.8 10ab7c919f [refactor] Retire DecodeInputBuffers / PrefillInputBuffers in favor of CudaGraphBufferRegistry (#27192) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-03 20:52:56 -07:00
Cheng WanandClaude Opus 4.8 c9ca56da8c Unify full→SWA index translation in init_forward_metadata; drop pool caches (#27091) 
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-03 16:12:27 -07:00
Cheng Wan 61aa3293d3 Revert "Fix TokenizerManager crash on top_logprobs with tensor values" (#27187) 2026-06-03 14:53:28 -07:00
Cheng WanandClaude Opus 4.8 e485ad6ac1 Fix hybrid linear attention dispatch by layer id with draft-worker awareness (#27120) 
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-03 14:11:14 -07:00
Cheng WanandClaude Opus 4.8 45604a0f4a [refactor] Unify CUDA graph runner input buffers behind CudaGraphBufferRegistry (#26742) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-03 10:54:10 -07:00
Cheng Wan 73b53e7a87 Revert "Support NextN = 2/4 in DSV32" (#27138) 2026-06-03 01:29:52 -07:00
Cheng Wan 202e618898 Revert "Fix hybrid linear attention misrouting plain-RadixAttention linear layers to the full backend (Ring-2.5-1T)" (#27116) 2026-06-02 23:27:57 -07:00
Cheng WanandClaude Opus 4.7 99da43b900 [refactor] init_forward_metadata 3-method ABC + side-channel removal + ForwardMetadata type rename (#26735) 
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-02 10:33:33 -07:00
Cheng WanandClaude Sonnet 4.6 ff8ed7a302 [refactor] unify cuda-graph capture/replay across attention backends (#26665) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-29 12:46:42 -07:00
Cheng WanandClaude Sonnet 4.6 ec075d8bc5 Fix DRAFT_EXTEND_V2 CG metadata: align test fixture and Triton with production seq_lens convention (#26651) 
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-29 02:46:45 -07:00
Cheng WanandClaude Sonnet 4.6 0fb0ea7aac test: add trtllm_mha EAGLE-draft CG runner coverage (chain) (#26669) 
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-28 23:46:06 -07:00
Cheng WanandClaude Sonnet 4.6 2dfbc3d781 test: strengthen CG-replay coverage with prod-fill padding, metadata invariants, and pad-ratio sweep (#26658) 
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-28 22:43:29 -07:00
Cheng Wan 84698b21e7 rename unittest as unittests (#26654) 2026-05-28 21:14:06 -07:00
Cheng WanandClaude Sonnet 4.6 e381312664 Revert "Fix FA DRAFT_EXTEND_V2 cache extent" (#26628) 
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-28 18:45:33 -07:00
Cheng WanandClaude Opus 4.7 f66f56c6bd Add attention-backend unit-test suite under test/registered/attention/unittest (#26517) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 17:30:31 -07:00
Cheng WanandClaude Opus 4.7 4f92e63c99 Let unittest._ShouldStop propagate through retry() so subTest+failfast works (#26616) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 16:26:17 -07:00
Cheng WanandClaude Opus 4.7 12e28bdf0c Fix FlashInfer SWA EXTEND-with-prefix correctness in merge_state path (#26513) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 01:16:58 -07:00
Cheng WanandClaude Opus 4.7 00cd6fb3d9 Add sliding-window mask support to TorchNativeAttnBackend (#26516) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 01:06:57 -07:00
Cheng WanandClaude Opus 4.7 8ca09a30f1 Allow Optional key/value in unified_attention_with_output split-op (MLA absorb fix) (#26515) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 01:04:07 -07:00
Cheng WanandClaude Opus 4.7 b429a30428 Expose Flex attention causal/decode masks as static methods (#26514) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 01:03:32 -07:00
Cheng WanandClaude Opus 4.7 e5f5d84780 Fix FA DRAFT_EXTEND_V2 cache extent (#26512) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 00:56:23 -07:00
Cheng Wan 51840ca459 Add xutizhou as code owner for eplb directory (#26479) 2026-05-27 03:32:35 -07:00
Cheng WanandCheng Wan 44922de48a fix(swa): downgrade translate_loc_from_full_to_swa key-change log from warning to debug (#26225) 
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
 2026-05-24 11:37:46 -07:00
Cheng WanandCheng Wan 83a18e687d Revert "[refactor] unify cuda-graph capture/replay across attention backends (#26134)" (#26166) 
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
 2026-05-23 02:32:08 -07:00
Cheng WanandCheng Wan 5964d30233 fix(swa): eliminate spurious translate_loc_from_full_to_swa warning in BCG and CG paths (#26152) 
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
 2026-05-23 00:01:12 -07:00
Cheng WanCheng WanClaude Sonnet 4.6
 		d226f75669 [refactor] unify cuda-graph capture/replay across attention backends (#26134) 
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-22 18:51:13 -07:00
Cheng WanandClaude Sonnet 4.6 d765dfd043 refactor(attn): init hisparse_coordinator before attn_backend; replace lazy property with init-time capture (#26012) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-21 16:03:42 -07:00
Cheng WanandClaude Sonnet 4.6 c5251a98a9 feat(model_runner): remove pool/backend refs from ForwardBatch via ForwardContext (#25983) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-21 14:01:49 -07:00
Cheng WanandClaude Opus 4.7 b765faee30 [MoE Refactor] deprecate forward_npu and NpuFuseEPMoE (#25678) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 10:25:17 -07:00
Cheng WanandClaude Sonnet 4.6 888a8794ef [Fix] DSV4 cached_loc invalidated when SWA mapping is rebuilt (#25889) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-20 22:38:12 -07:00
Cheng WanandClaude Sonnet 4.6 79b937aefb [Refactor] Encapsulate SWA loc translation inside SWAKVPool with per-batch cache invalidation (#25824) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-20 21:26:32 -07:00
Cheng WanandClaude Sonnet 4.6 a528eb7564 fix: rustfmt service_discovery.rs warn! line length (#25927) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-20 19:25:11 -07:00
Cheng Wan bdacb1be4d Update CODEOWNERS to replace 'nsa' with 'dsa' (#25861) 2026-05-20 00:31:23 -07:00
Cheng WanandClaude Sonnet 4.6 8131641bc6 [Refactor] Rename NSA → DSA: user-facing aliases, file/class/import rename (#25821) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-20 00:18:04 -07:00
Cheng WanandClaude Sonnet 4.6 052abcc0dd [Refactor] Pass PP start_layer via model constructor instead of forward_batch.token_to_kv_pool (#25825) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-19 22:16:07 -07:00
Cheng Wan a4b51d35ef Revert "[codex] Update Wan2.2 ModelOpt CI checkpoints" (#25845) 2026-05-19 21:45:20 -07:00
Cheng Wan b7267e8fce [CI] Enable weight prefetch for 8-gpu-h200 basic tests (#25684) 2026-05-18 14:12:45 -07:00
Cheng WanandClaude Opus 4.7 7158a255eb [MoE Refactor] Migrate flashinfer_cutedsl + DeepEP to MoeRunner (#25525) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-17 14:48:17 -07:00
Cheng WanandClaude Opus 4.7 be3c425788 [MoE] Unify DeepEPMoE+MoriEPMoE through AITER MoeRunner pre/post-permute (#23760) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-17 02:25:52 -07:00
Cheng Wan 568ba7216a Fix logging for inplace setting in the flashInfer-trtllm backend (#25522) 2026-05-17 02:20:14 -07:00
Cheng Wan 229cadec04 Update logging for inplace setting in MoE layer (#25499) 2026-05-16 17:12:11 -07:00
Cheng Wan 0c017db916 Update kl_div_thres to 0.02 in swa_radix_cache (#25497) 2026-05-16 16:43:12 -07:00
Cheng WanandClaude Opus 4.7 54221dd998 feat(moe): reuse prev-layer output as symm_output for FP4 routed MoE (#25379) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-15 12:05:40 -07:00
Cheng WanandClaude Opus 4.7 ee93795476 perf(mla): hybrid Triton fused cat+FP8-quantize for MLA chunked-prefill K/V (#25333) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-15 10:51:00 -07:00
Cheng Wan 3f7e538b2f Add NPU condition for cosine and sine caching (#25399) 2026-05-15 05:21:53 -07:00
Cheng WanandClaude Opus 4.7 ad4994dc1d DeepseekV2MoE: defer shared experts when routed kernel is non-mutating (#25279) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-14 22:20:55 -07:00
Cheng Wan dca9ba6321 perf(mla): TMA bulk-store set_mla_kv_buffer (up to 12× over baseline) (#25311) 2026-05-14 18:23:41 -07:00
Cheng WanandClaude Opus 4.7 1913cb4dbb Skip CI tests added in #24816 (broken on main) (#25329) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-14 18:12:13 -07:00
Cheng WanandClaude Opus 4.7 6c0633b0b1 fix(nvfp4): make process_weights_after_loading hot-reload-safe via alias-when-same-shape (#25190) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 16:57:39 -07:00
Cheng WanandClaude Opus 4.7 72b49bfac6 docker, ci: swap GB DeepEP source from fzyzcjy fork to deepseek-ai/DeepEP@hybrid-ep (#25113) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 13:25:15 -07:00
Cheng WanandClaude Opus 4.7 f2a90094c9 bench: fix wrong flag names in bench_one_batch{,_server} docstrings (#25143) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-12 23:46:47 -07:00
Cheng WanandClaude Opus 4.7 4fb40bffac perf(nvfp4): free unused source scales after weight processing (#25107) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-12 16:53:50 -07:00
Cheng WanandClaude Opus 4.7 044bb88a97 fix: SGLANG_RADIX_FORCE_MISS chunk-cache passthrough (#24950) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-11 00:07:46 -07:00
Cheng WanCheng WanClaude Opus 4.7
 		c7f674e427 [Bug] Add dsv4 state_type branch to mooncake disaggregation (#24878) 
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-10 01:13:46 -07:00
Cheng Wansihan-zzzClaude Opus 4.7
 		d1c5937428 env: add SGLANG_RADIX_FORCE_MISS to force radix prefix-cache miss (#24726) 
Co-authored-by: sihan-zzz <228612289+sihan-zzz@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-08 17:46:38 -07:00
Cheng Wan 7b707c9222 disable the combination of --enable-two-batch-overlap and --enforce-s… (#24720) 2026-05-08 14:27:35 -07:00
Cheng Wan b47fab6f5d [bugfix] Support MIXED forward mode in TBO splitter for DP attention (#24241) 2026-05-01 16:01:23 -07:00
Cheng Wan d48095ba53 Bypass torch.cuda.use_mem_pool generator-CM in SymmetricMemoryContext (#24190) 2026-05-01 01:25:49 -07:00
Cheng WanandClaude Opus 4.7 108bfd8b6a [MoE] Add Aiter MoE runner backend and purge aiter.fused_moe from quant methods (#23597) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-30 19:50:52 -07:00
Cheng WanandClaude Opus 4.7 c7878dbb6d [MoE] Deprecate act_and_mul_triton; fold filter_expert into JIT silu/gelu_and_mul (#23707) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-26 01:41:35 -07:00
Cheng Wan d9c72bdd2b Skip unselected experts in flashinfer_trtllm (#23493) 2026-04-23 17:30:19 -07:00
Cheng WanandClaude Opus 4.7 000a2525e1 Move expert_mask_gpu from FusedMoE layer to StandardDispatcher (#23585) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-23 17:17:27 -07:00
Cheng WanandClaude Opus 4.7 ebcc2b3eec ci: run weekly est_time update on Monday using p90 of last 15 runs (#23120) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-19 14:39:27 -07:00
Cheng WanandClaude Opus 4.7 5f7aee726a refactor(moe): de-duplicate triton MoE runner path into shared helpers (#23019) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-17 17:05:13 -07:00
Cheng Wan b5e4ae7b1a fix: match est_time updates by backend, not just suite (#22563) 2026-04-10 17:54:50 -07:00
Cheng Wan 0011d2aec0 fix: track est_time per suite instead of per backend (#22557) 2026-04-10 16:58:40 -07:00
Cheng WanandClaude Opus 4.6 3f39b3d811 feat: add weekly workflow to update CI test est_time values (#22545) 
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-10 15:03:37 -07:00
Cheng WanandClaude Opus 4.6 6d95602ea3 Reduce GPU memory for MoE parallel groups (#22515) 
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-10 13:23:23 -07:00
Cheng WanandClaude Opus 4.6 37107bee6f [Observability] Add pending token count to prefill log and get_load (#22480) 
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-10 02:05:21 -07:00
Cheng Wan c01ee848b0 Revert "fix: use consistent time denominator for throughput metrics in bench_one_batch_server" (#21276) 2026-03-23 22:14:54 -07:00