minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity
15,506 Commits 3 Branches 0 Tags
9057db94175e58ce59a234c60aea167c436ca5c2
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Cheng WanChunan ZengKe BaoYanbin JiangYuhao YangQiaolin YuZhichen ZengAurick QiaoJoseph
 		02236fa38c Add Inkling model support (#31681) 
Co-authored-by: Chunan Zeng <zcnrex@gmail.com>
Co-authored-by: Ke Bao <ispobaoke@gmail.com>
Co-authored-by: Yanbin Jiang <jybsuper@gmail.com>
Co-authored-by: Yuhao Yang <47235274+yhyang201@users.noreply.github.com>
Co-authored-by: Qiaolin Yu <qiaolin.yu@radixark.ai>
Co-authored-by: Zhichen Zeng <zczeng@uw.edu>
Co-authored-by: Aurick Qiao <aurick@thinkingmachines.ai>
Co-authored-by: Joseph <jk@thinkingmachines.ai>
 2026-07-19 22:57:37 -07:00
Cheng Wan fc2ef35308 [refactor] Move MLP collective flags onto ForwardFlags (#30802) 2026-07-10 17:43:39 -07:00
Cheng Wan c02b032da0 [fix] Repoint the prefetch-dispatch test at the loader's current config binding (#30631) 2026-07-09 02:21:04 -07:00
Cheng Wan d8791fb5e3 [docs] Add the sglang-runtime-context skill (#30494) 2026-07-09 02:11:36 -07:00
Cheng Wan 1f15308dca [refactor] Retire the legacy config accessor and the remaining process singletons (#30493) 2026-07-09 02:10:47 -07:00
Cheng Wan e703f9e566 [refactor] Adopt get_parallel() everywhere and close out the parallel wrapper surface (#30492) 2026-07-09 02:09:39 -07:00
Cheng Wan 06eb1b1838 [refactor] Split the DP gathered-buffer state between flags.dp and ctx.forward (#30491) 2026-07-09 02:09:21 -07:00
Cheng Wan fef2128e19 [refactor] Add the per-forward flags tier: ctx.forward (#30490) 2026-07-09 02:09:03 -07:00
Cheng Wan 65b14881c5 [refactor] Move the EP dispatcher and fusion-workspace manager state onto ctx.resources (#30489) 2026-07-09 02:08:35 -07:00
Cheng WanandClaude Opus 4.8 cc13e2eae7 [CI] Move piecewise CUDA graph (pcg) tests to nightly (#30563) 
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-07-08 16:07:08 -07:00
Cheng Wan 7709a1f358 [refactor] ctx.resources: named slots, stream leases, and workspace buffer leases (#30348) 2026-07-07 21:30:10 -07:00
Cheng Wan b7cca0bf8f [refactor] Collect MoE and DP-attention runtime state into typed flag groups (#30347) 2026-07-07 21:29:27 -07:00
Cheng Wan be32c57598 [refactor] Read resolved config from server_args fields; retire the flags mirror tier (#30346) 2026-07-07 21:28:34 -07:00
Cheng Wan b14f7b4f75 [refactor] Move model-capability adjustments into the resolution pipeline (#30299) 2026-07-07 21:26:55 -07:00
Cheng Wan c861896721 [refactor] Resolve config declarations onto server_args at the end of __post_init__ (#30297) 2026-07-06 18:04:44 -07:00
Cheng WanandClaude Fable 5 602c8615a1 [fix] Reconcile the legacy-getter ratchet baseline after racing merges (#30154) 
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-05 22:06:07 +08:00
Cheng Wan 3ea875fef4 [chore] Remove the stack-review placeholder file (#30149) 2026-07-05 02:39:45 -07:00
Cheng Wan 8fb99bbaf8 [refactor] Config resolution pipeline: full-stack review (10-PR series, review only) (#30137) 2026-07-05 00:00:07 -07:00
Cheng Wan 754524d8de [Fix] Skip cross-node probe in MultimemAllGatherer on single-node runs (fixes mooncake EP segfault) (#30139) 2026-07-04 15:36:23 -07:00
Cheng Wan 00f088f6be [fix] Wrap the sp_shard test entry point in sys.exit so failures propagate (#30138) 2026-07-04 14:23:40 -07:00
Cheng WanandClaude Fable 5 b941e337a4 [refactor] Rename Arg.model_overridable to Arg.resolvable (stack 15/15) (#30077) 
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-04 02:23:01 -07:00
Cheng WanandClaude Fable 5 7ea2284551 [refactor] Migrate the DeepSeek family and the parallel-request chains (stack 14/15) (#30076) 
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-04 02:22:50 -07:00
Cheng WanandClaude Fable 5 3836cba9ee [refactor] Migrate the moe_runner_backend / quantization resolution chains (stack 13/15) (#30075) 
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-04 02:22:38 -07:00
Cheng WanandClaude Fable 5 276fbfe880 [refactor] Migrate the page_size resolution chain (stack 12/15) (#30074) 
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-04 02:22:28 -07:00
Cheng WanandClaude Fable 5 abbb41a214 [refactor] Migrate the attention_backend resolution chain (stack 11/15) (#30073) 
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-04 02:22:17 -07:00
Cheng WanandClaude Fable 5 5c95bf15c8 [refactor] Add the post-process resolution stage; migrate sampling_backend (stack 10/15) (#30072) 
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-04 02:22:05 -07:00
Cheng WanandClaude Fable 5 ae29c5a1dc [refactor] Sweep disable_hybrid_swa_memory writers; close the dtype family (stack 9/15) (#30071) 
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-04 02:21:54 -07:00
Cheng WanandClaude Fable 5 4bf4db09f6 [refactor] Add predicate-keyed registration; migrate the Step3p family (stack 8/15) (#30070) 
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-04 02:21:43 -07:00
Cheng WanandClaude Fable 5 8d8f17e28c [refactor] Migrate the first override families: Mistral/Pixtral dtype, MiniMaxM2, MiMoV2 (stack 7/15) (#30069) 
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-04 02:21:32 -07:00
Cheng WanandClaude Fable 5 df6491d80c [refactor] Wire the config resolution pipeline (dispatch, stash, dual-apply, publish) (stack 6/15) (#30068) 
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-04 02:21:21 -07:00
Cheng WanandClaude Fable 5 c3d751b231 [refactor] Add the declarative model-override registry and resolution gate (stack 5/15) (#30067) 
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-04 02:21:09 -07:00
Cheng WanandClaude Fable 5 10f258257f [refactor] Add the resolved-flags tier + resolvable-field metadata (stack 4/15) (#30066) 
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-04 02:20:58 -07:00
Cheng WanandClaude Fable 5 ff57171f98 [refactor] Soft-deprecate the legacy global ServerArgs accessors + ratchet (stack 3/15) (#30065) 
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-04 02:20:46 -07:00
Cheng WanandClaude Fable 5 def20782cf [refactor] Move ServerArgs ownership into the runtime context (stack 2/15) (#30064) 
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-04 02:20:34 -07:00
Cheng WanandClaude Fable 5 6d662c9245 [refactor] Add a read-through server_args accessor to RuntimeContext (stack 1/15) (#30063) 
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-04 02:19:46 -07:00
Cheng Wan 88db9e033a Adjust KL_THRESHOLD for log probability calculations (#30101) 2026-07-03 23:53:51 -07:00
Cheng Wan 4028304579 [DSA] Disable indexer fusion by default to restore DeepSeek-V3.2 accuracy (#30088) 2026-07-03 20:50:37 -07:00
Cheng Wan 5f623ad24e Revert "Fix wrong RMSNorm fallback to old Flashinfer CUDA kernel when in PCG" (#30083) 2026-07-03 19:05:20 -07:00
Cheng Wan c21f6f19cf [MoE] Fix moe_fused_gate out-of-range expert id on all-NaN rows (fixes eagle_dp_attention crash) (#30079) 2026-07-03 18:48:51 -07:00
Cheng Wan 76f7f7c006 [Spec] Remove the ServerArgs clone + global save/restore hack from DFlashWorkerV2 (#29995) 2026-07-02 23:37:04 -07:00
Cheng Wanandlch1475369 bc25abb786 perf(triton): avoid per-step D2H .item() sync in cuda-graph loc translate (#29921) 
Co-authored-by: lch1475369 <lch1475369@gmail.com>
 2026-07-02 13:20:00 -07:00
Cheng Wan b558cc1abb feat(short-conv): shared ShortConvAttnBackend for ZAYA1 CCA + LFM2 short conv (#29867) 2026-07-01 20:08:00 -07:00
Cheng Wanandlch1475369 4a8e76805c feat(mem_cache): unified memory pool for hybrid Mamba / SWA models (#29678) 
Co-authored-by: lch1475369 <lch1475369@gmail.com>
 2026-07-01 13:21:59 -07:00
Cheng Wanandlch1475369 fc96edd297 feat(mem_cache): page-major (layer-major within a page) KV/state layout (#29533) 
Co-authored-by: lch1475369 <lch1475369@gmail.com>
 2026-06-29 14:49:54 -07:00
Cheng Wan 7b02eab7a6 Revert "[DeepSeek V3] Run routed experts on main stream in dual-stream MoE" (#29452) 2026-06-26 13:59:36 -07:00
Cheng Wan c4376aaa88 [Refactor] Remove dead out_cache_loc_swa buffers (#28968) 2026-06-23 00:03:30 -07:00
Cheng Wan 6fdcb9934c fix(runner): size eager static buffers for prefill budget and MLP-sync autotune (#28677) 2026-06-19 13:23:26 -07:00
Cheng Wan 2aa7b58aa7 refactor(runner): reuse a prepared static buffer for every dummy run (#28740) 2026-06-19 13:16:07 -07:00
Cheng WanandClaude Opus 4.8 856b0dc74b refactor(runner): move kernel warmup into the shared runner lifecycle (warmup()) (#28739) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-19 13:13:13 -07:00
Cheng WanandClaude Opus 4.8 d705a91de1 refactor(runner): add EagerRunner, own the eager path, polymorphic dispatch (#28386) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-19 13:05:43 -07:00
Cheng Wan a6db86d535 refactor(runner): split BaseRunner (shared) from BaseCudaGraphRunner (#28385) 2026-06-19 02:04:15 -07:00
Cheng Wan 1c6331cbd6 refactor(runner): rename runner replay/load/can_run for the shared surface (#28384) 2026-06-19 01:45:04 -07:00
Cheng Wan f83e4d5968 refactor(runner): unify eager-forward DP/MLP-sync padding into one helper (#28383) 2026-06-18 13:41:30 -07:00
Cheng Wan 67db2ac3e7 refactor(runner): unify pp_proxy_tensors forward kwarg into one helper (#28382) 2026-06-18 02:23:47 -07:00
Cheng Wan 53318911ca Add get_parallel(): a structured accessor for parallel-topology state (#28567) 2026-06-17 20:23:43 -07:00
Cheng Wan 448af67a98 ci: run AMD and NPU PR tests on PRs not targeting main (#28368) 2026-06-15 19:10:51 -07:00
Cheng Wan 27ba13358e [Spec] Clear dead DRAFT_EXTEND objects left after EAGLE v1 removal (#28133) 2026-06-13 13:04:51 -07:00
Cheng Wan bde6bccf39 [Spec] Remove deprecated EAGLE v1 DRAFT_EXTEND forward mode (#28129) 2026-06-13 13:00:21 -07:00
Cheng Wan 3a1417a0c1 [refactor] Fold FrozenKVMTPCudaGraphRunner onto the shared DecodeCudaGraphRunner base (#28081) 2026-06-12 16:55:53 -07:00
Cheng WanandClaude Opus 4.8 d601edab73 [Spec] Fix EagleDraftWorker draft-extend attn backend assignment (#28096) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-12 15:47:05 -07:00
Cheng Wan 2e74ff192c [DSV4] Use int64 for compressor out_loc tensors (#27973) 2026-06-11 17:45:34 -07:00
Cheng Wan 97a0031799 [lint] Enable Ruff UP037 to drop redundant quoted annotations (#27984) 2026-06-11 17:38:05 -07:00
Cheng Wan 9425ba2478 Remove extra_pytest_path from pr-test.yml (#27856) 2026-06-10 21:16:14 -07:00
Cheng Wan f4f30d7d23 [Fix] Use int64 seq_lens across all CUDA graph runners and backends (#27840) 2026-06-10 19:54:58 -07:00
Cheng Wan 70c71ba183 [NPU] Fix dead patch_model monkey-patch breaking NPU torch.compile capture (#27774) 2026-06-10 01:15:50 -07:00
Cheng Wan 95d8a75bc9 Bundle set_kv_buffer write targets into KVWriteLoc (loc + swa_loc) (#27695) 2026-06-09 23:09:51 -07:00
Cheng Wan 758fd4bb9a [SWA] Cache full→SWA out_cache_loc per forward across attention backends (#27617) 2026-06-09 22:57:51 -07:00
Cheng WanandClaude Opus 4.8 9097647090 Route the eager forward path through the CUDA graph input-buffer registry (#27407) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-06 14:35:53 -07:00
Cheng WanandClaude Opus 4.8 7dc7376697 fix(attn): delegate init_mha_chunk_metadata in HybridLinearAttnBackend (#27316) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-04 17:44:23 -07:00
Cheng WanandClaude Opus 4.8 0aa72a9e76 Replace skip_attn_backend_init with a batch-carried attention plan marker (+ staleness re-plan) (#27193) 
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-04 17:13:18 -07:00
Cheng WanandClaude Opus 4.8 10ab7c919f [refactor] Retire DecodeInputBuffers / PrefillInputBuffers in favor of CudaGraphBufferRegistry (#27192) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-03 20:52:56 -07:00
Cheng WanandClaude Opus 4.8 c9ca56da8c Unify full→SWA index translation in init_forward_metadata; drop pool caches (#27091) 
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-03 16:12:27 -07:00
Cheng Wan 61aa3293d3 Revert "Fix TokenizerManager crash on top_logprobs with tensor values" (#27187) 2026-06-03 14:53:28 -07:00
Cheng WanandClaude Opus 4.8 e485ad6ac1 Fix hybrid linear attention dispatch by layer id with draft-worker awareness (#27120) 
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-03 14:11:14 -07:00
Cheng WanandClaude Opus 4.8 45604a0f4a [refactor] Unify CUDA graph runner input buffers behind CudaGraphBufferRegistry (#26742) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-03 10:54:10 -07:00
Cheng Wan 73b53e7a87 Revert "Support NextN = 2/4 in DSV32" (#27138) 2026-06-03 01:29:52 -07:00
Cheng Wan 202e618898 Revert "Fix hybrid linear attention misrouting plain-RadixAttention linear layers to the full backend (Ring-2.5-1T)" (#27116) 2026-06-02 23:27:57 -07:00
Cheng WanandClaude Opus 4.7 99da43b900 [refactor] init_forward_metadata 3-method ABC + side-channel removal + ForwardMetadata type rename (#26735) 
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-02 10:33:33 -07:00
Cheng WanandClaude Sonnet 4.6 ff8ed7a302 [refactor] unify cuda-graph capture/replay across attention backends (#26665) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-29 12:46:42 -07:00
Cheng WanandClaude Sonnet 4.6 ec075d8bc5 Fix DRAFT_EXTEND_V2 CG metadata: align test fixture and Triton with production seq_lens convention (#26651) 
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-29 02:46:45 -07:00
Cheng WanandClaude Sonnet 4.6 0fb0ea7aac test: add trtllm_mha EAGLE-draft CG runner coverage (chain) (#26669) 
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-28 23:46:06 -07:00
Cheng WanandClaude Sonnet 4.6 2dfbc3d781 test: strengthen CG-replay coverage with prod-fill padding, metadata invariants, and pad-ratio sweep (#26658) 
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-28 22:43:29 -07:00
Cheng Wan 84698b21e7 rename unittest as unittests (#26654) 2026-05-28 21:14:06 -07:00
Cheng WanandClaude Sonnet 4.6 e381312664 Revert "Fix FA DRAFT_EXTEND_V2 cache extent" (#26628) 
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-28 18:45:33 -07:00
Cheng WanandClaude Opus 4.7 f66f56c6bd Add attention-backend unit-test suite under test/registered/attention/unittest (#26517) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 17:30:31 -07:00
Cheng WanandClaude Opus 4.7 4f92e63c99 Let unittest._ShouldStop propagate through retry() so subTest+failfast works (#26616) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 16:26:17 -07:00
Cheng WanandClaude Opus 4.7 12e28bdf0c Fix FlashInfer SWA EXTEND-with-prefix correctness in merge_state path (#26513) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 01:16:58 -07:00
Cheng WanandClaude Opus 4.7 00cd6fb3d9 Add sliding-window mask support to TorchNativeAttnBackend (#26516) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 01:06:57 -07:00
Cheng WanandClaude Opus 4.7 8ca09a30f1 Allow Optional key/value in unified_attention_with_output split-op (MLA absorb fix) (#26515) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 01:04:07 -07:00
Cheng WanandClaude Opus 4.7 b429a30428 Expose Flex attention causal/decode masks as static methods (#26514) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 01:03:32 -07:00
Cheng WanandClaude Opus 4.7 e5f5d84780 Fix FA DRAFT_EXTEND_V2 cache extent (#26512) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-28 00:56:23 -07:00
Cheng Wan 51840ca459 Add xutizhou as code owner for eplb directory (#26479) 2026-05-27 03:32:35 -07:00
Cheng WanandCheng Wan 44922de48a fix(swa): downgrade translate_loc_from_full_to_swa key-change log from warning to debug (#26225) 
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
 2026-05-24 11:37:46 -07:00
Cheng WanandCheng Wan 83a18e687d Revert "[refactor] unify cuda-graph capture/replay across attention backends (#26134)" (#26166) 
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
 2026-05-23 02:32:08 -07:00
Cheng WanandCheng Wan 5964d30233 fix(swa): eliminate spurious translate_loc_from_full_to_swa warning in BCG and CG paths (#26152) 
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
 2026-05-23 00:01:12 -07:00
Cheng WanCheng WanClaude Sonnet 4.6
 		d226f75669 [refactor] unify cuda-graph capture/replay across attention backends (#26134) 
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-22 18:51:13 -07:00
Cheng WanandClaude Sonnet 4.6 d765dfd043 refactor(attn): init hisparse_coordinator before attn_backend; replace lazy property with init-time capture (#26012) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-21 16:03:42 -07:00
Cheng WanandClaude Sonnet 4.6 c5251a98a9 feat(model_runner): remove pool/backend refs from ForwardBatch via ForwardContext (#25983) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-21 14:01:49 -07:00
Cheng WanandClaude Opus 4.7 b765faee30 [MoE Refactor] deprecate forward_npu and NpuFuseEPMoE (#25678) 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 10:25:17 -07:00
Cheng WanandClaude Sonnet 4.6 888a8794ef [Fix] DSV4 cached_loc invalidated when SWA mapping is rebuilt (#25889) 
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
 2026-05-20 22:38:12 -07:00