Commit Graph
139 Commits
Author SHA1 Message Date
Yuwei AnandClaude Fable 5 07d84ebd6d [2/N][Mixed] Mixed chunk prefill with spec enabled (#36933)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-31 10:48:07 -07:00
Cheng WanandClaude Opus 5 1a3e152f03 config: six more runtime readers ask the bags (#36973)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-29 04:19:14 -07:00
Cheng WanandClaude Opus 5 6ff2a20ccf config: the record is not an object that gets passed around (#36622)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-27 12:57:10 -07:00
Cheng Wan 5b7fc61306 config: resolution reads the declarations, not the fields (#36253) 2026-08-26 05:05:28 -07:00
Aurick QiaoandYuwei An 58ecbba0bd [Feature] Coordinate FullCG prefill across DP-attention ranks (#35640)
Co-authored-by: Yuwei An <ayw.sirius19@gmail.com>
2026-08-26 02:16:02 -07:00
cctryandcctry 37c09ff3d8 [Memory] Borrow CUDA graph pool storage for EAGLE sampling (#35375)
Co-authored-by: cctry <cctry@fb.com>
2026-08-18 16:19:41 -07:00
Cheng Wan 1ab713c334 config: the post-publish consumers of the supplied-instance surface read the bags
config: the speculative workers take page_size from the bags

Seven worker constructors stored `self.page_size = server_args.page_size` off
the handed record. They all run after publish and all keep a copy of a
process-level value, which is the first row of the plan doc's supplied-instance
disposition table -- so they read `get_schedule().page_size`, and a post-publish
override now reaches them like it reaches every other consumer.

The supplied-instance census named the seven pairs; the exposure ratchet in the
next member pins what remains after this batch of conversions.


config: the post-publish chunked_prefill_size consumers read the bags

Four of the ten supplied-instance `chunked_prefill_size` reads are plain
post-publish consumers -- the EPLB recorder's buffer sizing, the deep-gemm
compile warmup (five reads), the KV-cache builder's effective size, and the
ngram embedding manager's assert. All are reached from runner init, so they read
`get_schedule()`.

Two are deliberately left: `create_kt_config_from_server_args` builds a config
*from a supplied record* by name and contract, and `CanaryLaunchCapacities.from_args`
is the same shape. Converting those would change what the function is, not where
it reads -- the plan doc's disposition table says so per field.


config: the remaining post-publish graph/limit consumers read the bags

Three more of the census's supplied-instance debts are plain post-publish reads: the dspark worker's
cuda-graph decode sizes, the dspark planner's SPS table bound
(`max_running_requests`), and the LoRA manager's cuda-graph moe buffers. The
dspark worker is the clearest of them -- it already read
`get_exec().graph.cuda_graph_config.decode.bs` thirty lines below the instance
read, so the file disagreed with itself about where the same value comes from.

Left where the function's contract is "build a config from the record you are
handed" rather than "read this process's config":
`create_kt_config_from_server_args`, `DllmConfig.from_server_args`,
`CanaryLaunchCapacities.from_args`, `build_compilation_config`. Changing those
would change what the function is.


config: the runner, scheduler and offload manager take page_size from the bags

The same `self.page_size = server_args.page_size` shape as the speculative
workers, in the three remaining process-owned constructors: `ModelRunner`,
`Scheduler`, and the decode-side KV offload manager. The scheduler process
publishes before any of them run. The one path that did not is `ModelRunner`
constructed standalone -- `python -m sglang.benchmark.one_batch` and the manual
runner tests build it with no prior publish, and the constructor's own publish
sat below this read -- so that publish moves above the constructor's first bag
read instead of leaving a window where the runner half-exists unpublished.

Left where the read belongs to something else: `utils/common`'s predicates are
called only from the resolution pipeline with a `resolved_view`,
`allocation_sizing` takes the config its callers supply by contract, and
`CudaVmmFeatureTransport` is tokenizer-owned -- one per tokenizer worker, which
is the per-instance boundary.

The conversion left the offload manager parking a record it no longer
reads; the parked copy goes with the read (the constructor parameter stays
-- its hicache sizing still reads it directly).
2026-08-15 00:39:35 -07:00
weireweireandweireweire 6eb941a34c [Spec] Support MegaMoE for DSpark under dp attention (#34844)
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
2026-08-14 17:20:01 -07:00
fde9ad2531 [Feature] Add Muse Glimmer model support (#34262)
Co-authored-by: sglang-bot <232288953+sglang-bot@users.noreply.github.com>
Co-authored-by: Brayden Zhong <brayden.zhong@radixark.ai>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-08-11 15:41:52 -07:00
Liangsheng YinandBaizhou Zhang f6a6f5bf1e [CI] Trim redundant nightly test registrations (#34070)
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
2026-08-08 01:42:46 -07:00
Liangsheng Yin 9bd1461757 [CI] Bound the CUDA graph capture range in test launches and lift the spec fixture's admission cap (#33776) 2026-08-05 20:00:30 -07:00
Liangsheng Yin 3869fe556f [CI] Collapse the EAGLE launch matrix and the scoring engine boots on the per-commit runners (#33756) 2026-08-05 16:43:19 -07:00
Liangsheng Yin de34dd11e9 [CI] Fold duplicate-server suites and prune the retract matrix on 1-gpu-5090 (#33745) 2026-08-05 12:41:51 -07:00
Jason Mancuso 36853b8ffc [Spec] Support logprobs with DFlash (#33459) 2026-08-05 12:37:12 -07:00
Liangsheng Yin 0d99d91e49 [CI] Make B200 base-b suites single-GPU as prep for 1-gpu B200 runners (#33605) 2026-08-04 15:53:01 -07:00
Liangsheng Yin 0753663b8e [CI] Trim redundant B200 test registrations (#33586) 2026-08-04 13:22:00 -07:00
ashwini rathiandMa Mingfei 53804d609c [CI][XPU] Stabilize XPU CI: pin UMD/IGC, retry infra flakes, right-size EAGLE3 (#32438)
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
2026-08-04 16:28:52 +08:00
Vladislav NosivskoyandXinyuan Tong 154f0ac662 Fix DSpark and DP/EP (#33098)
Signed-off-by: Vladislav Nosivskoy <vladnosiv@gmail.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
2026-08-04 00:35:57 -07:00
Liangsheng YinandKaixi 6ab3231b97 [Perf] Skip the target-verify tree mask fill when the backend never reads it (#32886)
Co-authored-by: Kaixi <kaiximatteoc@nvidia.com>
2026-07-30 02:32:38 -07:00
Hồ Sỹ Thếandhnyls2002 d021990bf5 [DFLASH] Support grammar-constrained decoding in speculative verify (#30096)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-07-25 04:36:46 -07:00
Liangsheng Yin 3c5bf1f6d2 [Spec] Derive NGRAM grammar tree links on the host instead of reading back retrive_next_token (#32380) 2026-07-25 02:25:16 -07:00
ANSHUMAN TRIPATHY f0f78a6c93 Add deterministic inference for eagle parity test (#30026) 2026-07-24 12:48:21 +08:00
Xiaoyu ZhangandClaude Opus 4.8 62aa85d9aa [Kernel] Sweep missed dedicated kernels into kernels.ops (moe/quant siblings + dspark) (RFC #29630) (#32160)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 17:07:16 +08:00
Liangsheng Yin f35411ee81 [Spec] Enable grammar overlap scheduling for STANDALONE speculative decoding (#32110) 2026-07-23 01:47:27 -07:00
Liangsheng Yin 024639a372 [Perf] Stack dspark dense draft per-layer ctx KV projection into one GEMM (#31986) 2026-07-21 17:46:13 -07:00
Xiaoyu ZhangandClaude Opus 4.8 216b750c8f [Kernel] Sweep decoupled scattered kernels into sglang.kernels.ops (RFC #29630) (#31582)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-18 19:07:07 +08:00
Jorge António 26cb0fcdda Empty _REQ_TYPES_WITH_OPAQUE_FIELDS on the msgpack IPC path (#29465 Task 4) (#30182) 2026-07-15 14:55:06 -07:00
6cc9352dfe [Spec] Add DSpark: confidence-scheduled speculative decoding (#30261)
Co-authored-by: sglang-bot <232288953+sglang-bot@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Liangsheng Yin <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-07-12 17:25:26 -05:00
ashwini rathiandarathi-hlab 850719ebd9 [XPU] Unbreak stage-b: re-add --disable-decode-cuda-graph, quarantine EAGLE3 parity (#30048)
Co-authored-by: arathi-hlab <arathi.rajanraghavendran@intel.com>
2026-07-06 12:25:46 +08:00
Cao E 9df16b5ba9 [XPU] Remove redundant xpu graph backend and make xpu graph opt-in by default (#29911) 2026-07-03 15:58:56 +08:00
ANSHUMAN TRIPATHY 4fffc6448b Speculative decoding support on XPU (#23180) 2026-07-02 13:23:24 +08:00
Liangsheng Yin 909123ddb8 [misc] Use --cuda-graph-max-bs-decode in tests, examples, and docs (#29591) 2026-06-28 18:38:28 -07:00
f42ec350b4 [mtp] add rejection sampling for speculative decoding (#26312)
Co-authored-by: lyc508653 <lyc508653@alibaba-inc.com>
Co-authored-by: Qiaolin-Yu <liin1211@outlook.com>
Co-authored-by: Huiqiang Jiang <30883354+iofu728@users.noreply.github.com>
Co-authored-by: Yi Zhang <25844240+yizhang2077@users.noreply.github.com>
Co-authored-by: Yizhong Cao <114661107+cao1zhg@users.noreply.github.com>
2026-06-20 15:10:42 -07:00
Michael 62ab09a478 [AMD] register 2 spec tests to stage-b-test-1-gpu-large-amd (batch-5) (#28558) 2026-06-18 16:33:23 -07:00
Michael 5d1949152d [AMD] ci: add extra-a 1-gpu-large tier (fp8kv-triton, streaming-session, spec-standalone) (#28458) 2026-06-17 23:31:32 -07:00
Liangsheng Yin a663500ea9 [Test] Fold EAGLE return_hidden_states regression into spec triton suite (#28577) 2026-06-17 16:42:02 -07:00
5ea0d1d093 add dflash gemma4 support (#27471)
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 16:39:28 -07:00
Khoa PhamandClaude Opus 4.7 e4fd613def [Spec] Fix return_hidden_states under spec V2 (issue #26163) (#28496)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-17 16:21:36 -07:00
Yinghai LuandLianmin Zheng fcca4611fa [CAR] Let custom allreduce support VMM based allocation (#27593)
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
2026-06-16 04:49:05 -07:00
Qiaolin Yuandshuwenn e068355831 [spec decoding] supports step 0 in adaptive spec decoding (updating draft kv cache without draft decoding) (#23994)
Co-authored-by: shuwenn <2508695655@qq.com>
2026-06-15 22:21:26 -07:00
Jonah Bernardandjonahbernard bcd45d3ac7 [AMD][DFlash] Add DFlash test to AMD CI (#28112)
Co-authored-by: jonahbernard <Jonah.Bernard@amd.com>
2026-06-12 22:06:41 -07:00
Liangsheng Yin 3ffe72517f [Spec] Remove the dead spec V1 scheduler paths (#27977) 2026-06-11 18:31:13 -07:00
Liangsheng Yin c0480a88be [Spec] Retire Spec V1 (#27964) 2026-06-11 16:15:15 -07:00
David Wang 588d1f7bc9 [Feature] Spec V2 DFlash Support (#23000) 2026-06-10 19:27:42 -07:00
shuwennandQiaolin Yu 3600a9ac5f [SPEC] feat: init adaptive spec params from config (#27493)
Co-authored-by: Qiaolin Yu <liin1211@outlook.com>
2026-06-10 18:55:25 +00:00
111009ea54 [Feature] [Ngram spec] Support ngram spec v2 (#17260)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Ratish P <114130421+Ratish1@users.noreply.github.com>
2026-06-10 02:46:00 -07:00
decb88e0e3 Support spec v2 for Frozen-KV MTP; remove v1 worker (#27607)
Co-authored-by: Khoa Pham <khoa.pham@radixark.ai>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 15:30:20 -07:00
Liangsheng Yin 3fe6bc390b [Spec] Naming cleanup: contiguous draft-loc kernel + accepted->accept (#27599) 2026-06-08 15:04:58 -07:00
Liangsheng Yin 032c9efb46 Enable async-assert invariant probes by default in CI (#27461) 2026-06-06 16:23:21 -07:00
Liangsheng Yin 84ca0ffb8c Spec v2 tree drafting (topk>1) with page_size>1 (#26972) 2026-06-06 12:00:27 -07:00