Commit Graph
153 Commits
Author SHA1 Message Date
Jason Mancusoandhnyls2002 201c4b9b2a [RL, Spec] Introduce top-p mask capture for spec and add DFlash/DSpark impl (#34201)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-09-30 13:32:26 -07:00
Nan Jiang 24b6930076 [Sampling] Stream sampling masks as per-request arrays (#40986) 2026-09-25 12:20:25 -07:00
Ting SUN 752801e4d1 fix(sampling): validate sampling_seed is an int within int64 range (#28960)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
2026-09-24 17:25:13 +02:00
Nan Jiang c3685dff42 [Sampling] Add selected/support sampling logprob modes (#40932) 2026-09-23 21:38:19 -07:00
f31a7bd45c Use pinned memory for asynchronous sampling metadata transfers (#39777)
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-09-20 15:31:31 -07:00
Lifan Shen 8ea0ee300d perf(sampling): avoid GPU syncs when applying custom logit processors (#39234) 2026-09-18 17:09:30 -07:00
Jensen 5beb2fd552 [NPU] Avoid device synchronization in Ascend sampling (#39404) 2026-09-16 10:15:16 +03:00
jasonjk-parkandjasonjk 073fdf97cc Generalize auxiliary outputs (#39164)
Co-authored-by: jasonjk <jasonjk@twshared0085.36.lco2.facebook.com>
2026-09-15 12:05:13 -07:00
Yuxuan ZhangandXinyuan Tong 17ba2c2e7c Support non-strict GLM47 tool calls with EBNF constraints (#38890)
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-09-15 16:41:48 +08:00
fd7743e0e1 [Sampling] Support sampling masks with overlap scheduling (#36631)
Co-authored-by: ByronHsu <ByronHsu@users.noreply.github.com>
Co-authored-by: root <root@slurm-h200-208-179.slurm-compute.tenant-slurm.svc.cluster.local>
Co-authored-by: Byron Hsu <byron+per@periodiclabs.ai>
2026-09-10 14:52:44 -07:00
Byron HsuandByron Hsu a88e852fab [Sampling] Allow sampling-mask replay with DisallowedTokensLogitsProcessor (#38279)
Co-authored-by: Byron Hsu <24364830+ByronHsu@users.noreply.github.com>
2026-09-06 23:56:16 -07:00
3bac084d4e [Model] Add native IFM K2 Horizon serving support (#37654)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-09-03 16:39:43 +08:00
Alex NailsandAlison Shao 28262c20df [CI][RFC] Replace black-jupyter with ruff-format (#37210)
Co-authored-by: Alison Shao <a.shao@wustl.edu>
2026-09-02 19:46:08 -07:00
Byron HsuandByronHsu 046cdaabaa [Sampling] Capture masks from sampler support (#36630)
Co-authored-by: ByronHsu <ByronHsu@users.noreply.github.com>
2026-09-02 17:13:28 -07:00
EchO 5b7c62d5d6 fix(frontend): bound stop strings and regex patterns (#37029) 2026-08-29 22:25:26 -07:00
ec4bdbfa4a [Feature] Beam search support (#31626)
Co-authored-by: cswuyg <cswuyg@gmail.com>
Co-authored-by: cswuyg <496090217@qq.com>
Co-authored-by: Vedant Jhaveri <vedantjh2@gmail.com>
Co-authored-by: Vedant Jhaveri <vjhaveri@linkedin.com>
2026-08-26 16:56:15 -07:00
Alec SandAlec Solder fbafd1b123 Add sampling observer auxiliary output hooks (#35747)
Co-authored-by: Alec Solder <alecs@fb.com>
2026-08-21 19:28:18 -07:00
milesial 082aac8fce [Bugfix] Fix min-new-token EOS handling (#31378)
Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>
2026-08-19 15:29:22 -07:00
Lifan ShenandXinyuan Tong a2161ce682 Support thinking budget for Inkling (#33146)
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
2026-08-11 02:01:08 +08:00
Cheng Wan eda0ddc260 config: delete the dead get_server_args() bindings across the repo (#33888) 2026-08-07 22:41:53 -07:00
zijiexiaandClaude Opus 4.8 b819d2fb5b [Docs] Rename docs_new/ to docs/ (#32123)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-03 16:51:00 -07:00
Liangsheng Yin f5f021672a [Fix] Treat an empty grammar constraint as unset in SamplingParams (#33328) 2026-08-02 21:45:51 -07:00
Mark 741e33db81 fix(sampling): reject conflicting structural tag constraints (#32525) 2026-08-02 20:24:55 -07:00
Cheng Wan 55b6769b0e config: read resolved config via namespace accessors (#33013) 2026-07-31 15:06:59 -07:00
Lianmin Zheng 16a52bff23 [Refactor] Move sampling tokenizer validation helper (#32694) 2026-07-28 16:48:03 -07:00
Lianmin Zheng 9ca4023b13 [Core] Clean up array-like msgspec structs (#32688) 2026-07-28 16:25:22 -07:00
Lianmin ZhengandAlec S 9989077f24 Use native batched llguidance mask generation (#32412)
Co-authored-by: Alec S <10566873+alecsolder@users.noreply.github.com>
2026-07-25 16:36:32 -07:00
Liangsheng Yin 3da1071d56 [Spec] Hold the grammar bitmask in one GrammarMask type across all decode paths (#32409) 2026-07-25 15:27:43 -07:00
Raghavendra Vedula a9497e8d73 Guard min_new_tokens penalizer against None eos_token_id (#31973) 2026-07-22 23:24:25 +08:00
hari-hm a909077d22 Return top-p/top-k sampling mask/nucleas (#27408) 2026-07-13 17:58:51 -05:00
zijiexia 50ed4c011f Remove legacy Sphinx docs/ and finish the Mintlify cutover (#28964) 2026-07-13 15:06:08 -07:00
6cc9352dfe [Spec] Add DSpark: confidence-scheduled speculative decoding (#30261)
Co-authored-by: sglang-bot <232288953+sglang-bot@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Liangsheng Yin <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-07-12 17:25:26 -05:00
Cheng Wan 1f15308dca [refactor] Retire the legacy config accessor and the remaining process singletons (#30493) 2026-07-09 02:10:47 -07:00
Rain JiangandLianmin Zheng be1930133a Convert IPC dataclasses to msgspec.Struct with opt-in msgpack transport (#28688)
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
2026-06-26 12:04:03 -07:00
Lianmin ZhengandRain Jiang fd87a85388 Convert SamplingParams to msgspec Struct (#29198)
Co-authored-by: Rain Jiang <96632942+rainj-me@users.noreply.github.com>
2026-06-24 13:42:45 -07:00
Liangsheng Yin 556cf54d47 [Perf] Avoid per-decode-step host sync in min_new_tokens penalty (#28397) 2026-06-15 23:32:28 -07:00
0417951a86 [Bug Fix] Validate tokenizer-dependent features with skip_tokenizer_init (#27882)
Co-authored-by: Randall <randall@iterationlab.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-14 20:07:52 -07:00
Ting SUN b250bea994 fix(sampling): reject non-finite temperature in SamplingParams.verify (#28153)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
2026-06-13 23:41:36 -07:00
Qiaolin Yu e02f7ca482 [perf] remove several h2d sync (#28076) 2026-06-12 20:30:17 -07:00
Cheng Wan 97a0031799 [lint] Enable Ruff UP037 to drop redundant quoted annotations (#27984) 2026-06-11 17:38:05 -07:00
Mohammad Miadh AngkadandLianmin Zheng 52f221cce0 Fix Req array token-id concatenation (#26182)
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
2026-06-06 19:59:51 -07:00
fzyzcjy 268f4c82f1 Add rids/bootstrap-room int-hash plumbing for deterministic per-request identification (#26814) 2026-05-31 09:57:22 +08:00
ZeyuanChen2000 90eb894564 [NPU] fix model llava-onevision-qwen2-7b-ov torch compiles error in npu case (#26573) 2026-05-30 17:10:58 +08:00
RulinJuiceandRulinJuice 3f048c80b8 Reject repetition_penalty=0 in SamplingParams.verify() (#24874)
Co-authored-by: RulinJuice <265952454+RulinJuice@users.noreply.github.com>
2026-05-12 21:25:23 -07:00
cctry 0addd185af Fix /generate endpoint crash when sampling params contain null values (#23401) 2026-04-22 09:56:10 -07:00
KrishnanPrash a937ec31be fix: server crash when stop_token_ids contains null (#22175)
Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
2026-04-10 11:42:23 -07:00
72d3d8f4cf [Feature Restoration] repetition_penalty is essential for GLM-V models (#21258)
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-03-31 23:29:49 -07:00
Aurick Qiao a34e9ed64a Add adjusted_filter_batch (#21260) 2026-03-26 10:59:05 +08:00
Liangsheng Yin e6d40bff81 Revert "feat: reduce constrained-decoding overhead in TP" (#16845) 2026-01-10 11:39:38 +08:00
Raayan Dhar 76b3c698d6 feat: reduce constrained-decoding overhead in TP (#13947)
Signed-off-by: Raayan Dhar raayan.dhar@gmail.com <raayan.dhar@gmail.com>
Signed-off-by: raayandhar <raayan.dhar@gmail.com>
2026-01-10 00:38:32 +08:00